Integrating bulk data
This guide explains how to upload custom data to Contiamo.
Integrating bulk data
1. Custom data
Custom data integration lets you upload your own CSV files to Contiamo. You will then be able to use Contiamo’s intuitive interface to explore and visualize the data, and combine it with data from other sources.
The first step is to create a custom data source to hold your custom data. The second step is to add data contracts to the data source. Each data contract will be set up to receive data from a CSV file in a particular format.
1.1. Create a custom data source
A custom data source can be installed like any other data source: In the Data sources tab of the Manage section, click + Add new and select the custom data type:
All you have to do at this stage is enter a name, hit Enter, and a new custom data source will be created. Next, you will need to set up data contracts for the data source.
2. Uploading simple data tables
2.1. Set up a (basic) data contract
In order to complete this step you should already have a CSV file ready. Our setup process will discover the data structure from this file.
After creating the custom data source, you should be presented with the screen below. (If you have navigated away, select the custom data source from the installed data sources list in the manage section, and click on Data contracts.)
Click + Add new contract on the right, then enter a contract name and hit Enter. A blank data contract will be created. The sidebar on the left lets you perform various administrative tasks:
Select Columns under Structure in the sidebar. Click Discover structure from file in the upper right corner, and upload your CSV file.
Contiamo is pretty good at automatically detecting data types. You can edit the field names, as these names will be used in Contiamo. Once you are done, click Save Changes in the upper right corner.
One nice additional step is to check whether the ‘count’ metric automatically created by Contiamo is relevant to your data. Select Query Fields under Advanced in the sidebar. If it is meaningless, remove the ‘count’ metric.
Note: your data has not been uploaded yet. The CSV file you uploaded was only used to discover the structure of the data. Now that the data contract has been set up, the data can be uploaded from the CSV file as described below, in the same way additional data will be uploaded in the future as you generate it.
Note: If you have data from multiple, interrelated tables too complicated to upload in the way just described, please get in touch with us, we will be happy to help.
2.2. Upload data to an existing contract
In the Data contract section of your custom data source, click Manual under Upload data in the sidebar on the left. Upload your CSV file, and wait for the confirmation message at the top of the page. The upload should take a couple of minutes at most.
You can check that your data has been properly uploaded in the Data tab.
Tip: You can quickly check that the data is uploaded by checking the number of rows:
- Count the number of rows in your CSV file (excluding the header row).
- Before uploading the file, make a note of the number of rows currently in the data contract (this information is in the data contract overview).
- Upload the file as instructed above.
- Go back to the overview and check that the number of rows is increasing by refreshing your browser.
- You will know the data is uploaded once the number of rows reaches the expected total.
3. Uploading linked data tables
Linked data tables are tables related by primary/foreign keys.
Please contact us at firstname.lastname@example.org for help with linked tables.
4. Deleting data
A custom data source can be deleted in its entirety from the Summary view like any data source.
You can delete a specific data contract in a data source by selecting it in the Data contracts tab. In the overview, click on Delete contract. This removes the contract altogether, and cannot be undone.
Finally, you can clear the data from a data contract by selecting Clear in the Data section of the sidebar on the left. This will clear the data from the contract, but keep the existing file structure intact, so you can upload data from your CSV files again.
5. Setting up automated data uploads
You can upload CSV data with a simple HTTP POST request.
|Separate columns with||semicolon (
|Wrap column values in||double quotes (
|Decimal separator||dot (
|Thousand separator||comma (
|First row||Field keys defined in the data contract structure|
In order to get a customized endpoint and data token, please visit the data contract page and click on the API link in the Upload data section of the sidebar on the left.
|Form field name||file|
|Endpoint||Please visit the data contract for the customized endpoint|
|Access token||Please visit the data contract for a customized access token|
Note: You can send the access token as a url parameter. For instance:
$ curl -X POST -F "email@example.com" https://api.contiamo.com/12345678/apps/123456789/data_contracts/contracts/contract_key/upload/process?access_token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
It is then up to you to schedule data uploads via our API.