« Back to help center

Integrating bulk data

This guide explains how to upload custom data to Contiamo.

Contents

Integrating bulk data

1. Custom data

Custom data integration lets you upload your own CSV files to Contiamo. You will then be able to use Contiamo’s intuitive interface to explore and visualize the data, and combine it with data from other sources.

The first step is to create a custom data source to hold your custom data. The second step is to add data contracts to the data source. Each data contract will be set up to receive data from a CSV file in a particular format.

1.1. Create a custom data source

A custom data source can be installed like any other data source: In the Data sources tab of the Manage section, click + Add new and select the custom data type:

custom_data_app

All you have to do at this stage is enter a name, hit Enter, and a new custom data source will be created. Next, you will need to set up data contracts for the data source.

2. Uploading simple data tables

2.1. Set up a (basic) data contract

In order to complete this step you should already have a CSV file ready. Our setup process will discover the data structure from this file.

After creating the custom data source, you should be presented with the screen below. (If you have navigated away, select the custom data source from the installed data sources list in the manage section, and click on Data contracts.)

data_contracts

Click + Add new contract on the right, then enter a contract name and hit Enter. A blank data contract will be created. The sidebar on the left lets you perform various administrative tasks:

data_contract_sidebar

Select Columns under Structure in the sidebar. Click Discover structure from file in the upper right corner, and upload your CSV file.

discover_file_structure

Contiamo is pretty good at automatically detecting data types. You can edit the field names, as these names will be used in Contiamo. Once you are done, click Save Changes in the upper right corner.

save_changes

One nice additional step is to check whether the ‘count’ metric automatically created by Contiamo is relevant to your data. Select Query Fields under Advanced in the sidebar. If it is meaningless, remove the ‘count’ metric.

Note: your data has not been uploaded yet. The CSV file you uploaded was only used to discover the structure of the data. Now that the data contract has been set up, the data can be uploaded from the CSV file as described below, in the same way additional data will be uploaded in the future as you generate it.

Note: If you have data from multiple, interrelated tables too complicated to upload in the way just described, please get in touch with us, we will be happy to help.

2.2. Upload data to an existing contract

In the Data contract section of your custom data source, click Manual under Upload data in the sidebar on the left. Upload your CSV file, and wait for the confirmation message at the top of the page. The upload should take a couple of minutes at most.

upload_confirmation

You can check that your data has been properly uploaded in the Data tab.

upload_confirmation

Tip: You can quickly check that the data is uploaded by checking the number of rows:

number_of_rows

3. Uploading linked data tables

Linked data tables are tables related by primary/foreign keys.

Please contact us at support@contiamo.com for help with linked tables.

4. Deleting data

A custom data source can be deleted in its entirety from the Summary view like any data source.

You can delete a specific data contract in a data source by selecting it in the Data contracts tab. In the overview, click on Delete contract. This removes the contract altogether, and cannot be undone.

Finally, you can clear the data from a data contract by selecting Clear in the Data section of the sidebar on the left. This will clear the data from the contract, but keep the existing file structure intact, so you can upload data from your CSV files again.

5. Setting up automated data uploads

You can upload CSV data with a simple HTTP POST request.

CSV file Instructions
Encoding UTF-8
Separate columns with semicolon ( ; )
Wrap column values in double quotes ( "" )
Decimal separator dot ( . )
Thousand separator comma ( , )
First row Field keys defined in the data contract structure

In order to get a customized endpoint and data token, please visit the data contract page and click on the API link in the Upload data section of the sidebar on the left.

Endpoint Instructions
HTTP method POST
Content-Type multipart/form-data
Form field name file
Endpoint Please visit the data contract for the customized endpoint
Access token Please visit the data contract for a customized access token

Note: You can send the access token as a url parameter. For instance:

$ curl -X POST -F "file=@filename.csv" https://api.contiamo.com/12345678/apps/123456789/data_contracts/contracts/contract_key/upload/process?access_token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

It is then up to you to schedule data uploads via our API.