A Dataflow is an integration pipeline to access files from a source, read the files, process or transform the data, and save each entry of the CSV file to the Capillary Database. You can create a dataflow using predefined templates.

This article provides detailed steps on creating a data flow pipeline to inject data from an SFTP server to the Capillary database.

Currently, Connect+ supports only FTP integration. 

Dataflow is an integration configuration in Connect+. You can add various integrations in a workspace (organization).

No two dataflows should use the same FTP directory. Only one dataflow is processed in that case.

To add a new integration, follow these steps.

  1. On the Connect+ home page, select the workspace for which you want to add an integration.
  2. Click Add Dataflow.

  1. In Dataflow name, enter a name for the integration.
  2. To select an integration template, click + Select template > Select your preferred template. >click Done.
  • Intouch Customer V2 Add: To capture data of new customer registrations.
  • InTouch Transaction V2 Add: To capture transaction details.
  • Transaction V2 Add with Filter: To capture transaction details with a filter. You can apply a filter condition to the transaction data.
  • Data Reconciliation: In this, the source data will not be compared to a file exported through Data Export Framework. Instead, the source data will be compared directly with all the events (transactions in this scenario) present in Insights+ (backend). This way, it also it removes an additional step of exporting data.
    While configuring the template in Connect+, you just need to point the FTP location where source data will be dumped and specify the type of event in  Connect+. The Data Reconciliation flow compares the data directly with the events in the backend of Insights+ to generate the missing events. 
  1. Click Continue to proceed. You will see configuration options based on the template selected.
  2. Integration configurations vary with the source type. The following are the steps involved in FTP integration.

FTP/SFTP Integration

- Step 1: Configure SFTP (SSH File Transfer Protocol) Server

- Step 2: Configure Field Mapping & Data Transformation

- Step 3: Provide API Authentication Details (OAuth)

- Step 4: Configure Dataflow Frequency