Continuous Data Import from External Apps

This guide will show you how to import data from an external app and keep it up-to-date in your app.

If you want to see the end result right away, you can start with a pre-built scenario or check out one of the examples in our Examples App.

Otherwise, follow the steps below.

To make this guide specific, we will import Users, but you can use the same approach for any other type of data.

Data Source

First, create a Universal Data Source that points to Users in every app you want to import users from.

By selecting the Users data model, you instruct integration.app to automatically pick the most appropriate data collection in each external app this Data Source is used for.

Field Mapping

To keep field mapping configuration in one place and let your customers change it if needed, create a Field Mapping.

Select the Users data source you created in the previous step.

You can then define default field mapping that will be used across all applications.
You can leverage our Unified Data Model to pre-populate mappings for standard fields across all applications.

This screenshot shows the simplest possible configuration for "Import Users" field mapping that maps fields from the external app to "id" and "name" fields that your app will receive.

Full Data Import

To perform the initial import of all users from the external app (and full re-imports in the future if you need to), use the "List Data Record" action.

Create a new action that looks like this:

Let's break it down.

Data Source

Select the Users Data Source you created in the first step. You will re-use this data source in the following steps, so it's good to keep it consistent.

Pagination

If an external application has a lot of users, this action will return users page by page. Each response will contain a cursor that you can use to fetch the next page.

To make it work, you need to:

• Add cursor to the input schema of the action.
• Use cursor in the pagination config of the action.

Field Mapping

Select the Import Users field mapping you created in the second step. This will ensure that the data you receive is mapped consistently to your data schema across different blueprints.

Testing

To test this action, apply it to an external app you have an account in, create a connection, and run the sample code in the right panel:

Incremental Data Import

To receive incremental updates from external apps, create a Flow triggered by change events in the Users data source.

The flow will look like this:

It will be triggered by data-record-created, data-record-updated, and data-record-deleted events in the Users data source.
Then it will fetch the full user record (except for deleted users) and send them to your application.

The step that fetches the user record will look like this:

It uses the same Data Source and Field Mapping as in the "Get All Users" action above.

The step that sends users to your app looks like this:

It simply sends the user record to the URL specified by you in the body of the POST request.

Testing

To test this flow, apply it to an external app you have an account in, create a connection, and make a change in one of the users: create, update, or delete it.
You should see a flow run and a request sent to the URL you specified.

Extending Functionality

There are many things you can add to the basic functionality described here:

• To configure requests made to your app (add authentication, etc), see Internal API.
• Use Field Mapping UX to let your users customize how fields are mapped from the external app to your app.
• Internal Data Schemas will let you define custom per-customer field schemas that will be used in the field mapping.
• If you want to not just import data, but send updates back, check out Bi-Directional Sync.

Live Examples

• Continuous Import of Files