Binary Data and Files

Engine API works with data in JSON format.

When you need to work with binary data (uploading or downloading files), the data is stored in a separate storage and a URI to access it is provided in JSON.

Files Endpoint

For convenience, integration engine provides a special /files endpoint that accepts binary files and returns a pre-signed URL to access them.

You can use it to:

• Download files from external apps: you can use the /files endpoint in connector code or your custom code, then return the URL to access them in the output.
• Upload files to external apps: you can use the /files endpoint in your code and pass the URL to the input of the code running in the engine.

Expiration

Files are stored in the engine storage and are available for a limited time. In our cloud version, the files will be available for 1 hour.
In self-hosted version, this time may vary based on how your storage is configured and can last up to 7 days.

Custom Code Example

Here is an example of a custom code that can run in the engine and download a large file from an external app:

import axios from 'axios'

export default async function myCustomFunction({externalApiClient, engineApiClient}) {
    const fileResponse = await externalApiClient.makeApiRequest({
        path,
        method: 'get',
        responseType: 'stream', // We recommend using streams to work with large files to avoid memory issues
        returnFullResponse: true, // Return full response to get headers and pass them on to the next request
    })

    const response = await engineApiClient.post('/files',
        fileResponse.data,
        {
            headers: {
                'Content-Length': fileResponse.headers['content-length'],
                'Content-Type': fileResponse.headers['content-type'],
            },
            // Without this, we load the whole file into memory (which can be several GB).
            // https://github.com/axios/axios/issues/1045
            maxRedirects: 0,
        }
    )

    return {
        fileUri: response.downloadUri,
    }
}