Data collections allow working with data in an external application using a consistent API.

Spec

The data collection specification defines the structure and capabilities of a data collection.

It is located in the root of the data collection directory (data/<collection-key>/spec.yml).

Here's an example:

contacts:
  name: Contacts
  fieldsSchema:
    type: object
    properties:
      id:
        type: string
        readOnly: true
      email:
        type: string
      firstName:
        type: string
      lastName:
        type: string
      fullName:
        type: string
        writeOnly: true
  methods:
    list:
      filterFields:
        - email
    create:
      requiredFields:
        - email
    update:
      excludedFields:
        - id
    delete: {}
    findById: {}
    match:
      fields:
        - email
    search: {}
  events:
    created:
      implementationType: webhook
    updated:
      implementationType: pull-latest-records
    deleted:
      implementationType: custom-pull

Properties

Parameters

Data collection parameters are used to configure the data collection implementation.
They are passed to execution context of all data collection functions as parameters argument.

You should only add parameters to the data collection when you cannot use function-level parameters.

If collection-level and function-level parameters overlap, function-level parameters take precedence.

Fields Schema

This Data Schema defines fields of the records in this data collection.

It is recommended that all the fields that could participate in any operation of this collection be present in the fields schema. If some fields are only available in read or write operations, you can mark them as readOnly or writeOnly.

Furthermore, you can configure which fields are available for which operations (i.e. create or update) in the corresponding operation configuration.

Custom Fields

If a data collection supports custom fields that can be configured on a per-connection basis, you can implement a customFields function that returns the custom fields schema.\

The custom fields schema will be merged with fieldsSchema when returning the collection's specification.

Methods

You can configure the following methods for the data collection:

Data Collection Functions

These are non-method functions that affect the data collection functionality.

When you want to transform fields that are coming from the API before returning them from the connector, or transform fields that come from the client before sending them to the API, you can use fieldsFromApi and fieldsToApi functions respectively.

These functions will be automatically applied to fields for all method and event implementations.

fieldsToApi will be automatically applied after method is executed and before sending the request to the API.
fieldsFromApi will be automatically applied after the method is executed and before the result is returned to the client.

This could be useful if:

• Fields in the API are inconsistent between endpoints.
• Field structures in the API are too complex and you want to simplify them.

Inheritance (Mixins)

When multiple data collections share the implementation of some of their functions, you can use mixins.

Example:

# spec.yml
name: Users
extends:
  - key: _webhooks
    parameters:
      objectKey: user

If a collection uses a mixin, all the functions (and methods) from the mixin will be used as the collection's functions unless overridden.

If a mixin collection has parameters, each collection that uses the mixin can provide different parameter values.

Data Collection Implementation Workflow

If you are an AI agent, use this to populate your ToDo list when building a data collection. Make sure to execute each step in order.

• Give this data collection a unique name. It should be plural form of the name of the objects in the data collection, i.e. "Users" or "Meeting Notes".
  • If data collection with the same name already exists - determine if it's the same data collection. If yes - update it instead of creating a new one. If not - give a more specific name to one or both of the data collections.
• Data collection key should be a kebab case of the name (i.e. meeting-notes).
• Look through list of mixin data collections (their keys start with _ by convention) and determine if this collection should extend any of them. If so - populate teh extends section of the data collection.
• Implement methods.
  • Determine which methods does data collection support and add them to the specification
  • Implement each method following its implementation workflow.
  • If methods availability or configuration is conditional (differs from connection to connection) - implement dynamic specification for the data collection.
  • If all the methods use the same parameter - remove it from the methods and add to the data collection specification directly.
• Implement fields schema.
  • Fields schema should include all the fields that can be read or written by any of the methods.
  • If different methods use conflicting field schemas, pick the simples / most human-readable option and use it in the collection fields schema.
    • Use fieldsFromApi and fieldsToApi to transform fields from/to API.
      • If there is no way to do it in a consistent way across all methods, implement methods that require transformation as javascript and apply transformations right in the method implementation.
  • If data collection supports custom fields or the list of fields changes from one connection to another, create customFieldsSchema function implementation.
    • If custom fields are not incremental (i.e. you need to both add and remove fields from default fields schema), implement this with dynamic data collection specification.
  • Mark fields that can only be used in read methods (list/search/match/findById) but not write methods (create/update) as readOnly.
    • If you used fields or excludedFields in some of the methods to achieve the same - remove those configurations from corresponding methods. Only use fields or excludedFields if you cannot achieve the same with readOnly.
  • Mark fields that can only be used in write methods, but not in read methods as writeOnly.