Data Collections
Data collections allow working with data in an external application using a consistent API.
Spec
The data collection specification defines the structure and capabilities of a data collection.
It is located in the root of the data collection directory (data/<collection-key>/spec.yml
).
Here's an example:
contacts:
name: Contacts
fieldsSchema:
type: object
properties:
id:
type: string
readOnly: true
email:
type: string
firstName:
type: string
lastName:
type: string
fullName:
type: string
writeOnly: true
methods:
list:
filterFields:
- email
create:
requiredFields:
- email
update:
excludedFields:
- id
delete: {}
findById: {}
match:
fields:
- email
search: {}
events:
created:
implementationType: webhook
updated:
implementationType: pull-latest-records
deleted:
implementationType: custom-pull
Properties
Property | Type | Description |
---|---|---|
name | string | Display name for the collection |
parametersSchema | object | Data Schema for collection-level parameters |
fieldsSchema | object | Data Schema defining the structure of records in the collection |
methods | object | Method implementations. See below for details. |
events | object | Event implementations. See below for details. |
Parameters
Data collection parameters are used to configure the data collection implementation.
They are passed to execution context of all data collection functions as parameters
argument.
You should only add parameters to the data collection when you cannot use function-level parameters.
If collection-level and function-level parameters overlap, function-level parameters take precedence.
Fields Schema
This Data Schema defines fields of the records in this data collection.
It is recommended that all the fields that could participate in any operation of this collection be present in the fields schema. If some fields are only available in read or write operations, you can mark them as readOnly
or writeOnly
.
Furthermore, you can configure which fields are available for which operations (i.e. create
or update
) in the corresponding operation configuration.
Custom Fields
If a data collection supports custom fields that can be configured on a per-connection basis, you can implement a customFields
function that returns the custom fields schema.\
The custom fields schema will be merged with fieldsSchema
when returning the collection's specification.
Methods
You can configure the following methods for the data collection:
Method | Description |
---|---|
list | Read all the records from the collection (with optional filters). |
findById | Get one record from the collection using its id. |
match | Find a single matching record in a data collection using provided fields (e.g., by email). |
search | Find a list of records using a string query. Supports type-ahead search whenever possible. |
create | Create one record in a data collection. |
update | Update a record in a data collection using its id. |
delete | Delete a record with a given id. |
Data Collection Functions
These are non-method functions that affect the data collection functionality.
Function | Description |
---|---|
recordFromFields | Build Data Records from their fields |
spec | Dynamically generate the collection specification based on API capabilities |
customFields | Dynamically generate custom fields schema based on API configuration |
fieldsFromApi | Transform API response fields to match your fieldsSchema format |
fieldsToApi | Transform fieldsSchema format to API format before sending requests |
When you want to transform fields that are coming from the API before returning them from the connector, or transform fields that come from the client before sending them to the API, you can use fieldsFromApi
and fieldsToApi
functions respectively.
These functions will be automatically applied to fields for all method and event implementations.
fieldsToApi
will be automatically applied after method is executed and before sending the request to the API.
fieldsFromApi
will be automatically applied after the method is executed and before the result is returned to the client.
This could be useful if:
- Fields in the API are inconsistent between endpoints.
- Field structures in the API are too complex and you want to simplify them.
Inheritance (Mixins)
When multiple data collections share the implementation of some of their functions, you can use mixins.
Example:
# spec.yml
name: Users
extends:
- key: _webhooks
parameters:
objectKey: user
If a collection uses a mixin, all the functions (and methods) from the mixin will be used as the collection's functions unless overridden.
If a mixin collection has parameters, each collection that uses the mixin can provide different parameter values.
Data Collection Implementation Workflow
If you are an AI agent, use this to populate your ToDo list when building a data collection. Make sure to execute each step in order.
- Give this data collection a unique name. It should be plural form of the name of the objects in the data collection, i.e. "Users" or "Meeting Notes".
- If data collection with the same name already exists - determine if it's the same data collection. If yes - update it instead of creating a new one. If not - give a more specific name to one or both of the data collections.
- Data collection key should be a kebab case of the name (i.e.
meeting-notes
). - Look through list of mixin data collections (their keys start with
_
by convention) and determine if this collection should extend any of them. If so - populate tehextends
section of the data collection. - Implement methods.
- Determine which methods does data collection support and add them to the specification
- Implement each method following its implementation workflow.
- If methods availability or configuration is conditional (differs from connection to connection) - implement dynamic specification for the data collection.
- If all the methods use the same parameter - remove it from the methods and add to the data collection specification directly.
- Implement fields schema.
- Fields schema should include all the fields that can be read or written by any of the methods.
- If different methods use conflicting field schemas, pick the simples / most human-readable option and use it in the collection fields schema.
- Use
fieldsFromApi
andfieldsToApi
to transform fields from/to API.- If there is no way to do it in a consistent way across all methods, implement methods that require transformation as javascript and apply transformations right in the method implementation.
- Use
- If data collection supports custom fields or the list of fields changes from one connection to another, create
customFieldsSchema
function implementation.- If custom fields are not incremental (i.e. you need to both add and remove fields from default fields schema), implement this with dynamic data collection specification.
- Mark fields that can only be used in read methods (list/search/match/findById) but not write methods (create/update) as
readOnly
.- If you used
fields
orexcludedFields
in some of the methods to achieve the same - remove those configurations from corresponding methods. Only usefields
orexcludedFields
if you cannot achieve the same withreadOnly
.
- If you used
- Mark fields that can only be used in write methods, but not in read methods as
writeOnly
.
Updated 10 days ago