Custom Pull

Custom Pull Strategy provides a customized approach for retrieving lists of created, updated, or deleted records.

This strategy consists of two methods: subscribe and pull.

How It Works

When a subscription is created, a state is generated and stored. It allows pulling events starting from the subscription time.
When events are pulled, the state is used to generate events since the state was created, as well as the new state for the next pull.

Configuration Example

# spec.yml
name: Collection Name
events:
  created:
    implementationType: custom-pull
    methods:
      subscribe:
        implementationType: javascript
        filePath: events/custom-subscribe.js
      pull:
        implementationType: javascript
        filePath: events/custom-pull.js
  updated:
    implementationType: custom-pull
    methods:
      subscribe:
        implementationType: javascript
        filePath: events/custom-subscribe.js
      pull:
        implementationType: javascript
        filePath: events/custom-pull.js
  deleted:
    implementationType: custom-pull
    methods:
      subscribe:
        implementationType: javascript
        filePath: events/custom-subscribe.js
      pull:
        implementationType: javascript
        filePath: events/custom-pull.js

Functions

File: <event-type>-subscribe.<ext>, for example created-subscribe.js

The subscribe method initializes the subscription to events and retrieves a unique identifier for future pulls.

Arguments:

parameters – object, parameters (matching parametersSchema) of this particular data collection.

Result:

state – object, subscription state. This state will be saved and passed as state to the pull function.

Example:

// Example: Subscribe to delta changes and return the deltaLink.
export default async function subscribe({ apiClient, parameters }) {
  let cursor = false
  const path = `${parameters.path}/delta`

  while (true) {
    const response = await apiClient.get(`${cursor ? cursor : path}`)

    if (response.deltaLink) {
      break
    } else {
      cursor = response.nextPageLink
    }
  }

  return { state: { deltaLink: response.deltaLink } }
}

pull

File: <event-type>-pull.<ext>, for example created-pull.js

The pull method fetches records based on the subscription and stores them in the state.
It also generates events for each created, updated, or deleted record.

Arguments:

state – object, state returned from subscribe function.
parameters – object, parameters (matching parametersSchema) of this particular data collection.

Result:

events – list of objects representing events with type and record fields.
state – updated state information, including the new information for subsequent pulls.

Example:

// Example: Pull records using the deltaLink and generate events.
export default async function pull({ apiClient, state, parameters }) {
  const events = []
  let cursor = false

  while (true) {
    const response = await apiClient.get(
      `${cursor ? cursor : state?.deltaLink}`
    )

    response.value.forEach((record) => {
      events.push({
        type: "created",
        record: record,
      })
    })

    if (response.deltaLink) {
      break
    } else {
      cursor = response.nextPageLink
    }
  }

  return {
    events,
    state: {
      deltaLink: response.deltaLink,
    },
  }
}