When it comes to obtaining complete lists of entities exposed by the API, a distinction needs to be made.

Some entities (i.e. models) can be retrieved all at once with a single API call, while others (i.e. records), are returned by the API in the form of pages. In the latter case, the response will contain the total number of resources in the meta.total_count property:

{
  "data": [...],
  "meta": {
    "total_count": 140
  }
}

The pagination that the API offers is offset-based: this means that you can control the results that are returned with the parameters page[limit] and page[offset]:

• page[limit] is the maximum number of entities to be returned

• page[offset] is the (zero-based) offset of the first entity returned in the collection (always defaults to 0)

Setting a page[limit]=5 and page[offset]=5 will return entities 6 through 10:

curl \
  -H 'Accept: application/json' \
  -H 'Authentication: Bearer <YOUR-API-TOKEN>' \
  https://site-api.datocms.com/items?page[limit]=5&page[offset]=5

Handling pagination with our JavaScript client

Our JavaScript client makes the list() method available for fetching collections of entities. As we have seen, depending on the endpoint, the result may contain all the entities in the collection, or just one page.

// Returns all the models 
const itemTypes = await client.itemTypes.list();
// Returns a single page of records
const items = await client.items.list();

In the case of a paginated endpoint, you can configure the pagination with page.limit and page.offset:

// Returns the first 10 records 
await client.items.list({ page: { limit: 10 } });
// Returns records 6 through 10
await client.items.list({ page: { limit: 5, offset: 5 } });

Paged iterators

In the case of paginated entities, the client also provides the listPagedIterator() method, which allows for fetching all the pages of the collection in a simplified manner, without manually handling offset-based pagination.

You can use this method in an async iteration statement:

// We'll be building up an array of all records using an AsyncIterator
const allRecords = [];
for await (
  const record of client.items.listPagedIterator(
    // You can define any query parameter that the endpoint permits,
    // except for page (refer to the following example for clarification)
    { filter: { type: "article" } }
  )
) {
  allRecords.push(record);
}
console.log(allRecords);

The method listPagedIterator() offers a few options to configure its behavior:

• The concurrency option determines how many API calls can be performed in parallel (up to a maximum of 10). The default setting is 1, implying that the calls are made sequentially, not in parallel.

• The perPage option specifies the size of the pages in the sub-requests that it will carry out in the background.

for await (
  const record of client.items.listPagedIterator(
    // You can define any query parameter that the endpoint permits,
    // except for page
    { filter: { type: "article" } },
    // Pagination options
    { concurrency: 5, perPage: 100 },
  )
) {
  // ...
}

Manually retrieving the total count of a query result

Normally, our listPagedIterator handles pagination for you, but if you need to retrieve the total count of a query, you can use the rawList() command to access the response's meta.total_count property.

In this example, we query records (items) of a certain model type (page) using rawList() with a filter, and then access meta.total_count for the total.

const records = await client.items.rawList({
        filter: {
            type: 'page' // API key (that you gave it) or ID (from its URL)
        },
        page: {
            limit: 0 // We don't need any actual records, just the meta
        }
    })
console.log(records.meta.total_count) // Returns `11`