# How to work with WebSked Collections

Source: https://dev.arcxp.com/websked/developer-docs/how-to-work-with-websked-collections/

This document walks you through populating WebSked Collections dynamically and querying for items in the Content API that match a simple tag query or a section query.

For example, if you want to render 10 items from a Collection, but the Collection has been manually populated with only four published items, dynamic Collections lets you backfill those remaining six slots based on a story tag or a section.

WebSked deduplicates content between your backfill and your Collection, which ensures the same item never appears in your backfill if you already manually added it to your Collection.

## Workflow

Follow these steps to take advantage of dynamic Collections:

1. [Creating a Collection](https://docs.arcxp.com/en/products/websked/managing-websked-collections.html#UUID-3d793056-d890-dab0-c9c2-9e3830868ae0%5FUUID-bb37faed-0a96-9c8b-d1d1-423e5dfee39e) in WebSked or through the Collections API.
2. [Add a backfill query](#setting-the-backfill-query) for either a list of tags or a list of sections to the Collection.

When the Collection is requested through the Content API, the Content API:

1. Checks if there are enough items in the Collection to fulfill the request and return those items.
2. If slots remain, the Content API looks at the query attached to the Collection and queries for matching items.
3. The Content API then appends the query results to the original collection results, de-duplicates any items, and returns that response.

## Setting the backfill query

Most organizations can set the backfill query in WebSked, but you can use the Collections API as well. The specified tags or sections that you use to backfill a Collection are saved in the collection’s Arc XP Native Specification (ANS) object under a trait called `dynamic_items`. The following code sample provides an example of a Collection with `dynamic_items`:

```
{
  "_id": "ZYXWVUTSRQPONMLKJIHGFEDCBA",
  "type": "collection",
  "version": "0.10.6",
  "created_date": "2017-06-24T09:50:50.52Z",
  "last_updated_date": "2017-06-24T09:50:50.52Z",
  "canonical_website": "washpost",
  "content_aliases": [ "man-dog-bite-collection", "dog-collection-1" ],
  "dynamic_items": {
    "type": "sections",
    "ids": ["/politics"]
  },
  "credits": {
    "by": [
      {
        "name": "John Q. Reporter",
        "byline": "John Q. Reporter",
        "org": "The Washington Post",
        "type": "author",
        "version": "0.10.6"
      }
    ]
  },
  "language": "en",
  "location": "Washington, D.C.",
  "canonical_url": "/news/dogs/man-bites-dog",
  "short_url": "http://wapo.st/1Crp6bY",
  "headlines": {
    "basic": "The Man Bites Dog collection",
    "twitter": "Lots of Items about Man Biting Dog"
  ...
  ...
}
```

The `dynamic_items` trait has two allowed types: sections and tags. The `dynamic_items.ids` property takes a list of values that you want the query to match by. The previous example backfills your Collection with stories from the `/politics` section.

The example backfills with stories that are tagged with either `dogs` or `cats`:

```
    ...
     "dynamic_items": {
        "type": "tags",
        "ids": ["dogs", "cats"]
      },
    ...
```

Most dynamic Collections are configured through WebSked. See [Collection API](https://dev.arcxp.com/api/websked/collections-api/).

The next section explains more about using tags and sections to populate the backfill for more than one website.

## Querying the Content API for Collections

Querying the Content API for a dynamic Collection is no different than a static Collection. By default, if the `dynamic_items` trait of a collection is filled with a query, Content API backfills the Collection with queried items. These items are denoted in the response by `additional_properties.backfill: true`.

Two endpoints in the Content API can return a Collection:

- `/content/v4/collections`
- `/content/v4`.

The `/content/v4` endpoint does not support dynamic backfill and can return only the first 20 static items of a Collection. If you want to use dynamic backfill, you must use the `/content/v4/collections` endpoint, which is better optimized for returning Collections.

The `/content/v4/collections` endpoint returns 20 items at a time and allows pagination through the rest of the possible static or dynamic items. Arc XP strongly encourages using the `/content/v4/collections` endpoint for all collections requests.

### Endpoint comparison

The following table shows the features and limitations of each of the Collections endpoints.

| Feature                    | /content/v4/collections/ | /content/v4/ |
| -------------------------- | ------------------------ | ------------ |
| Returns denormalized items | YES                      | YES          |
| Dynamic Collections        | YES                      | NO           |
| Pagination                 | YES                      | NO           |
| Per page limit             | 20                       | 20           |
| Total items limit          | 500                      | 20           |
| Backfilled items limit     | 100                      | n/a          |

### ElasticSearch query

When the Content API receives a request for a collection with `dynamic_items`, it turns the `dynamic_items` into an ElasticSearch query. The system uses the ElasticSearch query to find matching results to add to your collection responses.

For dynamic backfilling through tags, the Content API queries against `taxonomy.tags.slug` for each of the tags in the list. For sections, the Content API queries against `taxonomy.sections._id` for each of the sections in the list. In both options, the default sort is `display_date:desc`.

### Limit

While a collection can include up to 500 static items, dynamic backfill works only up to 100 total items. For example, if your collection has 60 static items, the Collection API backfills 40 items to get to a total of 100 items. If your collection has 200 static items, the Collection API does not backfill any items.

### Published versus unpublished content

The `/content/v4/collections` endpoint provides two query parameters for accessing published or unpublished content. By default, the `/content/v4/collections` endpoint returns only published collections and the published stories within. However, you can change these query parameters to get the exact combination of published and unpublished content you want:

| ?published     | ?include\_unpublished | Response                                                                 |
| -------------- | --------------------- | ------------------------------------------------------------------------ |
| TRUE (default) | FALSE (default)       | Published Collection Published Content within Collection                 |
| TRUE           | TRUE                  | Published Collection Published and Unpublished Content within Collection |
| FALSE          | n/a                   | Unpublished Collection Draft Content within Collection                   |

## How to fetch all Collections

The Collections API allows you to retrieve your organization’s configured collections.

By default, the API returns up to 20 records per request, but you can safely increase this limit to 100 records in a single call.

If you need to retrieve more than 100 records, you must paginate through the results.

The following Python script demonstrates how to accomplish this using pagination.

```python
import requests
from urllib.parse import urljoin, urlparse, parse_qsl, urlencode
from datetime import datetime

def _to_epoch_ms(value):
    if value is None:
        return None
    if isinstance(value, (int, float)):
        return int(value)
    s = str(value).strip()
    try:
        return int(float(s))
    except Exception:
        # try ISO parse (handles "Z")
        try:
            if s.endswith('Z'):
                s = s[:-1] + '+00:00'
            dt = datetime.fromisoformat(s)
            return int(dt.timestamp() * 1000)
        except Exception:
            return None

def fetch_collections(base_url,
                      initial_query='',
                      total=100,
                      page_size=20,
                      headers=None,
                      collections_path='collections'):
    """
    Fetch up to `total` collections from the collections endpoint in pages of `page_size`.
    Pagination params used: limit, startAt, lastUpdatedDate (ms since epoch).
    Returns list of collection objects (trimmed to `total`).
    """

    # NOTE: page_size is capped at 100.
    # enforce page size limits (max 100, min 20)
    try:
        page_size = int(page_size)
    except Exception:
        page_size = 20
    if page_size < 20:
        page_size = 20
    MAX_PAGE_SIZE = 100
    if page_size > MAX_PAGE_SIZE:
        page_size = MAX_PAGE_SIZE

    # prepare base URL
    base = base_url.rstrip('/') + '/'
    endpoint = urljoin(base, collections_path)

    # parse any initial query string into params
    parsed = dict(parse_qsl(initial_query.lstrip('?')))

    parsed['limit'] = str(page_size)

    results = []
    last_coll = None

    while len(results) < total:
        # add pagination from last_coll if present
        params = parsed.copy()
        if last_coll:
            coll_id = last_coll.get('id')
            if coll_id:
                params['startAt'] = str(coll_id)
            last_updated = _to_epoch_ms(last_coll.get('last_updated_date') or last_coll.get('lastUpdatedDate'))
            if last_updated:
                params['lastUpdatedDate'] = str(last_updated)

        resp = requests.get(endpoint, params=params, headers=headers or {})
        resp.raise_for_status()
        payload = resp.json()

        if isinstance(payload, list):
            items = payload
        elif isinstance(payload, dict) and isinstance(payload.get('data'), list):
            items = payload.get('data')
        else:
            items = []

        if not items:
            break

        results.extend(items)
        last_coll = items[-1]

        # stop if last page smaller than page_size
        if len(items) < page_size:
            break

    return results[:total]

if __name__ == '__main__':
    import os
    import sys

    # Example usage:
    # python paginated_get_collections.py
    #   https://api.{org}.arcpublishing.com/websked/collections/v1 '?name=foo' 5000 100
    if len(sys.argv) < 2:
        print("Usage: python paginated_get_collections.py <base_url> [initial_query] [total] [page_size]")
        sys.exit(1)

    base_url = sys.argv[1]
    initial_query = sys.argv[2] if len(sys.argv) > 2 else ''
    total = int(sys.argv[3]) if len(sys.argv) > 3 else 100
    page_size = int(sys.argv[4]) if len(sys.argv) > 4 else 20

    # get AUTH token from env (e.g. export API_TOKEN=...)
    token = os.environ.get('API_TOKEN')
    headers = {'Authorization': f'Bearer {token}'} if token else None

    cols = fetch_collections(base_url, initial_query, total, page_size, headers)
    print(f"Fetched {len(cols)} collections")
```
