Skip to content

Sync Your Content Catalog with Compass

Last updatedView as Markdown

Arc XP Compass can only recommend content it knows about. This guide gets your catalog into Compass and keeps it in sync as you publish, update, and unpublish, so recommendations never point at content Compass hasn’t seen or at pages that no longer exist.

How you do this depends on your CMS:

Arc XP CMS: confirm the managed sync

If your content lives in the Arc XP CMS, content ingestion is handled for you by a platform-managed IFX integration, arc_xp_compass. There is no bundle for you to build, install, or configure, and you do not need to create or share a CMS token.

What the integration does:

  • During onboarding, Arc XP loads your existing published stories into Compass.
  • After that, the integration subscribes to Arc XP Composer story events and forwards each story to the Collector content endpoint. Publish-type events (create, first publish, republish, update) upsert the story into the catalog; unpublish and delete events remove it.
  • Only published content is ingested. Stories with no website association, such as drafts, are skipped.
  • Each story’s Arc ID (the Arc XP Native Specification (ANS) _id) is its item_id. If you send item_id anywhere else, such as on forwarded events, use that same Arc ID. See the Item ID Guidance.

To confirm ingestion:

  1. Publish a test story.
  2. After a short delay, check that it appears in a recommendations response for your site_id.
  3. If it does not appear, contact Arc XP Client Support.

Other CMSs: load and stream your catalog

If you use another CMS, you build content sync yourself. It has two parts that work together:

  • A one-time bulk load of your existing back catalog, so there is a corpus to recommend from at launch. Also use it to re-import after a CMS migration, a content model change, or a catalog cleanup.
  • An ongoing sync that forwards every publish, update, and delete from your CMS (or an intermediary service) to Compass within seconds.

Both use the same endpoint, POST /collector/v1/content, which takes a single UpsertContent object per request. The content endpoint has no batch variant: “bulk load” means streaming each item through the single-item endpoint with appropriate pacing, idempotency, and verification.

Before you begin

  • Compass is provisioned for your organization.
  • You have a collector Headless API token. It is a server-side secret: load it from a secret store and never hard-code it or ship it to a browser. See Provision and Manage Compass API Keys.
  • You have the base URL for your organization, https://{org}-config-prod.api.arc-cdn.net. Your Arc XP contact provides the exact host during onboarding.
  • You have an export of your published catalog. Deleted, unpublished, and draft items must not be sent.

1. Settle your item_id scheme

The upsert keys on (site_id, item_id), so a bulk load is the moment you commit to your item_id scheme. The same content item must always produce the same item_id, and the value the bulk load sends must match what the live sync and every forwarded event will send for that same item. There is no retroactive rebind: a mismatch orphans the bulk-loaded record the first time that item updates through the live path.

Settle stability, format, length, and no-PII rules against the Item ID Guidance before you send the first request.

2. Map your export to UpsertContent payloads

Every row in your export maps to one `UpsertContent` payload with action: "publish". At minimum, each payload needs:

Payload fieldWhat it isWhere it comes from in a typical CMS export
actionAlways "publish" for bulk load.Constant.
item_idStable identifier for the content item.The CMS’s primary identifier for the item (for example, an Arc ID).
site_idThe website or property the item belongs to.The site or brand the item is published under.
typeOne of article, video, podcast.The item’s content type.
timestampPublication date, ISO 8601 with explicit offset.The item’s published-at timestamp, not the export time.
titleDisplay title.The item’s headline or title field.
categories, tags, author, is_premium, metadataOptional but recommended.Whatever taxonomy and attribution fields your export carries.

For every content-payload field, including body, transcript, url, and thumbnail_url, see Content endpoint in the Compass API Developer Guide and the POST /collector/v1/content reference.

The endpoint accepts one JSON object per request, sent with Content-Type: application/json. It does not accept CSV, NDJSON, or a JSON array. Those are export formats you may have on disk, but the wire format is always one JSON object per call. Whatever shape your export arrives in, only the parser at the head of the pipeline changes:

  • NDJSON (one JSON object per line). Stream the file line by line; each line is already a request body. No transformation is needed if the producer writes UpsertContent-shaped objects directly.
  • JSON array. Stream-parse with jq -c '.[]' to emit one object per line, then treat the output as NDJSON.
  • CSV. Map each row to an UpsertContent object. List-valued fields (categories, tags) need a delimiter convention in the CSV, typically ; or |, that the client splits into a JSON array before sending.
  • Database dump. Page through the source table and project each row into an UpsertContent object the same way.

3. Send a test payload

Before you start the full load, submit one item by hand. Save the UpsertContent object as article-001.json:

{
"action": "publish",
"item_id": "ARTICLE-001",
"site_id": "acme-news",
"type": "article",
"timestamp": "2026-04-08T09:00:00+00:00",
"title": "Election results, day three",
"categories": ["Politics"],
"tags": ["election-2026", "results"],
"author": "Jane Doe",
"is_premium": false,
"metadata": {}
}

POST it with curl, reading the body from the file with -d @:

Terminal window
curl -sS -X POST \
"https://{org}-config-prod.api.arc-cdn.net/collector/v1/content" \
-H "Content-Type: application/json" \
-H "X-API-Key: $COLLECTOR_API_TOKEN" \
-d @article-001.json

A successful submission returns:

HTTP/1.1 202 Accepted
Content-Length: 0

For one-off smoke tests, pass the body inline with -d '{...}' instead of -d @file.

4. Run the bulk load

Repeat the parse, build, POST, and record loop for every record in your export. Because the endpoint takes one item per request, “bulk” is a client-side concern: how many items per second you push, and how many you push in parallel.

  • Pace on aggregate throughput, not per-request. Whether you run the load as one process or many parallel workers, Compass sees the sum of your request rate, and that aggregate is the only number that matters. Per-request 202s confirm the payload was queued; they do not confirm the downstream pipeline is keeping up. Start conservative (a few tens of requests per second across all workers is a reasonable opener for an initial backfill), watch for 5xx responses and growing latency, and ramp only when you have evidence of headroom.
  • Use modest concurrency. A small number of concurrent workers (for example, 4–16) per export job is usually enough to saturate ingest capacity without overwhelming downstream processing. Increase only if you observe headroom; back off on 5xx.
  • Send in any order. The upsert keys on (site_id, item_id), not on send order.
  • Plan for resumability. Track which item_ids returned 202 so a partially completed run can resume without re-sending the whole catalog. Resending is safe (see Idempotency), but skipping already-sent items shortens recovery time.
  • Finish before launch. Run backfills before any user-facing recommendations are exposed. See the Pre-launch phase of the Compass Onboarding Checklist.

Handle errors during the load

The content endpoint follows the shared Collector status contract in Error and retry behavior: retry 5xx and 429 with exponential backoff, stop and fix credentials on 401/403, and fix the row instead of retrying on 422. A 422 on one item does not stop the rest, because each request is independent.

One bulk-load nuance: watch your 422 rate across the load. A systematic spike, rather than the odd row, almost always means a mapping bug in your export, so pause and fix the transform before you finish the run.

Idempotency

A publish payload is an upsert keyed on (site_id, item_id). Reposting the same payload overwrites the stored item with the same content. There is no duplicate, no version bump visible to the caller, and no downstream double-counting. This means:

  • Retrying a request after a transport error is safe.
  • Re-running a partial bulk load against the same export is safe.
  • Re-running the bulk load after editing your export (for example, adding a missing categories value) is the supported way to correct catalog metadata.

This guarantee holds only while the item_id for a given content item stays the same between runs. Re-running with a different item_id orphans the original record. Do not send action: "delete" during a bulk load of published content. For the full republish and soft-delete semantics, see the Item ID Guidance.

5. Wire the ongoing sync

After the back catalog is loaded, configure your CMS to send POST /collector/v1/content requests whenever content is published, updated, or deleted. Re-running the bulk load is not a substitute for the live sync. At minimum, your CMS (or an intermediary service) must:

  1. Listen for publish, update, and delete events in your CMS.
  2. Transform each event into the appropriate action: "publish" or action: "delete" payload described under Content endpoint.
  3. POST the payload to /collector/v1/content with your collector token in the X-API-Key header.
  4. Retry on transport-level failures (connection errors, 5xx responses). Do not retry on 202 Accepted. That means the payload was accepted for asynchronous processing.

Send the same item_id for each item that the bulk load sent. The goal is that every publish, update, and unpublish in your CMS reaches the content endpoint, ideally within seconds.

6. Verify the catalog

The endpoint returns 202 Accepted as soon as the payload is durably committed to the ingestion pipeline, not when the item is fully visible to the model. To verify that a load completed:

  1. Reconcile acknowledgements. A successful 202 per item, with a per-item count matching the export’s row count, is the canonical “done” signal for the ingestion step.
  2. Spot-check through the Recommendations API. After the load completes, call GET /experiences/recommend/v1/recommendations with a test user_id against the same site_id. A non-empty recommendations array confirms items have reached the model. Empty arrays after a known-good bulk load usually indicate a site_id mismatch between ingestion and the read call.
  3. Round-trip a sample of item_ids. Pick a handful of IDs from your export, confirm they show up in recommendations responses for diverse synthetic users, and confirm each one resolves in your CMS.
  4. Review error rates in your client. A bulk load with even a small percentage of 422 errors usually points to a systematic mapping bug in the export. Investigate before you declare the load done.

If recommendations still come back empty, see Troubleshoot Compass Recommendations.

When not to use the content endpoint

  • User-behavior events. Use POST /collector/v1/events or POST /collector/v1/events/batch. See Forward Events from a CDP or Server and the Compass Batch API guide.
  • Tombstoning items deleted in the source CMS through a bulk load. Bulk load is insert-or-upsert only. To remove items, send action: "delete" payloads explicitly.

See also