Sync Your Content Catalog with Compass
Arc XP Compass can only recommend content it knows about. This guide gets your catalog into Compass and keeps it in sync as you publish, update, and unpublish, so recommendations never point at content Compass hasn’t seen or at pages that no longer exist.
How you do this depends on your CMS:
- Arc XP CMS customers: content sync is platform-managed. Go to Arc XP CMS: confirm the managed sync.
- Other CMSs: you load your back catalog once, then stream ongoing changes to the Collector content endpoint. Go to Other CMSs: load and stream your catalog.
Arc XP CMS: confirm the managed sync
If your content lives in the Arc XP CMS, content ingestion is handled for you by
a platform-managed IFX integration, arc_xp_compass. There is no bundle for you
to build, install, or configure, and you do not need to create or share a CMS
token.
What the integration does:
- During onboarding, Arc XP loads your existing published stories into Compass.
- After that, the integration subscribes to Arc XP Composer story events and forwards each story to the Collector content endpoint. Publish-type events (create, first publish, republish, update) upsert the story into the catalog; unpublish and delete events remove it.
- Only published content is ingested. Stories with no website association, such as drafts, are skipped.
- Each story’s Arc ID (the Arc XP Native Specification (ANS)
_id) is itsitem_id. If you senditem_idanywhere else, such as on forwarded events, use that same Arc ID. See the Item ID Guidance.
To confirm ingestion:
- Publish a test story.
- After a short delay, check that it appears in a recommendations response for
your
site_id. - If it does not appear, contact Arc XP Client Support.
Other CMSs: load and stream your catalog
If you use another CMS, you build content sync yourself. It has two parts that work together:
- A one-time bulk load of your existing back catalog, so there is a corpus to recommend from at launch. Also use it to re-import after a CMS migration, a content model change, or a catalog cleanup.
- An ongoing sync that forwards every publish, update, and delete from your CMS (or an intermediary service) to Compass within seconds.
Both use the same endpoint, POST /collector/v1/content, which takes a single
UpsertContent object per request. The content endpoint has no batch
variant: “bulk load” means streaming each item through the single-item endpoint
with appropriate pacing, idempotency, and verification.
Before you begin
- Compass is provisioned for your organization.
- You have a collector Headless API token. It is a server-side secret: load it from a secret store and never hard-code it or ship it to a browser. See Provision and Manage Compass API Keys.
- You have the base URL for your organization,
https://{org}-config-prod.api.arc-cdn.net. Your Arc XP contact provides the exact host during onboarding. - You have an export of your published catalog. Deleted, unpublished, and draft items must not be sent.
1. Settle your item_id scheme
The upsert keys on (site_id, item_id), so a bulk load is the moment you commit
to your item_id scheme. The same content item must always produce the same
item_id, and the value the bulk load sends must match what the live sync and
every forwarded event will send for that same item. There is no retroactive
rebind: a mismatch orphans the bulk-loaded record the first time that item
updates through the live path.
Settle stability, format, length, and no-PII rules against the Item ID Guidance before you send the first request.
2. Map your export to UpsertContent payloads
Every row in your export maps to one `UpsertContent` payload with action: "publish". At minimum, each payload needs:
| Payload field | What it is | Where it comes from in a typical CMS export |
|---|---|---|
action | Always "publish" for bulk load. | Constant. |
item_id | Stable identifier for the content item. | The CMS’s primary identifier for the item (for example, an Arc ID). |
site_id | The website or property the item belongs to. | The site or brand the item is published under. |
type | One of article, video, podcast. | The item’s content type. |
timestamp | Publication date, ISO 8601 with explicit offset. | The item’s published-at timestamp, not the export time. |
title | Display title. | The item’s headline or title field. |
categories, tags, author, is_premium, metadata | Optional but recommended. | Whatever taxonomy and attribution fields your export carries. |
For every content-payload field, including body, transcript, url, and
thumbnail_url, see Content endpoint
in the Compass API Developer Guide and the POST /collector/v1/content reference.
The endpoint accepts one JSON object per request, sent with
Content-Type: application/json. It does not accept CSV, NDJSON, or a JSON
array. Those are export formats you may have on disk, but the wire format is
always one JSON object per call. Whatever shape your export arrives in, only the
parser at the head of the pipeline changes:
- NDJSON (one JSON object per line). Stream the file line by line; each
line is already a request body. No transformation is needed if the producer
writes
UpsertContent-shaped objects directly. - JSON array. Stream-parse with
jq -c '.[]'to emit one object per line, then treat the output as NDJSON. - CSV. Map each row to an
UpsertContentobject. List-valued fields (categories,tags) need a delimiter convention in the CSV, typically;or|, that the client splits into a JSON array before sending. - Database dump. Page through the source table and project each row into
an
UpsertContentobject the same way.
3. Send a test payload
Before you start the full load, submit one item by hand. Save the
UpsertContent object as article-001.json:
{ "action": "publish", "item_id": "ARTICLE-001", "site_id": "acme-news", "type": "article", "timestamp": "2026-04-08T09:00:00+00:00", "title": "Election results, day three", "categories": ["Politics"], "tags": ["election-2026", "results"], "author": "Jane Doe", "is_premium": false, "metadata": {}}POST it with curl, reading the body from the file with -d @:
curl -sS -X POST \ "https://{org}-config-prod.api.arc-cdn.net/collector/v1/content" \ -H "Content-Type: application/json" \ -H "X-API-Key: $COLLECTOR_API_TOKEN" \ -d @article-001.jsonA successful submission returns:
HTTP/1.1 202 AcceptedContent-Length: 0For one-off smoke tests, pass the body inline with -d '{...}' instead of
-d @file.
4. Run the bulk load
Repeat the parse, build, POST, and record loop for every record in your export. Because the endpoint takes one item per request, “bulk” is a client-side concern: how many items per second you push, and how many you push in parallel.
- Pace on aggregate throughput, not per-request. Whether you run the load
as one process or many parallel workers, Compass sees the sum of your request
rate, and that aggregate is the only number that matters. Per-request
202s confirm the payload was queued; they do not confirm the downstream pipeline is keeping up. Start conservative (a few tens of requests per second across all workers is a reasonable opener for an initial backfill), watch for5xxresponses and growing latency, and ramp only when you have evidence of headroom. - Use modest concurrency. A small number of concurrent workers (for example,
4–16) per export job is usually enough to saturate ingest capacity without
overwhelming downstream processing. Increase only if you observe headroom; back
off on
5xx. - Send in any order. The upsert keys on
(site_id, item_id), not on send order. - Plan for resumability. Track which
item_ids returned202so a partially completed run can resume without re-sending the whole catalog. Resending is safe (see Idempotency), but skipping already-sent items shortens recovery time. - Finish before launch. Run backfills before any user-facing recommendations are exposed. See the Pre-launch phase of the Compass Onboarding Checklist.
Handle errors during the load
The content endpoint follows the shared Collector status contract in
Error and retry behavior:
retry 5xx and 429 with exponential backoff, stop and fix credentials on
401/403, and fix the row instead of retrying on 422. A 422 on one item
does not stop the rest, because each request is independent.
One bulk-load nuance: watch your 422 rate across the load. A systematic spike,
rather than the odd row, almost always means a mapping bug in your export, so
pause and fix the transform before you finish the run.
Idempotency
A publish payload is an upsert keyed on (site_id, item_id). Reposting the
same payload overwrites the stored item with the same content. There is no
duplicate, no version bump visible to the caller, and no downstream
double-counting. This means:
- Retrying a request after a transport error is safe.
- Re-running a partial bulk load against the same export is safe.
- Re-running the bulk load after editing your export (for example, adding a
missing
categoriesvalue) is the supported way to correct catalog metadata.
This guarantee holds only while the item_id for a given content item stays the
same between runs. Re-running with a different item_id orphans the original
record. Do not send action: "delete" during a bulk load of published
content. For the full republish and soft-delete semantics, see the Item ID Guidance.
5. Wire the ongoing sync
After the back catalog is loaded, configure your CMS to send
POST /collector/v1/content requests whenever content is published, updated, or
deleted. Re-running the bulk load is not a substitute for the live sync. At
minimum, your CMS (or an intermediary service) must:
- Listen for publish, update, and delete events in your CMS.
- Transform each event into the appropriate
action: "publish"oraction: "delete"payload described under Content endpoint. POSTthe payload to/collector/v1/contentwith your collector token in theX-API-Keyheader.- Retry on transport-level failures (connection errors,
5xxresponses). Do not retry on202 Accepted. That means the payload was accepted for asynchronous processing.
Send the same item_id for each item that the bulk load sent. The goal is that
every publish, update, and unpublish in your CMS reaches the content endpoint,
ideally within seconds.
6. Verify the catalog
The endpoint returns 202 Accepted as soon as the payload is durably committed
to the ingestion pipeline, not when the item is fully visible to the model. To
verify that a load completed:
- Reconcile acknowledgements. A successful
202per item, with a per-item count matching the export’s row count, is the canonical “done” signal for the ingestion step. - Spot-check through the Recommendations API. After the load completes, call
GET /experiences/recommend/v1/recommendationswith a testuser_idagainst the samesite_id. A non-emptyrecommendationsarray confirms items have reached the model. Empty arrays after a known-good bulk load usually indicate asite_idmismatch between ingestion and the read call. - Round-trip a sample of
item_ids. Pick a handful of IDs from your export, confirm they show up inrecommendationsresponses for diverse synthetic users, and confirm each one resolves in your CMS. - Review error rates in your client. A bulk load with even a small
percentage of
422errors usually points to a systematic mapping bug in the export. Investigate before you declare the load done.
If recommendations still come back empty, see Troubleshoot Compass Recommendations.
When not to use the content endpoint
- User-behavior events. Use
POST /collector/v1/eventsorPOST /collector/v1/events/batch. See Forward Events from a CDP or Server and the Compass Batch API guide. - Tombstoning items deleted in the source CMS through a bulk load. Bulk load is
insert-or-upsert only. To remove items, send
action: "delete"payloads explicitly.
See also
- Item ID Guidance — choosing a stable
item_id. - Content endpoint — payload fields, body and transcript rules, and deletes.
- Provision and Manage Compass API Keys — the collector token.
- Forward Events from a CDP or Server — the other producer-side job.
- Compass Onboarding Checklist — phase-by-phase rollout steps.