How to use the data extraction framework to bulk download user data

Arc XP Subscriptions provides the Data Extract Framework for generating reports. The API is protected by a rate limit, which by default allows 4 requests per hour.

Note

Before starting, ensure you meet the prerequisites defined for Manage User Accounts: Developer Interface.

You can download all extracts by POSTing to the following developer API (with a valid PAT):

curl --location --request POST 'https://api.sandbox.staging.arcpublishing.com/retail/api/v1/extract/<extract name>' \
--header 'Authorization: Bearer {ARC_ACCESS_TOKEN}' \
--header 'Arc-Site: staging' \
--header 'Content-Type: application/json' \
--data '{
"startDate": 1727200066000,
"endDate": 1729792066000
}'

Each request is required to submit a start date and end date, which represent the time range in Unix epoch time for which you need to retrieve records for.

The extracts contain a stream of individual JSON documents rather than one single JSON document. This is due to the potential size of the extract and that the APIs cannot hold that amount in memory at the same time. To validate as JSON when loading into your system, wrap the file in [ ] (brackets), and then do a find and replace all operation, at the end of each object replacing } with }, except for the last object. The response will be in JSON and the format will be unique for each type of report.

Response from API

{"site":["theguide"],"subscriptions":[],"clientId":"abf2073a-ce5b-4dbd-bed9-941aae676427","address":[],"identity":[{"lastLogin":1728070635000,"type":"Password","createdOn":1728070635000}],"profile":{"emailVerified":true,"firstName":"PW Cassandre","lastName":"Test Stark","contact":[],"attribute":[],"email":"automation.1728070629511@gmail.com"},"sales":{"firstName":[],"lastName":[],"secondLastName":[],"createdOn":1728070635000},"status":"Active"}
{"site":["theguide"],"subscriptions":[],"clientId":"ee87743f-3157-4977-8af2-52ad60793b75","address":[],"identity":[{"lastLogin":1728070686000,"type":"Password","createdOn":1728070686000}],"profile":{"emailVerified":false,"firstName":"PW Kenyatta","lastName":"Test Green","contact":[],"attribute":[],"email":"automation.1728070682954@gmail.com"},"sales":{"firstName":[],"lastName":[],"secondLastName":[],"createdOn":1728070686000},"status":"Active"}

Making the response a valid JSON

[{"site":["theguide"],"subscriptions":[],"clientId":"abf2073a-ce5b-4dbd-bed9-941aae676427","address":[],"identity":[{"lastLogin":1728070635000,"type":"Password","createdOn":1728070635000}],"profile":{"emailVerified":true,"firstName":"PW Cassandre","lastName":"Test Stark","contact":[],"attribute":[],"email":"automation.1728070629511@gmail.com"},"sales":{"firstName":[],"lastName":[],"secondLastName":[],"createdOn":1728070635000},"status":"Active"},
{"site":["theguide"],"subscriptions":[],"clientId":"ee87743f-3157-4977-8af2-52ad60793b75","address":[],"identity":[{"lastLogin":1728070686000,"type":"Password","createdOn":1728070686000}],"profile":{"emailVerified":false,"firstName":"PW Kenyatta","lastName":"Test Green","contact":[],"attribute":[],"email":"automation.1728070682954@gmail.com"},"sales":{"firstName":[],"lastName":[],"secondLastName":[],"createdOn":1728070686000},"status":"Active"}]

Data in the extracts has limited overlap with just enough information to allow clients to (re)link records in their own systems.

If you are handling Subscriptions, the financial extract contains user IDs (UUIDs) to allow the client to link the information collected about a user to any financial transactions that they may have made. Similarly, subscriptions can be linked to both users and financial transactions via user IDs in the subscription extract and subscription IDs in the financial extract.

The following are the current reports and the extract(s) that the client should use for generating the equivalent reports in its own system.

Types of extracts

Arc XP Subscriptions has the following extracts available:

Extract name	Description
financial	Orders and financial transactions
user	Registered accounts
activity	All changes made to subscriptions and users accounts: Subscription Starts, Terminations, Renewals, etc
subscription	Subscriptions with Payment Details
csrAudit	An audit log of CSR activities
metrics	Statistics with paywall display and subscription conversions
product	Products in Retail — Available only to Commerce custumers
event	Events in Retail — Available only to Commerce custumers
order	Order details — Available only to Commerce custumers

In the cURL command, you need to replace <extract name> with any of the values provided in the ‘Extract Name’ column, depending on the information you’re looking for..

Possible error messages

Extracts are limited to four per hour and there are no other limits. If the limit is exceeded, you receive an HTTP 429 error response.

200311; Missing startDate
200312; Missing endDate
200313; StartDate should be before the endDate
300134; Invalid report type

Transitioning From Reports to Extracts

If you are using reports, you should consider using the extracts in place of the reports. The following are the current reports and the extract(s) that the client should use for generating the equivalent reports.

Report type	report API	Extract name
refund-audit	POST sales/api/v1/report/schedule	financial
subscription-event	POST sales/api/v1/report/schedule	activity
subscription-event-v2	POST sales/api/v1/report/schedule	financial
subscription-summary	POST sales/api/v1/report/schedule	subscription
financial-report	POST sales/api/v1/report/schedule	financial
financial-report-v2	POST sales/api/v1/report/schedule	financial
banco-sabadell-financial-report	POST sales/api/v1/report/schedule	financial
current-subscription-summary	POST sales/api/v1/report/schedule	subscription
future-failed-payment	POST sales/api/v1/report/schedule	financial, subscription
whitelisted-email-domain-summary	POST sales/api/v1/report/schedule	subscription
whitelisted-email-domain-members	POST sales/api/v1/report/schedule	subscription
applied-price-changes	POST sales/api/v1/report/schedule	subscription
upcoming-price-changes	POST sales/api/v1/report/schedule	subscription
duplicate-vindicia-payments	POST sales/api/v1/report/schedule	subscription
sign-up-summary	POST identity/api/v1/report/schedule	user
data-export	POST identity/api/v1/report/schedule	user
account-activity-last-login	POST identity/api/v1/report/schedule	user
account-activity-last-modified	POST identity/api/v1/report/schedule	user

Finantial extract

Extract name: financial

The financial extract contains a list of financial transactions, tracked by Arc XP Subscriptions, that took place between caller-specified start- and end-dates.

The caller receives a comma-separated stream of records where each record is terminated by a new line (\n). There is no formatting to the data. Each record consists of the following fields:

Field name	Description
clientId	The customer’s, to whom the record is attributed, client ID wrapped in double quotes.
site	The client site to which the transaction belongs, wrapped in double quotes.
transactionDate	The epoch time (in ms) that the transaction took place.
amount	The total amount of the transaction.
currency	The currency of the transaction.
transactionType	The type of transaction. Can be one of: Payment: the transaction was a payment Refund: the transaction was a refund GiftPayment: the transaction was a payment for a gift subscription GiftRedemption: the linkedFinancialTransactionId indicates which GiftPayments this transaction relates to
tax	Total taxes charged with the transaction.
orderNumber	Arc’s order number associated with the transaction.
sku	A comma separated list of SKUs purchased as part of the order, wrapped in double quotes.
paymentPartner	The name of the payment gateway that processed the transaction.
creditCardType	The type of credit card used in the transaction.
subscriptionId	A comma separated list of subscription IDs that were purchased as part of the order, wrapped in double quotes.
amortizationStart	For subscriptions – the start of the amortization period.
amortizationEnd	For subscriptions – the end of the amortization period.
line1	Address line1 wrapped in double quotes.
line2	Address line2 wrapped in double quotes.
country	Address country wrapped in double quotes.
locality	Address locality wrapped in double quotes.
region	Address region wrapped in double quotes.
postal	Address postal wrapped in double quotes.
providerReference	An ID assigned by the payment gateway to the transaction.
taxProviderReference	An ID assigned by the tax gateway to the transaction.
financialTransactionId	An ID assigned by Arc to the transaction – guaranteed to be unique and can be used for preventing duplicates inserts into client’s own database.
linkedFinancialTransactionId	Links transactions with transaction types GiftPayment and GiftRedemption. Note: transactions of type GiftPayment will see this value update when the associated gift subscription is redeemed.
orderType	Parent (indicates a new order), Renewal (indicates that the order was for a subscription renewal)
phone	The phone number associated with the order. As submitted by customer when creating the order.
orderEmail	Email address associated with the order. As submitted by customer when creating the order.
orderFirstName	First name associated with the order. As submitted by customer when creating the order.
orderLastName	Last name associated with the order. As submitted by customer when creating the order.
orderSecondLastName	Second last name associated with the order. As submitted by customer when creating the order.
quantity	Number of items purchased.
price	The price of an individual item.
priceCode	The price code from Arc’s Retail system associated with the transaction.
campaignCode	The campaign code associated with the order. As submitted by customer when creating the order.
gift	Is the transaction for a gift? (boolean)
taxInclusive	Were taxes calculated as inclusive?
firstSix	First six of credit card - if available from payment gateway
lastFour	Last four of credit card – if available from payment gateway
cardExpiration	Credit card expiration – if available from payment gateway
cardholderName	Cardholder name – if available from payment gateway
identification	Identification number – if available from payment gateway
documentType	Type of identification – if available from payment gateway
gatewayTransactionID	TransactionID provided from Payment Gateway if applicable
initialTransaction	Flag to indicate first transaction of subscription
reasonCode	Customer care ReasonCode in case of refund transaction
notes	Any additional note for the transaction

User extract

Extract name: user

The user extract provides information that centers around the user. The caller must specify start- and end-dates that control how data appears in the extract based on date and time that data was first collected about the user. Note that the two extracts with the same bounding dates can look different due to users making updates.

Field Name	Description
clientId	The customer’s, to whom the record is attributed.
site	The client site to which the user belongs.
status	Any from [Active, Disabled, Anonymized, PendingAnonymize]
profile	User’s profile array
email	User’s main email
emailVerified	Flag if email is verified or not
firstName	User’s first name
lastName	User’s last name
secondLastName	User’s second last name
attribute	Array of user attributes
attribute:name	Attribute name
attribute:value	Attribute value
attribute:type	Attribute type [String, Number, Boolean, Date]
contact	Array of user contact details
contact: phone	User’s phone number
contact:type	Phone number type
address	Array of all address types: profile, bill and shipping
address: line1	Address line 1
address: line2	Address line 2
address: country	Address country
address: locality	Address locality
address: region	Address region
address: postal	Address postal code
identity	Array of all sign-in identity’s for the user
identity: createdOn	Identity created or, in the case of a social connection, linked date
identity: lastLogin	Last successful login date
identity: type	Login type [Password, Facebook, Google, Apple, Cognito, OIDC]
sales	Array of Sales contact information
sales: firstName	All user’s first names in Sales
sales: lastName	All user’s last names in Sales
sales: secondLastName	All user’s second last names in Sales
sales: createdOn	User created in Sales
subscriptions	Array of Subscription IDs

Activity extract

Extract name: activity

The activity extract contains information about changes that have been made to user and subscription objects. This includes notes entered by CSRs, failed payment attempts, cancelations, and terminations.

Field name	Description
clientId	The customer’s, to whom the record is attributed.
site	The client site to which the event belongs.
activityDate	Activity date
note	Financial/event/user note
activityType	Type of activity
subscriptionID	SubscriptionID if activity belongs to a subscription
subscriptionSKU	Current Product SKU of subscription
priceCode	Current price code of subscription
campaignCode	Subscription campaign code
productName	Product name if activity was a financial event
repClidentID	CSR ID if event took place in the customer service admin
transactionID	Object unique Id for which activity created on, financial, event or user note
financialTransactionID	Unique Id from financial activity
currency	Financial transaction currency type
amount	Financial transaction amount
tax	Financial transaction tax amount
paymentProvider	Payment processor
reasonCode	Customer care reason code
reasonValue	Additional note from activity

Subscription extract

Extract name: subscription

The subscription extract provides a list of all paid and non-paid subscriptions. All lines have SKUs while only paid subscriptions have price codes. Each record consists of the following fields:

Field name	Description
clientId	The customer’s ID, to whom the record is attributed.
site	The client site to which the subscription belongs.
paymentPartner	Payment processor.
subscriptionId	Subscription ID
status	Current subscription status [Active, Terminated, Canceled, Suspended, Gifted, Dunning, Vindicia]
startDate	Subscription created on date
terminationDate	Subscription terminated on date, if subscription is in a terminated status
sku	Current Product SKU of the subscription
creditCardLastFour	Last four digit of credit card used for payment
creditCardExpiration	Expiration date of credit card used for payment in MMYY format
nextEventDate	Next renewal date
priceCode	Current price code of subscription
cycleIndex	Current retail price cycle
group.accessCode	Nonce code used by the user to get access to their group subscription
group.listName	Name of the group subscription
group.packageID	Group package ID

Audit extract

Extract name: csrAudit

The audit extract contains details of CSR sales event activity allowing clients to track actions taken by their representatives.

Field Name	Description
eventTime	UTC time when the event activity happened
csrClientId	Customer Support Representative ID who executed the activity
site	Site for which Customer Support Representative did the activity
eventType	Activity performed by Customer Support Representative
event	Input detail of the event like emailId, clientId, search param
csrIP	IP address of Customer Support Representative
subscriptionID	Subscription ID if event is on Subscription

Paywall Metrics Extract

Extract name: metrics

The paywall metrics extract contains data on paywall views, offers, and checkout to help make business decisions that improve subscription market strategy.

Field Name	Description
eventType	Activity performed by user: display OR sign_in OR checkout OR purchase
documentID	Data on the activity: productSku, priceCode, campaignCode, offerId
paywallRuleID	Paywall rule that triggered the event to be started

Extracts that are available only to Commerce Customers

Product Extract

The product extract contains details of all product data in retail.

Field Name	Description
createdBy	The id of the user who created the product
createdOn	The UTC date the product was created on
modifiedBy	The ID of the user who last modified the product
modifiedOn	The UTC date the product was last modified on
id	The id of the product
name	name of the product
sku	sku of the product
typeId	Subscription(1), Physical(2), Override(3), Event(4);
schema	Custom schema associated with the product
specifications	Custom schema of specifications associated with the product
hasVariant	Flag to indicate if the product has variants
active	Boolean value to indicate if the product is in active status
default	n/a
location.createdBy	The id of the user who created the location
location.createdOn	The UTC date the location was created on
location.modifiedBy	The id of the user who last modified the location
location.modifiedOn	The UTC date the location was last modified on
location.id	The id of the location for this product
location.name	The name of the event
location.seoId	The unique friendly ID
location.description	The location description
location.statusTypeId	Available(1), UnAvailable(2)
location.schema	Custom schema associated with the location
location.address.createdOn	The UTC date the address was created on
location.address.modifiedOn	The UTC date the address was last modified on
location.address.id	The id of the address for the location
location.address.typeId	Location(1)
location.address.country	The address country
location.address.line1	The address line1
location.address.line2	The address line2
location.address.locality	The address locality
location.address.region	The address region
location.address.postal	The address postal
location.address.schema	Custom schema associated with the address
subtype.createdBy	The ID of the user who created the subtype
subtype.createdOn	The UTC date the subtype was created on
subtype.modifiedBy	The ID of the user who last modified the subtype
subtype.modifiedOn	The UTC date the subtype was last modified on
subtype.id	The ID of the subtype associated to this product
subtype.name	Name of the subtype
subtype.description	description of the subtype
subtype.slug	The unique friendly ID

Event Extract

The event extract contains details of all event data in retail.

Field Name	Description
createdBy	The ID of the user who created the event
createdOn	The UTC date the event was created on
modifiedBy	The id of the user who last modified the event
modifiedOn	The UTC date the event was last modified on
id	The id of the event
name	The name of the event
dateTime	The UTC date of the event
eventStatusId	Scheduled(1), Cancelled(2), Onhold(3)
seoId	The unique friendly ID
schema	Custom schema associated with the event
location.createdBy	The id of the user who created the location
location.createdOn	The UTC date the location was created on
location.modifiedBy	The id of the user who last modified the location
location.modifiedOn	The UTC date the location was last modified on
location.id	The id of the location for this event
location.name	The name of the event
location.seoId	The unique friendly ID
location.description	The location description
location.statusTypeId	Available(1), UnAvailable(2)
location.schema	Custom schema associated with the location
location.address.createdOn	The UTC date the address was created on
location.address.modifiedOn	The UTC date the address was last modified on
location.address.id	The id of the address for the location
location.address.typeId	Location(1)
location.address.country	The address country
location.address.line1	The address line1
location.address.line2	The address line2
location.address.locality	The address locality
location.address.region	The address region
location.address.postal	The address postal
location.address.schema	Custom schema associated with the address

Order Extract

The order extract contains details of all order data in sales.

Field name	Description
orderDate	The date the order was placed
amount	Base amount of the order (no tax, shipping, etc.)
currency	Ex:USD
orderNumber	Unique id for the order
firstName	Customer first name
lastName	Customer last name
phone	Customer phone
orderId	Internal order ID
shipping	Shipping cost
email	Customer email
paymentPartner	Payment provider name
status	Pending(1), PartiallyPaid(2), Paid(3), Cancelled(4), AwaitingApproval(5), Approved(6), Rejected(7), Authorized(8), Declined(9), Void(10), Preauthorize(11), PreApproval(12)
subtotal	Base amount of the order (no tax, shipping, etc.)
total	Total (including tax and shipping)
totalTax	Sum of taxes
billingAddress.line1	The address line1
billingAddress.locality	The address locality
billingAddress.region	The address region
billingAddress.country	The address country
billingAddress.postal	The address postal
billingAddress.taxexempt	Flag indicating if this address is tax exempt
transaction[].transactionType	Payment(1), Refund(2), GiftPayment(3), GiftRedemption(4), Authorize(5), Void(6), Error(7)
transaction[].transactionId	Unique ID for the transaction
transaction[].paymentId	Payment id for the transaction
transaction[].transactionDate	Date of the transaction
transaction[].paymentPartner	Payment Provider name for the transaction
transaction[].lastFour	Last four of CC
transaction[].cardType	Type of CC
transaction[].cardExpiration	Expiration of CC
transaction[].amount	Amount of transaction
transaction[].tax	Tax on transaction
transaction[].total	Sum of amount and tax
orderLine[].productPrice	The price of the product purchased
orderLine[].quantity	The quantity of the product purchased
orderLine[].tax	The tax of the product purchased
orderLine[].currency	Ex:USD
orderLine[].productName	The product name
orderLine[].productSku	The product sku
orderLine[].subtotal	Quantity x Price of this line
orderLine[].total	Quantity x Price + tax of this line
orderLine[].taxInclusive	Flag indicating if taxes apply
orderLine[].productTypeId	DigitalSubscription(1), Event(2), Merchandise(3)
orderLine[].schema	Custom schema associated with the line
orderLine[].event.name	Event name
orderLine[].event.eventDateTime	Date of the event
orderLine[].event.categories[].name	Category names associated with the event
orderLine[].event.location.name	Location name of the event
orderLine[].event.location.address.country	Location country of the event
orderLine[].event.location.address.line1	Location line address of the event
orderLine[].event.location.address.line2	Location line address of the event
orderLine[].event.location.address.locality	Location locality of the event
orderLine[].event.location.address.region	Location region of the event
orderLine[].event.location.address.postal	Location postal of the event