Experience Platform · batch ingestion
The load that half failed
Every night the ecommerce orders batch comes in: 10,000 purchases and €500,000 in total. Tonight, some rows carry the amount as text. Choose how it is ingested and how it is reprocessed once the source is fixed. See how many purchases and how much money Profile and the data lake see.
Example assumptions
Example batch, orders, amounts and percentages. Each order is worth €50 on average.
Two nights
What gets in on each load
Tonight's orders load and tomorrow's reprocessing, row by row.
Profile and data lake
What each place counts
The same orders can add up differently in Profile, in the data lake and in the sales report.
The recommendation
Partial ingestion with a threshold, a stable _id and reprocessing only what failed
A failed load is not serious. What is serious is fixing it by reloading everything with new identifiers.
- 01
Partial ingestion with a threshold
That way a few bad rows do not take down the good ones. The threshold depends on what the business tolerates. Above it, the whole batch should fail so someone looks at it.
- 02
A business _id
The order number, not a random identifier that changes on every load. With the same _id, Profile discards the repeated event.
- 03
Reprocess only the failed rows
Download the error diagnostics, fix those rows and load only those. Reloading the whole file creates duplicates in the data lake even with a stable _id.
- 04
Deduplicate where you analyse
In Query Service and in reports, one record per _id. And an alert on failed batches, so you do not find out from Monday's report.
Explore this decision
- The load is green. What about the campaign?
A completed load does not prove a campaign works. Separate technical health, data freshness and recovery in AEP operations.
- Fix the data or fix the report?
A misclassified channel can be fixed at source, during ingestion or in CJA. Compare who receives the correction and what happens to historical data.
- The purchase is in AEP. Where does it go missing in CJA?
Trace a purchase missing from CJA through the dataset, connection, historical import, data view and filters before loading the data again.