A planning guide from Data & AI Lab. All numerical examples below are hypothetical.
A pipeline is ready to build when the team can explain what it must deliver, how to recognise incorrect output and how to recover from a failed run. Choosing an orchestration tool does not answer those questions. Start with one source and one business decision, then use the checklist below to expose the gaps.
1. Define the decision and its deadline
Write a sentence that connects the output to an action. For example: “The operations manager reviews yesterday’s dispatched orders at 09:00 to identify delayed deliveries.” That sentence tells you which event matters, who needs the result and when stale data becomes a problem.
Now separate the source deadline from the pipeline deadline. If the warehouse system completes its export at 08:45, promising a validated dashboard at 08:30 is impossible regardless of the cloud platform. Agree on a visible freshness label and an escalation route for missing exports.
- Who makes the decision and what do they do differently?
- Which timezone defines “yesterday”?
- What is the latest acceptable source timestamp?
- Should a late run block the dashboard or display the last valid result with a warning?
2. Agree on a source contract
Document the grain: what does one row represent? An order header and an order line are different grains. Joining headers to multiple lines and then adding a header-level total can multiply revenue. Record the primary key, update timestamp, delete signal, expected fields and the contact responsible for schema changes.
For a hypothetical order-line feed, the key might be (order_id, line_id). Confirm whether that pair is unique across all stores or whether a store identifier belongs in the key. Check the data rather than relying only on a column name.
Ask for at least three representative samples: a normal delivery, a corrected delivery and a deleted or cancelled record. Include a source with no new records so you can distinguish “nothing changed” from “the export failed.” Treat access permissions and retention as part of this contract.
3. Choose a loading strategy you can explain
| Approach | Useful when | Question to resolve |
|---|---|---|
| Full refresh | The dataset is small enough to reload within the agreed window. | Can you publish the new snapshot atomically so users never see a half-loaded table? |
| Timestamp increment | The source has a dependable change timestamp. | How are late updates, tied timestamps and deletes captured? |
| Change data capture | You need detailed changes and the source supports an appropriate change log. | Who monitors log retention, connector lag and schema changes? |
Do not select continuous processing simply because it sounds modern. If the decision happens once each morning, a dependable daily batch may be easier to operate. If the business action depends on changes within minutes, test that requirement against source availability and operational cost.
4. Write checks with an owner and an action
“Check data quality” is not an acceptance criterion. Specify what fails, what the pipeline does and who responds. Separate hard failures from warnings; one missing optional label should not necessarily stop every report.
| Check | Example rule | Response |
|---|---|---|
| Key uniqueness | No duplicate order-line keys in the published snapshot. | Stop publication and retain the previous valid snapshot. |
| Freshness | The export covers the agreed business date. | Display stale status and notify the source owner. |
| Reconciliation | Dispatched totals match the source control report using identical filters. | Investigate mismatches before stakeholder sign-off. |
| Schema | Required fields exist and can be parsed. | Quarantine the delivery and record which fields changed. |
Volume checks need context. A zero-row day might be correct on a holiday. Establish expectations using the source calendar and representative history, then record exceptions instead of silently lowering the threshold.
5. Prove that reruns are safe
Suppose a daily batch contains 1,000 order lines and fails after writing 600. A blind append on retry can leave 1,600 rows. A staging-and-publish pattern, a replaceable partition or a key-based merge may solve this, depending on the source semantics. The acceptance test is concrete: run the same batch twice and confirm that the published result remains correct.
Test a late correction too. If yesterday’s amount changes today, does the pipeline update the earlier record or create a duplicate? Keep enough delivery metadata to identify the source file or change position behind a row. Define how far back the team can replay and what source retention limits that promise.
6. Finish with an operational handover
Before adding more sources, have someone other than the builder recover a deliberately failed run using the runbook. Record the schedule, credentials owner, alert destination, recovery steps, source retention, reconciliation rules and the person who accepts the output.
A useful first milestone is one source, one validated output and one demonstrated recovery. Expand after the team can operate that slice. If definitions are still disputed, work through the metric contract guide before adding dashboard complexity.