Reporting Operations
Part of Data integration for BI
Handling missing source data without silently filling gaps
Distinguish absent source deliveries from genuine zero activity and decide how a BI report should show incomplete data.
When expected source data is missing, keep the gap visible until its cause and effect are understood. Check whether the source had no qualifying activity, delivered late, omitted records or supplied records the pipeline rejected. Show zero only when the agreed source coverage supports zero; an absent delivery does not.
Compare expected and received data
Start with the source delivery rather than the chart. Define what constitutes a complete delivery: a dated file, table partition, range of change identifiers or another bounded set. Compare what was expected with what arrived and what the pipeline accepted.
| Observed state | Initial reporting treatment |
|---|---|
| Expected delivery absent | Mark the affected period unavailable or retain the last accepted result with its cutoff. |
| Delivery present with zero eligible records | Confirm its complete-period signal and eligibility rule before showing zero. |
| Delivery present with rejected records | Inspect the rejects and hold or qualify affected figures. |
| Delivery late or corrected later | Record which published periods may need revision. |
The report owner decides the treatment under its release rule. An unaffected segment may still be usable if its population is clearly labelled.
Preserve the evidence of the gap
Keep the expected period, actual arrival time, source identifier, received and rejected counts, pipeline run and affected report. Retain earlier run states after a successful retry so investigators can identify which result readers previously saw.
A successful transfer status does not prove that all expected records reached reporting. In supported Azure Data Factory Copy activity scenarios, configured fault tolerance can skip incompatible data, and skipped data can be logged by enabling a session log within the copy activity. Review the monitoring details for the specific copy run alongside the expected source delivery.
Do not make a graph appear continuous by silently replacing a missing value with zero, carrying forward a prior value or interpolating between known values. Each changes the meaning. If an analytical method calls for an estimate, approve and label it separately from observed data.
Key Metrics to Preserve When Handling Missing Data
- Expected period
- Defined by source delivery schedule (e.g. daily file, monthly partition)
- Actual arrival time
- Timestamp of when data was received in the pipeline
- Source identifier
- File name, table partition, change ID range, or other unique source reference
- Received count
- Number of records successfully transferred
- Rejected count
- Number of records skipped due to validation or transformation errors
- Pipeline run ID
- Unique identifier for the copy activity execution
- Affected report
- Name or ID of the downstream report impacted by the gap
Release and repair deliberately
Name the business owner who decides whether a partial result can support the decision and the integration owner who investigates the cause. Specify the smallest affected period, source, location or measure. A multi-location report, for example, might show arrived locations as a limited population while withholding or clearly marking the all-location total.
When the source arrives, apply a defined replay or backfill rule. Check for missing or repeated records and identify earlier reports or exports whose figures changed. Before the next release, confirm that required deliveries cover the agreed period, rejects have an approved treatment and readers can see the correct cutoff.
