Testing a pipeline after a source schema change

After a source schema changes, test the affected path before accepting the next BI result. Compare the old and new field contract, exercise the change with representative records, then check extraction, mappings, delivered data and affected report figures. A completed pipeline run does not establish that the result still means the same thing.

Classify the change

Ask the source owner when the change took effect, which tables or payloads changed, what each field meant before and after, and whether historical records were altered.

An unchanged name and type do not establish unchanged meaning. A status code may keep its format while its business use changes.

Impact of Source Schema Changes on Pipeline Testing

Check connector behaviour

When source metadata changes between executions, Azure Data Factory mapping data flows can accept flexible schemas. This makes the flow late-binding: drifted column names are not available in design-time schema views. Seeing a field at the source is not proof it reached reporting.

Datastream can stream ongoing changes using CDC or provide a historical snapshot through backfill. Check the destination data for the intended change; a configured replication path alone does not establish that reporting output is correct.

If the final report uses Power BI, check its model separately: refresh behaviour depends on storage mode, and only Import models require a source-data refresh. Confirm the model and report show the intended output after the change.

Key Considerations in Schema Change Testing

Run a bounded test

Use authorised sample records from before and after the change, including values that challenge the mapping. Write the expected output for each, then check:

For a renamed completion timestamp, inspect records on both sides of the change. An old destination column might remain while new records no longer populate it.

dbt's data tests can assert conditions such as non-null values, accepted values and relationships, but a passing assertion only confirms the rule it tests; it cannot establish an untested change in business meaning.

If a required mapping fails, hold the affected result or retain its last accepted version with the earlier cutoff.

If the change predates detection, identify the affected period, repair or replay it under a bounded rule and review downstream figures again. Record which historical results changed so report owners can explain the correction.

Bounded Test Procedure After Schema Change