Monitoring unexpected null values in critical fields

A null value means a field has no recorded value; it does not always mean a process failed. Monitoring is useful when a team knows which fields should be filled, under what conditions and by what point in the workflow. A blanket “no nulls” rule often produces noise, while a missing order key or reporting date can corrupt a key measure.

Define the required population

For each critical field, write an explicit rule. An invoice date may be required only after an invoice is issued. A cancellation reason may be expected only for cancelled orders. A shipping address may be irrelevant to a digital delivery.

Apply the check to rows where the field is genuinely required. Specify whether empty strings and placeholder values should count as missing too.

Choose a denominator that makes the trend meaningful. The null rate among issued invoices answers a different question from the null rate among all order drafts.

Capture both the number of affected records and the percentage, since a small percentage of a very large batch may still matter operationally. Break the result down by source, status and ingestion day before assuming the problem is global.

Detect change without inventing a universal threshold

Compare recent batches with a relevant baseline, allowing for seasonal or process changes. A move from almost no missing values to a concentrated spike in one integration is a useful signal. A field that has always been optional needs a different rule.

Agree on alert levels with the report owner and source-system owner. A threshold should reflect the business consequence of the missing value, not an arbitrary round number.

Preserve sample failing record IDs and the test version. If the source delivers a value after an initial event, check whether the alert is really about latency. For imported BI models, note the model refresh time before comparing a source extract with a report. Otherwise a timing mismatch can be mistaken for data loss.

Respond to the cause

Trace missingness through the form, API, transformation and model. A field may be absent in the source, renamed in a payload, dropped in a join or transformed into null after a type conversion. Assign the correction to the layer that introduced it. Backfill only when the original value can be established; a default that makes a chart complete may change its meaning.

Document accepted exceptions and revisit them. If a new business process makes the field optional, change the rule and metric definition openly. An unexpected gap prompts a clear investigation before it becomes a decision made from incomplete data.

Tracing Missing Values Through Data Layers