businesswoman, presentation, men's suit, hand, keep, control, control element, quality control, certification, check, rubber stamp, inspection, review, examination, symbol, sale, selling, sample, trade, test, business, label, circular, high class, high quality, quality, quality control, quality control, quality control, certification, inspection, review, quality, quality, quality, quality, quality
Photo by geralt on Pixabay

Data Governance

Part of Data quality checks

Monitoring unexpected null values in critical fields

A null value means a field has no recorded value; it does not always mean a process failed.

This is an editorial pen name used by the publication, not a claim of an independently verified personal identity. Original editorial credit: Desk Governance.

A null value means a field has no recorded value; it does not always mean a process failed. Monitoring is useful when a team knows which fields should be filled, under what conditions and by what point in the workflow. A blanket “no nulls” rule often produces noise, while a missing order key or reporting date can corrupt a key measure.

Define the required population

For each critical field, write an explicit rule. An invoice date may be required only after an invoice is issued. A cancellation reason may be expected only for cancelled orders. A shipping address may be irrelevant to a digital delivery.

Apply the check to rows where the field is genuinely required. Specify whether empty strings and placeholder values should count as missing too.

Choose a denominator that makes the trend meaningful. The null rate among issued invoices answers a different question from the null rate among all order drafts.

Capture both the number of affected records and the percentage, since a small percentage of a very large batch may still matter operationally. Break the result down by source, status and ingestion day before assuming the problem is global.

Detect change without inventing a universal threshold

Compare recent batches with a relevant baseline, allowing for seasonal or process changes. A move from almost no missing values to a concentrated spike in one integration is a useful signal. A field that has always been optional needs a different rule.

Agree on alert levels with the report owner and source-system owner. A threshold should reflect the business consequence of the missing value, not an arbitrary round number.

Preserve sample failing record IDs and the test version. If the source delivers a value after an initial event, check whether the alert is really about latency. For imported BI models, note the model refresh time before comparing a source extract with a report. Otherwise a timing mismatch can be mistaken for data loss.

Respond to the cause

Trace missingness through the form, API, transformation and model. A field may be absent in the source, renamed in a payload, dropped in a join or transformed into null after a type conversion. Assign the correction to the layer that introduced it. Backfill only when the original value can be established; a default that makes a chart complete may change its meaning.

Document accepted exceptions and revisit them. If a new business process makes the field optional, change the rule and metric definition openly. An unexpected gap prompts a clear investigation before it becomes a decision made from incomplete data.

Tracing Missing Values Through Data Layers

  1. Source SystemField missing or not populated
  2. API / Integration LayerField renamed or dropped in payload
  3. Transformation PipelineValue converted to null during type casting
  4. Data Model / BI ReportJoin operation removed record due to mismatch

More from Data Governance