Data completeness: a pipeline that reports its own shortfall

Summary

The collectors of a market analytics system pull open data on procurement and job listings and run as Temporal tasks: the results are written to a database and delivered to a CRM, and Better Stack watchdogs wait for a heartbeat after every run.

The task: a run that collected less than everything must look like a failure, not a success.

The trigger was a neighbouring collector that lost about 10% of its feed while its watchdog stayed green (section 6). On 13 August 2026 we checked whether "collected less than everything" ever reached a run's verdict. It never did: 44 runs in the history without a single failure, and 100.00% watchdog availability for the week. That did not mean "everything was collected"; it meant "a shortfall is invisible by construction". Within a day we added a denominator, a second pass over what was missed and a "fully collected" verdict, and verified the changes with two-sided tests and on the deployed code. We made the checks strict only where measured shortfall was zero: a watchdog that turns red on healthy runs is worse than none.

1. Why the shortfall was invisible

The data path: collection → Temporal activity → workflow → delivery bridge → watchdog.

Temporal retries an activity and propagates a failure only if the activity raises an exception. Every activity we examined returned a statistics dictionary instead. So on a partial collection the activity completed successfully, so did the workflow, and the bridge sent its heartbeat to the watchdog. A retry policy was configured but never kicked in: there was nothing for it to learn about.

We measured that claim: 44 executions in the history, not one failure; 100.00% watchdog availability over the week.

2. Where data was lost

1. Card collection. If a page could not be fetched after retries, the card silently disappeared: there was no loss counter. There was no denominator either. "1,439 of 1,439" could not be told apart from "1,439 of 1,600" — neither by the watchdog nor by a person reading the run's result.
2. Pagination. A lost search page looked like the end of the results: an empty response body and "no more matches" produced the same empty list.
3. Batched writes. A failed batch was swallowed silently, and the shortfall was visible only by subtracting two numbers by hand. Worse, the same loss was sometimes loud and sometimes silent, depending on whether the database server managed to return an error or the connection dropped.
4. CRM delivery. There was an error counter, but it had no effect on the run's outcome: the heartbeat to the watchdog was sent unconditionally.

3. Measure the frequency first, then add strictness

A watchdog that turns red on a healthy run gets switched off within a week, and things end up worse than before. So before deciding on strictness we needed the shortfall frequency. The run history could not answer that: it had no denominator.

We ran the collectors with every request counted: production parameters, the same node, the same code (checksums of the deployed files matched against the repository), and no database writes.

• Contracts: 1,439 of 1,439, with not one failure across 1,475 requests.
• Tenders: 293 of 293, 311 requests; here the window was 7 days, wider than in production.
• Job listings: 30 of 30 pages.

Where the history did print a denominator, errors turned up in CRM delivery: in 2 runs of 27. In one, 6 cards of 60 did not arrive (the CRM answered "too many requests", HTTP 429); in the other, 1 of 76 (a duplicate name).

4. What we changed

• A denominator. Collection counts what was found, what was missed and how many pages were lost.
• A second pass over what was missed, at a slow pace. That is why no threshold is needed: a one-off blip of the source is cured by the retry, and whatever is still missing after a pause is no longer a blip.
• A verdict. "Fully collected" travels from collection to the bridge, and on a partial collection the heartbeat is suppressed. The activity deliberately does not fail: what was collected still has to be delivered, and it is the watchdog that should turn red, not delivery.
• Pagination. A lost page no longer looks like the end of the results.
• Writes. Failed batches are reported by name.

Where we deliberately kept things lenient: delivery. Errors occurred in 2 of 27 runs, so the watchdog would turn red every two weeks on a live system. First we need a retry that is safe against duplicates: a write retried blindly after a timeout can create a second card for the same organisation.

Fig. 1. The "fully collected" verdict travels from collection to the bridge. On a partial collection the heartbeat is suppressed, and since 26 September the workflow also fails — after the collected data has been delivered.

5. Verification

• Two-sided tests. 19 checks: the guard fires on a partial collection and stays quiet on a full one. The second half matters more: a watchdog that cries wolf gets switched off. All 19 fail on the old code and all pass on the new one.
• Verification on the deployed code — along all four changed paths, not just a neighbouring one. Tender collection: 120 found, 120 parsed, collection complete. The bridge received the verdict from collection and ran without errors, and so did delivery. Job-listing collection: 559 collected, 559 written.

6. The neighbouring collector: about 10% of the feed behind a green watchdog

This class of bug is where it all started: the same day it was found and fixed in a neighbouring open-data auction collector. There, only a complete absence of rows counted as a failure, and any partial collection counted as success. Six runs in a row it lost 41, 45, 49, 57, 58 and 58 cards out of roughly 550 — about 10% of the feed — with exit code 0 and a green watchdog.

Who is to blame shows in the shape of the failures. In every run the first hundred cards came through clean, and after that failures grew by a steady 13–14 per hundred. Random instability of the source would have taken its share from the first hundred too. This is what an exhaustible request budget looks like: the shortfall was created by the collector's pace itself. The check: the same cards, requested one at a time with a pause, came back without errors.

Feedback has to be calculated. The obvious fix is an adaptive pace: after a failure the pause between requests doubles, after a success it is multiplied by 0.9. It looks like overload protection, but such a pair has an equilibrium — the failure share p at which the pause stays the same on average:

p · ln 2 = (1 − p) · ln(1/0.9), which gives p ≈ 13.2%.

That is exactly the failure share seen after the first hundred cards (13–14 per hundred): such a fix would have changed nothing. The overall loss per run is lower, about 10%; by our estimate, the clean first hundred cards pull it down. With a multiplier of 0.99 instead of 0.9 the equilibrium is 1.4%. The feedback constants set the failure share, so they are calculated, not tuned by eye.

What was done in that collector. One pace budget for the whole run; the default pace was lowered, because the old one was itself a cause of the shortfall. A withdrawn lot (a 404 response) counts as normal feed attrition, not a failure. The second pass over missed cards runs at a slow pace. Any shortfall that survives it returns a separate exit code 3 and suppresses the heartbeat.

7. Since 26 September a partial collection also shows as a workflow failure

The decision "the activity does not fail, the watchdog turns red" still stands. But a green run status next to a red watchdog is only half the truth: in Temporal a partial run looked exactly like a full one. Since 26 September the workflow fails with an explicit error: expected versus actual in the message, the full result in the details, and the error marked as non-retryable. It fails only after delivery: the collected data is delivered, the watchdog gets no heartbeat, and the failure is visible in Temporal and in the workflow-failure alert.

8. What is still open

• A change in the source's page layout still looks like a healthy run. The page arrives with status 200 but no links are found in it: collection concludes there are no more pages and declares itself complete. A page that failed to arrive is now caught; a page that changed is not. As of 26 September this was still open. Strictness on "zero links in a run" waits for a measurement of the lower bound of results, otherwise we would again get a watchdog that turns red on a healthy run.
• Strict delivery comes after a duplicate-safe retry and a measurement of the errors that remain.

Takeaways

• A watchdog proves that a run reached its end, not that the data was collected. As long as those two statements diverge silently, a green watchdog means nothing.
• A collector needs a denominator, a second pass and a verdict. That is a data completeness check: what was collected is reconciled with the number of records the source itself declared. Data observability platforms usually catch volume and freshness anomalies statistically; a denominator from the source is more precise when the source provides one.
• Strictness comes after measuring frequency: a watchdog that cries wolf gets switched off.
• Feedback constants are calculated, not tuned by eye.
• Verify every changed path on the deployed code, not a neighbouring one.