The interlock worked and nobody knew: three days of automation silently stopped

This case is published anonymously, with no client name, no brand and no name for the third-party system involved: the project has a signed confidentiality and intellectual property agreement. What is told here is the failure mode, which is the reusable technical asset.

The system performs record revalidation: it walks a base of property owners, sends a message, records the reply and keeps the record alive. It cross-references a market CRM against a list of published records, and only contacts people present on both sides.

The chain

On a normal day the flow is this: sync the published list from the external system, build the authorised list from it, and send to whoever is on that list. Three links, in that order.

On the day of the failure, the first link broke. The sync failed, the published list froze on the previous date, and building the authorised list aborted rather than proceeding with the stale one. Both sending flows stopped. And stayed stopped for three days.

Everyone did the right thing

The hard part to accept is that none of those behaviours is a bug. Aborting on an incomplete list is the correct decision, and it exists because of an earlier incident: anyone missing from the authorised list gets marked with a state that is terminal — it does not return to the queue on its own. In one partial sync, five legitimate owners were marked that way and dropped out of the sending base.

After that, the interlock was made to prefer stopping everything over risking erasing people. It worked. That is exactly what it was built for.

From the outside, "no sends today" is indistinguishable from a slow day. It was that ambiguity, not the failure, that cost three days.

The real defect

The defect was not the interlock: it was that nobody found out it had tripped. A system that fails safely and silently is still a system that fails — except the clock runs in the problem’s favour, because the absence of a signal looks like normality.

A human caught it, three days later, looking at a number that did not add up.

The external system’s treacherous failure mode

The root cause deserves its own note, because it is the kind of thing an API’s documentation does not tell you. When that system applies a rate limit, it returns neither an authentication error nor a too-many-requests error. It returns an empty list, with a success code.

And the end-of-pagination condition, on the consuming side, is exactly "empty list". Which means rate limiting and the natural end of the data are the same event to the paginator. It stops believing it finished, and writes a truncated list — or none at all — with every appearance of complete success.

Measured the same day: asking for more than twenty records per page also returns an empty list rather than an error. Anyone trying to speed up pagination that way will believe it worked.

What went in its place

Three fixes, none of them in the link that broke. Sync with three staggered overnight attempts, so an isolated failure does not become a lost day. A staleness watchdog on the list inside the production auditor, which shouts when the data ages beyond expectation. And, in the manual path, an interlock that refuses to apply an incomplete list without explicit confirmation.

None of those three prevents the original failure. All three guarantee it is seen the same day.

The lesson, good for any automation

Failing safe is not the end of a reliability project — it is half of it. The other half is someone, or something, tasked with noticing. Every silent interlock needs a watchdog, and every watchdog needs a number that ages: not "did it error?", but "how long since this data was refreshed?".

The project was delivered on 16 August 2026, with all three fronts live and in use with real data, within the contracted window, and the contract paid in full.