The kill switch nobody connected

The entire kill-switch library was archived out of a private, experimental trading system. This required no change to the execution code and no rework of the factories that create the services.

The reason was simple: nothing on the order-submission path called this safeguard.

Earlier, the kill switch had received its own library, tests, a command-line tool, six alert rules, two dashboards and a 707-line operations runbook. The documentation described five layers of protection, declared the integration effective and ended with the status PRODUCTION READY.

The same commit, however, contained code that contradicted this declaration.

The system had no users and no production deployments. It was a private experiment, not an environment handling real client funds. This story is about an integration defect, not a confirmed production incident.

The mechanism was visible in the code.

The factory creating the order service did not pass the kill switch to it. The service itself tolerated its absence: the parameter was optional, and the checking function returned immediately if no safeguard had been supplied.

This raised no error. The operation could proceed without a check.

In addition, the implementation provided for fail-open behaviour: in some situations where the check failed, it allowed operation to continue for the sake of availability. The documentation described this as an intended property.

With the kill switch not connected, however, these additional error paths were not needed to bypass the safeguard. The function returned earlier.

So the safeguard could work correctly in an isolated test in which someone explicitly passed in its instance. That did not mean the service used it when executing orders.

What is more, the five declared layers of protection did not translate into five effectively connected checkpoints. The API gateway component existed but was not registered in the application. Other checks sat in classes that were never constructed on the executed paths.

A tested component and a component that takes part in execution are two different things.

The missing integration did not have to show up as a failure.

The code contained a health check that, when no kill switch was connected, returned the status OK and a message saying trading was enabled.

There is no evidence that this endpoint actually served requests in a running environment. The behaviour of the recorded code branch matters nonetheless: the absence of the safeguard was treated as a healthy state.

Monitoring showed a similar contrast.

The preserved database dump of one instance contained both kill-switch dashboards. The alert-rule tables, by contrast, held no rules at all. The alert files and documentation existed, but that did not prove the alerts were active in that instance.

In addition, next to the document announcing that the integration was complete, there was a P0-priority ticket describing that the kill switch was not connected to the trading services.

It contained eighteen acceptance criteria. By the time the document was deleted, none had been ticked off.

So this was not a problem that had never been written down anywhere. The declaration that the work was done and the description of the missing integration sat in the same repository.

The preserved sources do not settle who knew the contents of both documents or why the task remained open.

222 days passed between the declaration of effective integration and the archiving of the library.

This gap needs context, though. The repository examined shows a break of 168 days and almost 19 hours without commits. Development of the project was deliberately paused during that period.

This does not mean that the absence of commits proves the absence of all activity. But the whole 222 days cannot be equated with a period of uninterrupted work on the system and of ignoring the defect.

Once changes resumed, the problem with the old mechanism was documented, and the new implementation adopted a fail-closed rule: when the safety of an operation cannot be confirmed, it must be refused.

The old library was then archived.

The message of that change stated that the remaining imports of the kill switch were used only for type checking, meaning they did not execute while the program ran.

The archiving required no changes to the factories and no intervention in the executed logic.

This was one of the strongest confirmations of the problem: the safeguard could be taken out because none of the remaining active dependencies needed it.

The successor system removed significant flaws of the earlier solution.

The dependency on the kill switch became mandatory in the new component, and check errors were to lead to the operation being refused rather than let through.

That did not mean, however, that the corrected mechanism covered every execution route.

The preserved record documents a demonstration order executed through the new path. Analysis of that route's code found no call to the kill switch. The same gap was present in a later revision.

The route did, on the other hand, have a different, real safeguard: by design it was restricted to demonstration mode and rejected any attempt to send an order using real capital.

This distinction matters. No uncontrolled trading with real funds was shown.

What was shown is that the corrected safety mechanism still did not cover one of the routes on which operations were actually executed.

It is not known whether this was a deliberate consequence of restricting the route to demonstration mode or an integration gap that was left behind.

The fact that the implementation was corrected does not settle that question by itself.

The case shows three different levels of control.

The first concerns whether the mechanism exists. One can check its code, tests, documentation and interface.

The second concerns how the mechanism behaves once called. One can check whether, under given conditions, it blocks the operation, records its state correctly and responds to errors.

The third concerns integration: whether the actual execution path calls the mechanism at all.

In the case analysed, the most important problem lay at the third level. Getting the first two right could not solve it.

A safety review should therefore cover not only the component itself but also the path from the system's entry point to the operation it is meant to protect.

The existence of a safeguard is not evidence that the safeguard takes part in how the system works.

This is a reconstruction of a historical case, not a controlled experiment.

The findings concern mainly the structure of the code, the change history and the preserved documents. There is no evidence that the faulty path of the older system executed a real order or caused any harm.

Not all historical conversations and decisions have been preserved. The available material does not allow the full knowledge or motives of the people involved to be reconstructed.

The absence of commits in the repository examined does not prove the absence of activity in all related projects.

The restrictions of the successor system's working demonstration route were confirmed separately. The full effectiveness of all safeguards of the new system was not examined.

For me, the most telling element of this story remains the archiving of the kill switch.

The safeguard could be removed from the executed part of the system without changing its behaviour. That is stronger evidence of the missing integration than the contradiction in the documentation alone.

I do not know all the reasons behind the earlier decisions, and I do not intend to fill them in. The preserved code is enough, however, to show a concrete defect.

A kill switch does not have to be badly written to fail to protect. It is enough that the system never reaches it.