Solve Root Causes
Fix the cause, not the symptom.
Overview
Patching symptoms creates recurring incidents. We trace problems to their root, fix them there, and add the guardrail that prevents recurrence.
Why
Root-cause analysis is usually understood as thinking harder about your own code. The most expensive defects were not reachable that way. Twice, a design that was internally consistent, carefully reasoned and reviewed as correct was simply wrong about how something underneath it behaved, and no amount of re-reading our own reasoning would have found it. That sets a limit worth stating plainly: review cannot catch a wrong assumption about a dependency. Only reading the dependency can. Where a correctness or security property rests on how something else behaves, the property is a hypothesis until the primary source has been read. The second recurring root cause is quieter. A policy question that nobody noticed was a question gets answered implicitly, by whichever value happened to be at hand, and the answer is wrong for exactly the case that matters.
Standard
Trace a problem to its root, fix it there, and add the guardrail that makes recurrence unrepresentable rather than merely unlikely. When a property is derived from something else's behaviour, read that thing's own documentation before treating the property as established. When a defect turns out to have been a policy question answered implicitly, name the question and answer it explicitly, in the type or the configuration, where it can be seen. When several individually reasonable steps combine into a failure, the root cause is the missing procedure, not the person who took the last step.
What This Forbids
- Deriving a security property from assumed dependency behaviour
- Treating "it was reviewed" as evidence that a design is correct about its dependencies
- A fix that leaves the same class of mistake writable
- Attributing a compound failure to whoever was nearest the end of it
Examples
- A session-rotation policy keyed on activity rather than age never fired for the sessions that mattered: routine requests refresh activity every few minutes, so a daily user's credential could never reach the threshold, and the only sessions that rotated were ones already idle longer than the window. The fix was small. The finding was that nobody had decided, the policy had been answered by whichever timestamp was at hand.
- An access boundary built on a key prefix, rebuilt around separate containers after reading the provider's actual access semantics. The design had looked correct and had been reviewed as correct.
- A proxy-trust setting enabled the obvious way made the framework read a client-supplied address, letting an attacker present a fresh address per request and walk straight through a login throttle. The safe form counts hops from the trusted end, which cannot be spoofed.
- A byte limit on uploads that constrained the wire but not the decoder, because compression ratio is attacker-controlled. The cap had to move to decoded size.
Counter Examples
- An error arriving unrecognised as a generic server error, which let an attacker manufacture server errors at will and poison the error-rate signal the entire release process depends on. A small translation defect, protecting a monitoring guarantee.
- A secret required to boot, scaffolded into every generated environment file, and read by no code anywhere. A deployment could not start without supplying something that does nothing, and nobody could immediately say whether it was a real requirement never wired up or three lines to delete. That inability was itself the finding.
- A change moved log handling to a background process, which looked safe, and shipped without smoke tests, sign-off or post-deployment validation. It silently broke a separate business-critical feature, and the absence of post-deploy checks delayed detection until it failed in front of users. No single step was the cause; the missing guardrails were.
Lessons Learned
- Review cannot catch a wrong assumption about a dependency. Only reading the primary source can. This happened twice, in unrelated subsystems, and both designs had passed review.
- A default nobody decided is a bug waiting for a specific user.
- When a fix is small and the finding is large, the finding is the deliverable. Record it where the next person will meet the same decision.
- Some root causes are procedural. If a failure required four small omissions to line up, adding a guardrail is the fix and blaming the fourth omission is not.
Learned From
Related Documents· 3
Referenced By· 2
Version History
- v0.1.0
Principle established with placeholder content.
- v0.2.0strengthened
Written from real experience. Added the limit of review, that assumptions about dependencies are invisible to it, and the pattern of policy questions answered implicitly.