Learn · Engineering case studies

How live systems actually fail.From our own incident reports.

Nobody publishes their real failure analysis — which is a shame, because it is the most useful document a live operation produces. These are genuine post-mortems from our own trading system, with the account specifics removed and the engineering lessons kept. They are not trading tips. They are what production teaches that backtesting cannot.

The seam bug

Two subsystems agreed on everything except the shape of the data between them.

What happened

A component that generated trade signals and a component that consumed them were built at different times, by different sessions of work, against slightly different assumptions about the message format passing between them. Each side was individually correct. Each side's tests passed. But the producer wrote one schema and the consumer parsed another — so a class of perfectly valid signals was silently dropped at the seam. No crash, no error log, no alert: the consumer simply found nothing it recognized and moved on, every single time.

Why nobody noticed

Silence is ambiguous. A strategy producing no trades looks identical to a strategy whose trades are being eaten — from the outside, both are a quiet log. The absence of something is the hardest signal to monitor, because every monitoring instinct is built around events, and this failure produced none. The system was 'working' by every dashboard it had.

How it was caught

By refusing to accept silence at face value. The diagnostic that worked was replaying history through the producer's own code — not a reimplementation, the actual module — and proving that signals should have fired during the quiet window. That converted 'maybe no opportunities occurred' into 'something between A and B is eating signals', which is a bounded search instead of a mystery. From there, walking the seam found the schema mismatch in minutes.

What transfers

Integration points fail differently than components: both sides can be correct and the seam still wrong. Test the seam itself — a message written by the real producer, parsed by the real consumer.
Silence must be made falsifiable. For every 'nothing happened', build a way to answer: is nothing happening, or is something being dropped? Expected-activity checks (heartbeats, minimum-signal alarms) exist for exactly this.
When diagnosing, replay with the system's own code. If you rewrite the logic to test it, you are testing your rewrite, not the system.

The contaminated feedback loop

An automation quietly fed its own output back into the record it was judged by.

What happened

A paper-trading automation — the proving ground where strategies build a forward track record before real money — ended up writing entries into the same record used to evaluate strategies for promotion. The boundary between 'the experiment' and 'the measurement of the experiment' broke down. Track records that looked like independent forward evidence were partially self-generated, which meant the promotion decision was grading homework the automation had helped write.

Why this class of bug is vicious

Nothing is broken in the usual sense: every individual write is valid, every process is running, all tests pass. The corruption is in the meaning of the data, not its format — and the numbers move in a flattering direction, so nothing looks wrong. Contaminated evidence usually looks like good news. This is the trading-system version of training a model on its own test set: the metric keeps improving while its information content quietly goes to zero.

How it was caught, and the fix

By auditing provenance rather than values — asking of the track record not 'are these numbers right?' but 'who wrote each row, and was that writer supposed to?' The fix was structural, not a patch: separate namespaces for experiment output versus evaluation records, explicit tagging of every record's origin, and a rule that anything able to influence a promotion decision must be written only by the measurement path. Then the contaminated history was thrown out — painful, but a shorter track record you can trust beats a longer one you cannot.

What transfers

Draw a hard line between the system under test and the system doing the measuring. If one process can write to both sides, you do not have a measurement — you have a loop.
Audit data by provenance, not just by value. 'Who writes this, who reads it, and is that the design?' catches entire bug classes that value-checking cannot see.
Be most suspicious of metrics that improve on their own. Good news that arrived without a cause is a bug report wearing a party hat.

The gate that passed a lie

A strategy cleared rigorous statistical validation — because the data underneath was poisoned.

What happened

A strategy sailed through a validation gate that had killed dozens before it — strong scores across the board, a clean pass, top-tier grade. The statistics were computed correctly. The methodology was sound. But the historical dataset it was tested against carried a subtle defect that manufactured exactly the kind of pattern the strategy claimed to exploit. The gate did its job perfectly on inputs that were lying to it. Garbage in, rigorously validated garbage out.

Why rigor was not enough

Validation machinery inherits the credibility of its inputs. Deflated Sharpe ratios, cross-validation, overfitting probes — all of it operates downstream of the data, and none of it can detect that the data itself is wrong. This is the quant version of a compiler catching every syntax error while the spec is mistaken: the strongest gate in the world checks the argument, not the premises. Data defects — bad split adjustments, phantom prices, timestamps shifted by a session — are the premises.

How it was caught

The result was too good, and 'too good' was treated as a red flag instead of a victory lap. The autopsy re-derived the strategy's edge from raw, independently sourced data — and the edge evaporated. The pattern lived in the dataset's defect, not in the market. The pass was revoked, the postmortem published, and the pipeline gained a new permanent stage: data-sanity checks that run before any strategy statistics are computed at all.

What transfers

Extraordinary results demand an audit of the data before a celebration of the strategy. The prior for 'I found a huge edge' is that you found a data bug.
Validate the dataset as its own artifact — splits, dividends, gaps, timestamps, duplicates — independently of any strategy. Data checks that only run inside strategy research arrive too late.
Reproduce edges on independently sourced data before believing them. An edge that lives in only one vendor's file is a property of the file.

Module 7 of the curriculum builds a full debugging discipline around incidents like these. The complete strategy-level verdicts — every family tested, every death — live in the research graveyard.

About this curriculum: everything in the Learn section is for educational and informational purposes only. It does not constitute personalized investment advice, a recommendation to buy or sell any security or derivative, or an offer of advisory services. Crash Money Analytics does not know your individual financial situation, and nothing here should be treated as tailored to it. Trading involves substantial risk of loss. Consult a licensed financial advisor before making investment decisions. See our Risk Disclosures.