The Fill-Model Audit: 5 checks before you trust any backtest

A LowfieldLabs research note. Free to share.

Most backtests don’t fail because the strategy was wrong. They fail because the simulator’s fill model — the few lines of code that decide “you got filled” — doesn’t match how the venue actually executes orders. The strategy never gets tested; a fantasy about execution gets tested instead.

This is the five-step audit we run on every simulator before trusting a number it produces. It takes an afternoon. It has killed more of our own strategies than any market event has.

Check 1 — State the simulated fill condition explicitly

Open the fill-crediting code and write down, in one sentence, the exact event that records a fill. Not what you designed — what the code does. “Fill when the top-of-book bid strictly moves down through our resting level” is a sentence. “Realistic fill logic” is not.

If you can’t produce this sentence from the code in under ten minutes, that’s the first finding.

Check 2 — State the live fill condition

Now write the one-sentence version for the real venue: what event in the actual orderbook executes your order. On a central limit order book, a resting order fills when any contra-side order crosses its level — whether or not your feed has observed the book moving yet. Execution is mechanism-side, not observation-side. Simulators routinely get this backwards, because simulators only know what they’ve observed.

Check 3 — Compare: is the sim condition a strict subset of the live condition?

Put the two sentences side by side. There are three outcomes:

• Sim is stricter (a subset): the sim undercounts fills. A maker bot of ours required the observed top of book to move before crediting a fill; reality filled on any taker cross. Paper said +$83/hr. Live lost $119 in the first eight minutes — the fills the sim counted weren’t the fills the venue delivered.
• Sim is looser: the sim overcounts, crediting fills reality would never give you. This direction is worse, because it manufactures edge that doesn’t exist. A resimulation that fixed a looser entry-fill model flipped 6 of 7 parameter cells from PASS to FAIL.
• Equivalent: rare. Verify it anyway with Check 4.

Either mismatch direction invalidates the backtest. There is no safe side of this error.

Check 4 — Run the fill-count ratio test

Theory aside, measure it. Run the simulator over a small slice of real book-plus-tape data from a period when you know what actually executed, and compare:

ratio = simulated_fill_count / actual_fill_count   (same period, same orders)

If the ratio falls outside [0.5, 2.0], fail the simulator. Do not tune it into the band — the band is a tripwire, not a target. A sim that fills 5–10x more or less often than reality is answering a different question than the one you’re asking.

If you don’t have live fills yet, that gap is itself the answer: your backtest is unvalidated, and the cheapest validation available is a smallest-legal-size live probe, not more simulation.

Check 5 — Enumerate every fill path, then do the power arithmetic

Two checks in one, because they kill strategies together.

5a. Enumerate the artifact paths. Simulators frequently have more than one code path that can credit a fill. In a popular open-source replayer (quantpylib), we found two: a tape-driven path that fills resting orders queue-aware at their limit price — sound — and a book-snapshot path that, on order arrival, walks the displayed ladder and fills at the book’s prices with no queue competition and no tape corroboration. An order posted at a feed-lagged quote isn’t marketable when sent, but after a few hundred milliseconds of flight time it can be — so cancel latency mechanically manufactured fills through the snapshot path. Per-fill attribution showed the damage: at 800ms simulated latency, 90% of reported profit came from the artifact path, buying well through where real fills printed and still “winning” 88% of the time. A headline result that improved with latency — backwards for any real strategy — inverted to the expected ordering the moment the artifact path was disabled.

The rule this establishes: a result that moves monotonically in the wrong direction is an instrument alarm, not a discovery. Audit the instrument before you believe the finding.

5b. Run the power arithmetic before the strategy work. Compute σ/√n against the data that actually exists. In the same engagement, the corrected strategy showed the right sign across every configuration — and nothing was significant: reaching t = 2 required ~4,350 bars of quoting, and the entire available corpus held ~3,800. The honest reading could not be completed on all the data that exists. Five minutes of arithmetic on day one would have shown the candidate was unfalsifiable at realistic effect sizes. Run that arithmetic before the backtest, not after it.

The checklist, printable

1. ☐ Simulated fill condition stated from code, one sentence.
2. ☐ Live fill condition stated from venue mechanics, one sentence.
3. ☐ Subset analysis done; bias direction named; backtest treated accordingly.
4. ☐ Fill-count ratio measured on real tape; inside [0.5, 2.0] or sim failed.
5. ☐ Every fill-crediting path enumerated; PnL attributed per path; any wrong-direction monotonicity traced to its cause. Power arithmetic (σ/√n vs. available data) computed and the claim shown to be establishable.

A backtest that passes all five is worth acting on. Most don’t pass Check 3.

LowfieldLabs runs this audit — and a structured failure-mode audit for LLM agent systems — as a fixed-scope, fixed-price written engagement, delivered in 10 business days. No calls, by design: the entire engagement is a written record you can forward, ticket, and argue with. If this checklist just killed a strategy you were about to size up, the paid version is cheaper than the lesson.