Async Audits of Agent Systems and Backtests

LowfieldLabs sells written audits of systems that are about to cost you money. Two products, fixed scope, fixed price, fixed timeline. Everything is delivered in writing. There are no calls — that’s a design decision, not a limitation, and it’s explained below.

Product 1: Agent System Failure-Mode Audit

For: small teams shipping LLM agent systems — single-agent loops with tool use, or multi-agent topologies with orchestrators, judges, and evals.

What we review: your repo, your logs, and your eval setup, read against a documented failure-mode taxonomy. The taxonomy comes from published post-mortems and our own research corpus, and it covers the failure modes that actually burn production budgets:

• Reward-hacking surfaces — where your agent can satisfy the metric without doing the work, and whether your eval can tell the difference.
• Judge aggregation failure — LLM-as-judge setups where aggregation (mean-of-scores, majority vote, judge-of-judges) hides variance, rewards verbosity, or lets one miscalibrated judge dominate the verdict.
• Diversity collapse — multi-agent systems whose interaction structure contracts exploration instead of broadening it. Authority-in-the-loop, specialist-verticals, and dense everyone-to-everyone topologies are all documented to converge agents on safe generalities. This is the failure mode behind multi-agent research teams burning $1,500/week with zero usable output — we have the post-mortem.
• Prompt-injection exposure — your tool surface and retrieval paths, mapped against what an attacker-controlled document, web page, or tool response can actually reach.
• Eval contamination — benchmark leakage, train/test overlap in your few-shot banks, and evals that measure memorization instead of capability.

Deliverable: a written report containing severity-ranked findings, the evidence for each (file, line, log excerpt), a concrete fix for each, and an effort estimate so you can sequence the work.

Product 2: Backtest Integrity Audit

For: systematic traders and quant researchers at retail scale who are about to size up on the strength of a simulation.

What we review: your simulator or replayer’s fill model — the code path that decides whether your backtest “got filled.” This is where backtests lie. The audit applies a five-step procedure we developed auditing our own blowups:

1. Fill-condition statement — extract, from code, the exact event that credits a fill in simulation.
2. Live-condition statement — the exact event that credits a fill on the real venue.
3. Subset analysis — is the simulated condition a strict subset of the live condition (undercounts fills — the pattern behind a paper bot showing +$83/hr that lost $119 in its first 8 minutes live), or looser (overcounts — the pattern behind replayer results that invert under scrutiny)?
4. Fill-count ratio test — run the sim against a slice of real tape; the sim-to-live fill ratio outside [0.5, 2.0] fails the sim outright.
5. Artifact path enumeration and power arithmetic — every alternate fill path in the codebase (book-snapshot matching, queue-penalty-free crossing, latency-manufactured fills) enumerated and priced; then the statistical power arithmetic — σ/√n against the data that actually exists — run before you trust any effect size.

We found, in a popular open-source replayer, a book-snapshot fill path that manufactured fills out of cancel latency: at 800ms latency, 90% of reported profit came from the artifact, and a result that looked like it improved with latency inverted to the expected ordering once the path was disabled. If your backtest runs on code you didn’t audit line by line, assume it has a path like this.

Deliverable: a written report plus the audit worksheet, filled in for your system — fill condition stated, subset analysis done, ratio test result, artifact paths enumerated, power arithmetic computed for your data corpus.

What’s included

• A written report. Severity-ranked findings, evidence, fixes, effort estimates.
• For backtest audits: the completed fill-model audit worksheet.
• Async written Q&A for the full 10 business days — questions about scope, evidence, or findings, answered in writing within one business day.
• One async recheck within 30 days of delivery: you implement fixes, we verify them in writing.

What’s excluded

• We do not write the fixes. The report tells you what to change and why; implementation is yours.
• We do not re-run your full backtest suite or re-train your evals. We audit the code and the setup; we don’t reproduce your results end to end.
• We do not advise on strategy, model selection, prompts, or positioning. We audit failure modes, not business decisions.
• No ongoing retainer, no monitoring, no Slack channel. Fixed scope, delivered, done.

The no-calls policy

There are no calls. Not a discovery call, not a kickoff call, not a readout call.

This is a feature. Calls produce vague scope, undocumented commitments, and findings that live in someone’s memory. Our entire engagement is written: the intake questionnaire replaces the discovery call, the report replaces the readout, and async Q&A replaces the “quick sync.” You get a record of everything we said, in a form you can forward to your team, attach to a ticket, or argue with. Written findings are falsifiable; verbal findings are not. We audit systems for a living — we hold ourselves to the same standard.

If you need a call to buy this, we are not the right vendor.

Pricing and timeline

• Pilot audit: $1,500. One audit type, one system, up to three specific questions from you. This is the full product at a pilot price for first engagements.
• Standard audit: $5,000. Same deliverable, priority queue, and the 30-day verified-fixed recheck expanded to two findings clusters.

Delivery is 10 business days from receipt of intake materials and access. Payment terms on invoice; work starts when access does.

Who this is NOT for

• Teams that want a consultant embedded in their Slack. We deliver a document, not a presence.
• Teams that want validation. If you’re buying a rubber stamp for a launch decision already made, our findings will be an inconvenience. We report what the code shows.
• Anyone who can’t grant read access. We audit code and logs. Architecture slides and verbal descriptions are not auditable artifacts.
• Large organizations with procurement processes, security review boards, and MSAs that take longer than the audit itself. Our process is built for small teams that can grant repo access this week.
• Trading operations looking for strategy ideas or signals. We audit whether your backtest tells the truth. We do not tell you what to trade.

Contact: intake via the questionnaire. No calls.