In quantitative trading and backtesting systems, numerical reproducibility is the bedrock of trust. Given identical code and input data, computed results must never drift on their own.

This post chronicles the investigation of an oracle gate failure. What initially appeared to be an issue with application logic or tolerance configuration ultimately turned out to be hidden entirely outside the repository.

Background

The backtesting project includes an “oracle gate”: a mechanism that freezes a set of statistical outputs into a numerical benchmark. During test execution, current outputs are compared against this frozen benchmark; if any deviation exceeds the defined tolerance, the test is marked as FAIL. The sole purpose of this gate is to guarantee strict reproducibility across backtest runs.

In the baseline frozen on 2026-07-29, the maximum deviation recorded in the acceptance report was 2.27e-13, well within the configured tolerance threshold.

The Issue

During a routine test run, Stage D suddenly flagged a red failure, reporting a numerical drift of 4.10e-9.

The issue was initially logged on the progress board as “contradictory tolerance settings,” accompanied by three proposed remedies:

1. Widen the tolerance
2. Update the docstring
3. Re-freeze the oracle benchmark

All three proposals shared a common underlying assumption: the numerical drift originated from within the repository.

The first option—widening the tolerance—was particularly dangerous. A drift of 4.096e-9 sits right between 1e-9 and 1e-8. Relaxing the tolerance would turn the gate green immediately, but it would also permanently blind the system to numerical drifts of this magnitude.

Fortunately, a single note left on the board pointed in the right direction:

Before taking action, verify: the previous record logged a maximum statistical layer difference of 2.27e-13, but now it is 4.1e-9—a discrepancy of four orders of magnitude.

A gap of four orders of magnitude could not be reasonably explained by adjusting tolerance settings.

Root Cause Investigation

1. Drift Across All Three Stages

A column-by-column breakdown of the data revealed that the drift was not isolated to Stage D; all three stages had experienced drift:

Stage	Report Baseline	Observed
C	2.27e-13	7.32e-11
D	2.27e-13	4.10e-9 (FAIL)
E	5.68e-14	4.11e-11

Notably, the drift was confined to confidence interval fields: ci_lower, ci_upper, ci_paired_*, and mde—the exact locations in the statistics module calling stats.t.ppf. Other metrics, such as EV, std, N, win_rate, and p_paired, maintained discrepancies of ≤1e-13.

A key detail emerged: the maximum difference for std in Stage C was exactly 2.273737e-13, matching the figure in the acceptance report down to the last digit. This confirmed that when the baseline was originally frozen, the maximum error across the entire statistical layer stemmed from std, while the error in the CI fields was even smaller (meaning the CI calculations were extremely precise at the time). The drift was introduced later.

2. Disproving Two Hypotheses

During the investigation, I formulated and subsequently disproved two initial hypotheses:

Hypothesis 1: Was SciPy upgraded in the interim? Checking the package installation timestamp:

scipy-1.16.2.dist-info   Oct 21 20:28:14 2025

The oracle baseline was frozen in late July, whereas SciPy had been installed nine months prior and remained untouched. The same package version running on the same machine could not yield two distinct outputs for identical inputs.

Hypothesis 2: Did the codebase invoke different API execution paths? I questioned whether certain scripts invoked higher-precision APIs while the statistics module fell back to a less accurate stats.t.ppf. I tested eight syntax variations:

stats.t.ppf(0.975, df=111)         = 1.9815667570310707   err=4.3830e-11
stats.t.ppf(0.975, 111)            = 1.9815667570310707   err=4.3830e-11
stats.t.ppf(0.975, df=111.0)       = 1.9815667570310707   err=4.3830e-11
stats.t.isf(0.025, 111)            = 1.981566757031071    err=4.3830e-11
stats.t(111).ppf(0.975)            = 1.9815667570310707   err=4.3830e-11
stats.t.interval(0.95, 111)[1]     = 1.9815667570310707   err=4.3830e-11
special.stdtrit(111, 0.975)        = 1.9815667570310707   err=4.3830e-11
special.stdtrit(111.0, 0.975)      = 1.9815667570310707   err=4.3830e-11

All eight variants yielded identical results. This test failure pointed directly to the root cause: if calling the API in different ways within the same Python interpreter produced identical output, the variance had to reside in the interpreter runtime itself. The final line of the execution output printed .venv.

3. Establishing Ground Truth First

Before determining which environment was correct, I took a decisive step: establishing the ground truth. I used mpmath to evaluate the expression to 40 decimal places of precision:

mpmath  high-precision t(0.975, df=111) = 1.9815667570749010056
  vs oracle  diff: 5.58116e-18
  vs current diff: 4.38303e-11

The comparison proved that the frozen oracle benchmark was mathematically correct, whereas the current environment introduced a larger error.

This finding ran counter to the common intuition that “the most recently executed environment is newer and therefore more accurate.” It also confirmed that choosing to “re-freeze the oracle” would have locked in less accurate values as the new baseline truth.

4. The Final Verdict: Environment Inconsistency

Comparing the two Python environments revealed the discrepancy:

/Volumes/.../.venv/bin/python3   py 3.13.3   scipy 1.17.0   err=2.2204e-16
/Library/Frameworks/.../python3  py 3.13.x   scipy 1.16.2   err=4.3830e-11

The two Python environments hosted different versions of SciPy, causing a full order of magnitude difference in stats.t.ppf precision. The baseline was originally frozen using .venv, whereas subsequent gate checks were executed against the system’s unmanaged python3.

Switching back to .venv and re-running the gate resulted in exit 0, with the maximum discrepancy across all three stages matching the acceptance report digit for digit.

Resolution and Remediation

The root cause was not simply a matter of “remembering to use .venv”—that would only treat the symptom.

The systemic flaw stemmed from two factors:

1. requirements.txt did not pin explicit versions for SciPy and NumPy.
2. The README instructed users to execute unmanaged python3 commands.

Consequently, which environment executed the tests depended on transient shell environment variables unrecorded in configuration files. While a precision variance of this scale has zero impact on final backtest decisions, it violates bit-for-bit reproducibility—the sole reason for maintaining a numerical gate.

The permanent fixes included:

• Explicitly pinning SciPy and related numerical library versions in requirements.txt.
• Updating the README setup and execution instructions to enforce running tests and gates inside a dedicated virtual environment (.venv), removing reliance on unmanaged system states.

Key Takeaways and Retrospective

This debugging process yielded several insights beyond the immediate bug:

1. Vindicating Past Findings

Section 8 of the acceptance report recorded a prior event: during implementation, a 4.096e-9 variance had been reported, leading two external reviews to fail the build. However, the planning phase subsequently dismissed the issue as “non-reproducible,” overruling the failures with the following rationale:

A difference of 4.096e-09 relative to a magnitude of 588 represents roughly 36,000 ULP, far exceeding typical floating-point noise. This proves the existence of a genuine algorithmic difference at the time, which has since been resolved via self-correction.

The analysis appeared rigorous, confident, and mathematically sound.

In reality, neither party was wrong, nor was there any algorithmic discrepancy in the application logic. The implementation and review phases ran against system python3, whereas the planning phase evaluated tests inside .venv—both observed correct outputs for their respective environments. The reviewers, originally assumed to have “read the text without executing the code,” had likely re-run the tests faithfully.

A seemingly airtight ULP argument had mistakenly discredited two reviewers who executed the tests correctly.

2. Equal Confidence, Different Approaches

When the previous investigation encountered 4.096e-9, it concluded with “cannot reproduce on my machine,” backed by ULP calculations. The flaw in that reasoning was not carelessness, but jumping from “unreproducible locally” to “the other party is wrong” without verifying an unstated premise: that the local execution environment matched the external environment exactly.

Facing the red gate this time presented the same temptation to declare who was right or wrong. However, I chose to compute high-precision ground truth via mpmath first. Asking “which value is mathematically correct” takes precedence over asking “who made a mistake.” The same confidence of “my environment is correct” was the fault in the previous round; this time I sidestepped it by anchoring to verified ground truth first.

3. The Trap of Predicted Baselines as Arbiters

In project workflows, the planning phase often pre-computes predicted values within design docs to serve as reference benchmarks for implementation and review. When implementation produces an error, predicted baselines serve as an effective anchor.

However, if the planning phase itself generates incorrect predictions, those baselines become “false anchors.” When implementation yields correct results that conflict with the plan, engineers waste time in cycles of self-doubt and re-verification, unable to confirm correctness because the reference source itself is flawed.

This mechanism acts as an accelerator when the planning phase is correct, but becomes a blind spot when it is wrong—and the process itself cannot distinguish between the two scenarios. Resolving this structural flaw cleanly remains an open challenge.

4. Gates Cannot Self-Verify Their Runtimes

An oracle gate relies on a core premise: “identical code produces identical output.” When that premise is silently undermined by runtime variations, the gate cannot detect the root cause—it can only report numerical drift.

This highlights why widening tolerance thresholds is the most dangerous option: it fails to resolve the underlying issue while disabling the system’s diagnostic capabilities. Once widened, future environment drifts will pass silently without triggering a red gate.