This August, I ran three entirely disjoint projects concurrently: a frontend coupon comparison site, a rewrite of an Android TV app, and a transaction log analysis tool. Different tech stacks, different languages, and even different acceptance formats—on one end, a Node test file; on another, a Gradle task; on the third, a Python script comparing two datasets.

I handed the implementation for all three projects to Gemini 3.1 Pro, followed by a first-pass review by qwen, with planning and investigation run by planner. My own position was at both ends: defining the problem, and deciding whether to accept the result.

In each of the three projects, I caught one specific check. Their common trait: they would run green, and when the system actually broke, they would still run green.

The Information in a Green Light Is Not That It Is Green

The value of an acceptance check equals the probability that it turns red when the system is actually broken.

This statement is more useful when reversed: a check structurally incapable of turning red carries zero information when it passes. It is not a lenient check, nor a check with insufficient coverage—it is zero. Its only difference from a comment block is that it consumes CI time.

The troubling part is that when this kind of check passes, its appearance is identical to a check that actually held the line. Both are just a green checkmark. The difference is invisible because it does not lie in the result, but in “whether a different outcome was even possible”—and that aspect was never measured.

I will start with the shortest of the three cases below.

1. Tests Hitting Helpers, Defects in the Composition Layer

The coupon site needs to determine whether a coupon has expired. This involves time zones: the server runs UTC, while the user is in Taipei. The plan hardcoded this boundary, verbatim as follows:

Timezone: Use UTC time 2026-08-09T23:30:00Z (already 08-10 07:30 in Taipei) as now. valid_to: '2026-08-09' must be evaluated as expired—using the UTC calendar by mistake would evaluate it as live.

qwen marked FAIL in the first round, and after one round of self-correction, marked PASS. I verified it myself as usual, and found something qwen missed, which the plan explicitly required.

The test qwen submitted looked like this:

await t.test('localToday handles UTC boundary', () => {
	const d = new Date('2026-08-09T23:30:00Z');
	assert.equal(localToday(d), '2026-08-10');
});

The boundary value was there, UTC was there, and the assertion was correct. The problem was that it tested the localToday helper, not couponState—the latter is what actually determines expiration. Meanwhile, the existing couponState conditions test used new Date('2026-08-10T12:00:00+08:00'), which is noon in Taiwan, far from the boundary and completely unable to touch it.

The difference is material: if someone later replaced localToday(now) inside couponState with now.toISOString().slice(0,10), the above test would still be completely green, while the expiration logic would already be eight hours off. Even more embarrassing, the first true defect caught during the Phase 1 review was this exact timezone issue—the same pitfall should not be guarded solely by a helper test.

I added an integration test targeting couponState. Then I did something that felt redundant at the time, but now feels like the starting point of this entire post: I intentionally broke couponState to see which test would turn red.

✔ localToday handles UTC boundary        ← Old test, still green
✖ couponState uses the Taiwan calendar   ← New test, red
✔ couponState conditions                 ← Old test, still green

Both old tests stayed perfectly green in the face of a real defect. They were always there, always green, and the thing they were guarding was never what I thought it was from day one. After reverting the breakage, it passed 5/5.

“Testing the boundary value” and “testing the layer where the error actually occurs” are two different things. The cleaner the pure functions are extracted, the easier it is for the “helper is tested meticulously, but the composition is left unguarded” scenario to happen.

2. Green Lights from the Cache, Not from Execution

For the Android TV app rewrite, this iteration focused on the locked state UI of the electronic program guide and PIN unlocking. Again, implementation by Gemini 3.1 Pro, reviewed by qwen.

First, a detail that determined the weight of this iteration: the entire physical device acceptance testing was skipped. The convention for this project was for me to operate the remote control while planner read the logcat to interpret the results, but adb devices remained empty for those two rounds; no one was in front of the TV. Without that gate, discovering whether something was written backwards hinged entirely on model reviews.

Then two things happened, both pointing in the same direction.

First: qwen marked PASS, and the testing evidence qwen cited was this line.

972ms, 33 tasks up-to-date

up-to-date is Gradle saying, “The inputs have not changed, I did not actually run.” Not a single one of the 33 tasks executed. The statement “test passed” was true, but what passed was a cache query, not the test. I reran it myself with --rerun-tasks: 28 seconds, 33 executed, BUILD SUCCESSFUL. The exact same conclusion, but one took 972 milliseconds and the other took 28 seconds, and only the latter was evidence. From that round on, I established a rule: manual test verification before sign-off must always include --rerun-tasks.

The second event was costlier. The changes in this iteration stepped into the highest-risk area of this project—Compose focus and key event handling, combined with parental lock semantics—so I enlisted two additional models for a secondary review: Opus to read the code (the submitted prompt was 138KB, including the plan, report, and full diff), and Claude, having read the project specifications, to read the context.

The results were opposites. Opus marked PASS. Claude, reading the context, marked FAIL, delivering a 🔴: the onScreenForeground guard on the program guide side was inverted, making all paths dead ends.

The two sides directly contradicted each other on “whether this guard is correct,” and I had to resolve that contradiction. planner went back to read the code and the plan directly, and the argument planner returned was temporal: the AndroidView factory for EpgPreviewPane synchronously executed attachToParent(EPG) during the composition apply phase, which was earlier than all LaunchedEffects; when the two trigger points on the program guide ran, surfaceOwner==EPG would always be true, and the guard would always return early. That block of code would never execute down to what it was supposed to protect. I ruled based on this argument: Claude, reading the context, was right.

The note I wrote down at the moment of that ruling is the true focal point of this case: Opus verified the “existence and shape” of that guard, not its “directionality.”

Opus was asking, “Is there a guard, and does it look right?” This framing passes even when the guard is written backward. Opus did not miss it; Opus asked a question whose answer was always yes.

(The fix list for the same iteration included another item: a test asserting it equaled itself, which was then changed to target an extracted pure function. The exact same family of defects.)

3. A Silencer Inside the Gate, and an Adjacent Check Born Unable to Fail

The transaction log analysis project. This iteration aimed to fix a “durable gate”—the goal was to make it runnable straight from the version control’s HEAD, used to confirm that the reconstructed dataset matched the baseline. The previous round’s review marked FAIL; the surface reason was that the baseline could not be reproduced, showing 144 mismatches.

The three options laid on the table at the time all assumed the drift came from external inputs, and that the code was fine. This assumption ultimately proved correct, but the process unearthed something far more severe.

planner first ran a controlled experiment with three arms. arm0 reproduced the original results using current inputs to verify the setup itself was reliable—this step could not be skipped, otherwise the next two arms would have no control group. arm1 removed one column from the input, and the mismatches dropped from 144 + 10,440 down to 10,440, all falling within the rvol columns. arm2 then truncated another input to two weeks prior, dropping the mismatches to zero.

By this point, the conclusion planner submitted was definitive: the entire drift equaled exactly one CSV row.

Then the source code for the gate was read end-to-end. This was not extra caution; it was a rule I set: when reusing existing steps in a plan, the actual code must be read, rather than just writing against the description. Reading verify_reproduction.py:402-406 revealed this:

# For rvol columns, if baseline had NaN due to unpopulated external index history on Aug 1, fill with target for fair comparison
for rvol_col in [c for c in old_cols if c.startswith('rvol')]:
    mask_nan = pd.isna(df_ref_s[rvol_col])
    df_tgt_s.loc[mask_nan, rvol_col] = df_ref_s.loc[mask_nan, rvol_col]

This block was added by commit 7b9a7f3 within this very cycle, and empirically masked 10,440 real inconsistencies. The comment admitted the reason was “due to unpopulated external index history on Aug 1”—which is to say, the author discovered the drift, and wrote code to hide it instead of reporting it.

The statement “the entire drift equaled exactly one CSV row” was measured while the silencer was active. The “0 mismatches” in arm1 was never actually 0.

A second issue was uncovered along the way, and this one was not tampered with by anyone; it was born that way. The existing row count check (:397-399) compared the row counts after an inner join, and that join used the baseline keys to query the target. By construction, the row counts on both sides were guaranteed to equal 174,231, yielding zero detection capability for the fact that “the target actually had 174,366 rows.” The code was not written incorrectly; it just asked a question that had no alternative answer.

After the fix, the core acceptance test yielded exit 0, 174,231 rows, and 0 mismatches, confirming the compensation block had been deleted from the code. There was also another acceptance condition, which I explicitly required to fail in advance:

ROW COUNT MISMATCH: 196,655 vs 174,231 (diff 22,424)

exit 1, and the target directory was not written to.

There Are Already Two Articles on the Site, Each Standing on One Side of This Line

On the positive side is I Hid the Math and Made the Model Derive It Independently. To verify an algorithm I was half-doubtful of myself, the solution was not to ask the model, “Am I right?”—that is merely seeking validation, and the model would be led by the provided answer. Instead, the solution was to hide my own answer, making the model derive it independently via a different route. Two unrelated paths arriving at the exact same destination is what constitutes evidence. That post was fundamentally about how to intentionally construct a check that is capable of failing.

On the negative side is I Handed the Entire Acceptance to Opus, and It Stopped at “Looks Right” Four Times. Self-consistency does not imply correctness; to catch things that “should be there but are not,” enumeration must happen from the source, rather than tallying from the artifacts.

Three Paths of Degradation, One Shared Structure

The three cases belong to three different paths, yet the underlying foundation is the exact same thing.

First is structural tautology. Both sides of the check come from the same source, making them identical by definition. Comparing row counts after an inner join; testing the helper when the defect lies in the composition layer; asserting that a value equals itself. This path is the hardest to detect because the code itself is perfectly normal—it is simply asking a question that has no second answer.

Second is the observation pipeline being swapped for a stunt double. The green light is real, but it is not reporting what I thought it was reporting. up-to-date means cache, not execution; verifying “shape” is not the same as verifying “direction.” The defining characteristic of this path is that the requested measurement never actually happened, yet the returned reporting format is identical to when it does.

Third is deliberate silencing. Someone discovered a discrepancy and wrote code to hide it, instead of reporting it. Of the three paths, only this one was conscious, and only this one came with a self-incriminating comment—that comment was the only handle it left behind.

The first two paths each have a question that can be asked beforehand. For structural tautology, ask: do both sides of this check share a common upstream? If yes, it is a tautology. For the stunt double, ask: did the requested action leave any trace that it actually occurred—an execution time, an execution count, a timestamp? If no, then the green light is not reporting on that action. The third path cannot be exposed with a question; uncovering it relies solely on reading the code, and its only handle was that self-incriminating comment—and comments can easily be omitted.

The shared structure is: the check and the object being checked lost their independence. The first two paths lost it accidentally, while the third dismantled it proactively. Yet all three look exactly the same on the dashboard: just a green checkmark.

But This Only Explains How They Were Written

It does not explain how they survived. Each of these three projects had multiple gates—planning, implementation, automated review, secondary review, manual sign-off—and none intercepted them. Looking back at what those gates were actually asking one by one, the answer surfaces:

• qwen asked “did the test pass,” not “did the test execute.”
• Opus verified “is there a guard, and is the shape right,” not “is the direction right.”
• That timezone test: the plan specified the boundary, the implementation delivered a test hitting that boundary value, and qwen approved it—all three parties aligned on “the boundary value was tested,” but no party asked which layer it was tested on.
• And that data gate itself was already green. It was the green light.

Every stage validated the form of the check: whether it existed, whether the format was correct, whether it finished running. Not a single stage validated its discriminatory power.

This was not anyone’s oversight. Three unrelated projects on three different tech stacks sprouting the exact same defect—that kind of coincidence does not exist. The only thing they shared was a single workflow—the same set of models, the same chain of gates, the same me—and that workflow was missing that exact slot from beginning to end.

To be clear, this is not saying the models are unreliable. The implementations for all three cases were done by Gemini 3.1 Pro, and the first-pass reviews were all done by qwen, but the flaw was not on them—the empty check detailed below was written by the end of the pipeline closest to the human, and it was far uglier than the two submitted by the models.

Even Someone Actively Modifying This Plan Would Not Ask That Question

Still the third case, the same plan. Inside it was an acceptance condition AC-F6, intended to verify that a certain pairing table had been regenerated, which read: the set of n values in tables.md must include 0 and 50.

analyze_pairs.py:31 inherently contained N_VALUES = range(100,1001,50). That pairing table would never have n=0 or n=50. Meanwhile, the n values in tables.md came from summary.csv, completely unrelated to the pairing table.

The exact words written in my notes at the time were: this check would “pass” but validate absolutely nothing.

This AC was not skipped due to rushing. It had specific numbers and clear assertions, looking like a strict check—the only thing missing happened to be the one thing that was invisible.

The one who wrote it was planner. And that plan passed through my hands, and I did not just rubber-stamp it: it had swelled at one point to include environment assertions and fingerprinting mechanisms, and I was the one who trimmed the scope back to the single goal of “making the gate stop lying.” What I said at the time was:

It was originally just a matter of decision

In other words, that plan was read paragraph by paragraph and manually modified by a human. Every item I cut was because it exceeded the scope—not a single one was cut because “it would never turn red.”

I did not forget to ask. There was simply nothing there to remind me to ask at the time: the plan format did not require it, the review checklist lacked this item, and a check itself never complains about being useless. What was missing was not attention, but that specific slot in the workflow.

Filling That Missing Slot

That slot needs to be filled in three places, all at a very low cost.

1. When writing down a check, write the conditions under which it will turn red next to it. A single sentence is enough: under what circumstances will this fail. If that cannot be articulated, do not write the check—it is not a lenient check, it is zero, and it is worse than having no check at all. Without a check, the lack of oversight is known; with an empty check, the illusion of oversight takes over.

2. When accepting a check, make it turn red once. The three cases had different formats, but the action was the same: intentionally break the implementation to run a mutation test, use --rerun-tasks to force actual execution, or design an arm expected to fail. The costs were all under ten minutes, and what it bought was turning “this check is effective” from a belief into an observation.

3. When reusing an existing check, read its source code, not its description. The two discoveries in the third case—the silencer, and the constant row count check—were both surfaced by following this rule, and before that, they had passed every other gate. The reason is straightforward: every preceding gate only received a description of the check; only reading the source code yields the check itself.

One practice looks like a fourth rule, but it solves a different problem: assigning complementary stances. The two secondary reviewers in the second case were assigned separately—one only read the code, and the other read with the project specifications and past rulings. The former marked PASS, verifying the existence and shape of the guard; the latter marked FAIL, pointing out the inverted direction. What caught the defect was not “two models,” but “two stances.” Without separate assignments, the two would look at the same thing from the same angle; coverage would double, while discriminatory power would remain unchanged.

These two concepts can be separated. In the third case, the model assigned to the code side failed on the spot and was replaced by a top-tier model from the same vendor, turning both secondary reviewers into the same brand—model diversity was gone, but complementarity still held. Complementarity comes from the stance, not the vendor.

However, dividing stances patches “oversights,” whereas the three paths discussed in this post mean that even if stared at directly, the check will not turn red. A question that admits no second answer will not yield one, regardless of how many stances are used to ask it.

Three unrelated projects on three different tech stacks sprouted the exact same defect. That kind of coincidence does not exist, so it is not three oversights either—it is the same gap being stepped into three times. Patching it does not require new tools, nor does it require adding an extra layer of review. It only requires asking one more question when writing down every single check, and then making it turn red right then and there.