Appendix E: Validity Incidents
Section 7.3 draws six methodological lessons from the threats to validity that the study encountered. Table E.1 gives the dated record of all eighteen incidents: how each was detected, what it changed and how it was corrected. Results in Sections 6.2 to 6.7 were obtained after the corresponding correction. There is one exception: the Qwen3.6-35B cell in row 5 of Table 6.5 (incident 9). Results affected by incidents 12 and 17 were re-expressed rather than re-run.
Table E.1: Validity incidents in the order in which they were found.
| # | Found | Incident | Detection | Effect on results | Correction |
|---|---|---|---|---|---|
| 1 | 2026-09-20 | Annotation placeholder. The effective cost of gateway fights without lookahead value was printed as None; an audit replaced it with the numeric damage, assuming a formatting error. | The first 20 repeated episodes (Qwen3.5-397B and Qwen3.6-35B) won none, against 6 of 14 stored episodes; the stored reasoning read the placeholder as "not recommended yet". | 0 of 20; after reverting, 5 of 10 per model. The 20 episodes are kept as an unintended ablation. | Placeholder restored; a regression test pins the observation text at a reference decision. |
| 2 | 2026-09-20 | Two evaluation loops. The command-line loop and the platform loop applied different overrides under the same name "reactive"; all stored wins came from the platform loop. | Code review during incident 1. | The first comparison of new and stored episodes was confounded. | The harness uses the platform loop; every result records its loop. |
| 3 | 2026-09-20 | Wrong default prompt. A validation script mapped the simple floor to the prompt written for floors with shops. | Provenance check of the stored wins. | A hidden factor between runs. | The engine's default prompt per floor; every result records its prompt file. |
| 4 | 2026-09-21 | Stale process. A child process of an aborted batch continued an obsolete configuration next to the corrected one. | The result file recorded the old prompt; the time per episode halved when the process ended. | Three invalid episodes and about 2.5 hours of contention for the API. | Process stopped, results quarantined, cell re-run. |
| 5 | 2026-09-21 | Scorer gating. The penalty for a surviving boss applied only to floors with a stored reference. | Unit test written for the success criterion. | None on stored results; a new boss floor would have scored a surviving boss as a clean run. | Penalty keyed on the floor's goal. |
| 6 | 2026-09-21 | Models removed from the service. Three models of the early experiments (Qwen3.5-35B, Qwen3.5-122B, Qwen3.5-27B) were removed from the service. | Availability check against the service's model list. | Early model comparisons and the first complex-floor wins cannot be reproduced; the successor Qwen3.6-35B won 0 of 10 on the prompt with which its predecessor had won 2 of 18. | Only served models are used; the old presets are kept so that historical labels resolve. |
| 7 | 2026-09-24 | Silent strategic overrides. The strategic override replaced the model's choice whenever the heuristic judged another action much cheaper, without logging it. | Comparison of the model's answer with the executed action in every stored log. | 94 replaced turns in 68 episodes; all 21 stored simple-floor wins contained a replacement (Section 6.4.1). | Overrides reduced to mechanical cases and logged per turn; strategic rules only on explicit request. |
| 8 | 2026-09-25 | Annotations hidden with the overrides. Switching the overrides off also removed the annotations of fight actions, while door actions kept theirs. | A drop from 4 of 5 to 1 of 5 on the complex floor after a change to the door annotations (results not reported in Chapter 6). | Earlier results with overrides off were obtained with less information than reported. | Separate settings for overrides and annotations. |
| 9 | 2026-09-26 | Repair chose an unwinnable fight. With a reasoning cap of 8,192 tokens, truncated answers were repaired by constrained decoding, which chose the unwinnable boss on 12 turns of one episode. | Log inspection of a 1-of-5 result. | That cell measured the heuristic fallback rather than the model. | The repair call only offers actions the mechanical overrides would not replace; cap raised to 16,384. |
| 10 | 2026-09-26 | Crash on door-only action lists. A variable of the annotation code was undefined when no fight was listed. | Two tower runs ended with an error. | Both runs void and re-run. | Code fixed and covered by a test. |
| 11 | 2026-09-26 | Concurrency. Four experiments in parallel on one API key slowed Qwen3.5-397B from about 20 s to about 320 s per call. | Latency census. | No invalid result from the slowdown itself (the invalid cell of incident 4 came from a stale process), but runs of a day. | At most two concurrent experiments per key. |
| 12 | 2026-09-29 | Reference solver. The published solver keeps, for each set of cleared obstacles, the first state it expands instead of the best (Section 4.6.2). | An exact re-solve during the construction of the hand-designed floors. | The complex floor's optimum is 68, not 22; every claim of an optimal complex-floor win was an artefact. | Exact solver; all stored results re-expressed against 68. |
| 13 | 2026-09-29 | Planning harness. Four defects of the committed-plan loop (Appendix D.6). | Reading the logs of finished cells. | Several planning cells measured reactive or heuristic play. | Each defect fixed; affected results excluded. |
| 14 | 2026-09-29 | API quota. The daily request limit of the second API key was exhausted during a batch. | Empty answers and fallback turns in two planning cells. | Both cells (DeepSeek-V4-Flash, complex floor, commit-3 and commit-5) belong to the excluded results of Table D.7 (Appendix D.6). | Batches moved to the first key. |
| 15 | 2026-10-01 | Tower call count. The field llm_calls of tower result files counted every turn not taken by the engine, including executed plan steps. | Recount of the model calls from the reasoning events of the run logs. | The call counts first reported for the tower planning comparison were counts of decided turns. | Calls counted from the run logs; result files store model_calls from then on (Appendix D). |
| 16 | 2026-10-03 | Purchase enumeration. The in-order way-up line priced every combination of up to ten purchases of each of eight shop offers for each fight on the route. | A tower run of Qwen3.5-397B with a predecessor of the final observation slowed to about 80 minutes per turn from turn 560, on MT15 with 1,940 gold and 3,034 EXP unspent. | The run was stopped after reaching MT16 and has no result file. | Exact table of the gain per currency; the in-order line enumerates at most 20,000 combinations and otherwise prices ATK-only and DEF-only purchases. |
| 17 | 2026-10-04 | Hidden staircases. The up-staircases of MT18 and MT20, which the game hides until the hero talks to the princess and hands in the Cross, were shown from the start. | Tile-by-tile check of the floor files of the game data against the engine. | A tower run of Qwen3.5-397B with a predecessor of the final observation reached "MT21" instead of MT18; its highest floor was overstated by three floors. | Staircases hidden until their scripted events (Table B.4); the run's arrival on MT18 is the stored position continued in Section 6.7.3. |
| 18 | 2026-10-05 | Misattributed losing decision. The failures on the complex floor with the single-floor prompt were attributed to the yellow door (10,5), the conspicuous error that the solver-route prompt calls a trap. | Exact state value of every state of every stored complex-floor episode, computed with the exact solver. | No success rate changed; all 30 failures had been lost at an earlier fight, 29 of them at the Red Slime (11,8), and the door came after the loss. | Section 6.4.5 reports the decision after which the boss becomes unwinnable (Table B.3, Figure 6.2). |
Incidents 1 to 6 were found during the first campaign of repeated episodes.