anthony.wen/cds-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Tighten ATVM categorized run status verification

Browse Source 
require categorized PASS results to be confirmed from the child run log or Cloud Run Finished summary

forbid treating watcher host_results, grouped XML, or check-xml-files.ts alone as proof of PASS

record the 2026-04-24 false-PASS incident as a run learning
This commit is contained in:
anthony.wen
2026-04-24 08:46:37 -04:00
parent 86e7830d63
commit eb0e171e41
3 changed files with 15 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
atvm/docs/automation/guide.md
View File

@@ -88,6 +88,8 @@ Run ATVM CMC automation tests on the designated automation VM without unintended
- Do not treat the existence of a per-host reporter artifact by itself as proof that the host passed.
- For categorized grouped recovery, prefer the matching per-host reporter JSON or mochawesome result and carry through the real `failures`, `pending`, and failure message instead of assuming `PASS completed`.
- If grouped XML only contains `check-xml-files.ts`, cross-check the grouped result against the per-host reporter artifacts before posting or repeating status for that grouped sub-run.
- Do not report a categorized grouped sub-run as `PASS` from watcher `host_results`, grouped XML, or a lone `check-xml-files.ts` result by itself.
- Before reporting a categorized grouped sub-run as `PASS`, confirm that the matching child batch also passed in the live launch log or the final `Cloud Run Finished` summary for that child run.
- Treat saved watcher state under `/var/lib/atvm-run-watcher/<build>/state.json` as cached status only.
- For completed-run verification, confirm in this order:
- launch log under `/tmp/<build>.launch.log`
@@ -116,6 +118,7 @@ Completed-run verification sequence:
3. Use the `Cloud Run Finished` summary and Currents URL as the final parent-run check when present.
4. Compare that result against saved watcher state.
5. If there is any disagreement, replay the exact artifacts through the current watcher code in an isolated temp state directory before confirming the result.
6. For categorized runs, do not let a `check-xml-files.ts` child result override a failing child batch shown in the launch log or `Cloud Run Finished` summary.
## Config File / Gold Disk Mapping
- `cypress.atvm-config-gold.ts` -> Gold Disk 1