anthony.wen/cds-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Tighten ATVM categorized run status verification

Browse Source 
require categorized PASS results to be confirmed from the child run log or Cloud Run Finished summary

forbid treating watcher host_results, grouped XML, or check-xml-files.ts alone as proof of PASS

record the 2026-04-24 false-PASS incident as a run learning
This commit is contained in:
anthony.wen
2026-04-24 08:46:37 -04:00
parent 86e7830d63
commit eb0e171e41
3 changed files with 15 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
atvm/docs/automation/guide.md
View File

@@ -88,6 +88,8 @@ Run ATVM CMC automation tests on the designated automation VM without unintended
- Do not treat the existence of a per-host reporter artifact by itself as proof that the host passed.
- For categorized grouped recovery, prefer the matching per-host reporter JSON or mochawesome result and carry through the real `failures`, `pending`, and failure message instead of assuming `PASS completed`.
- If grouped XML only contains `check-xml-files.ts`, cross-check the grouped result against the per-host reporter artifacts before posting or repeating status for that grouped sub-run.
- Do not report a categorized grouped sub-run as `PASS` from watcher `host_results`, grouped XML, or a lone `check-xml-files.ts` result by itself.
- Before reporting a categorized grouped sub-run as `PASS`, confirm that the matching child batch also passed in the live launch log or the final `Cloud Run Finished` summary for that child run.
- Treat saved watcher state under `/var/lib/atvm-run-watcher/<build>/state.json` as cached status only.
- For completed-run verification, confirm in this order:
- launch log under `/tmp/<build>.launch.log`
@@ -116,6 +118,7 @@ Completed-run verification sequence:
3. Use the `Cloud Run Finished` summary and Currents URL as the final parent-run check when present.
4. Compare that result against saved watcher state.
5. If there is any disagreement, replay the exact artifacts through the current watcher code in an isolated temp state directory before confirming the result.
6. For categorized runs, do not let a `check-xml-files.ts` child result override a failing child batch shown in the launch log or `Cloud Run Finished` summary.
## Config File / Gold Disk Mapping
- `cypress.atvm-config-gold.ts` -> Gold Disk 1

10
atvm/docs/automation/run-learnings.md
View File

@@ -9,6 +9,16 @@ This file stores run-specific examples only when a run produced a new learning r
## Current State
- No run-learning entries recorded yet from `guide.md` source material.
## Run Learning: 2026-04-24 (Categorized watcher false-PASS guardrail)
- Observed failure mode:
- A categorized compute-migration run was incorrectly reported as `PASS` for `atvm121-ubuntu24.04` even though the actual Ubuntu grouped sub-run failed.
- The false `PASS` came from cached watcher `host_results` plus a grouped XML that only contained `check-xml-files.ts` with `failures="0"`.
- The authoritative launch log and `Cloud Run Finished` summary for that same child run showed `1 failing`.
- Action for future runs:
- Do not report a categorized grouped sub-run as `PASS` from watcher state, grouped XML, or `check-xml-files.ts` alone.
- Before reporting a categorized grouped sub-run as `PASS`, confirm that the matching child batch also passed in the live launch log or the final `Cloud Run Finished` summary for that child run.
- If watcher state or grouped XML disagrees with the launch log or child-run summary, treat the cached/grouped result as stale and report from the launch log plus per-host artifacts instead.
## Run Learning: 2026-03-08 (E2E redhat9.7, pure/fc)
- Request:
- template: `cmc-e2e`