From eb0e171e4107f30c56785d7da4cfc1b402541896 Mon Sep 17 00:00:00 2001 From: "anthony.wen" Date: Fri, 24 Apr 2026 08:46:37 -0400 Subject: [PATCH] Tighten ATVM categorized run status verification require categorized PASS results to be confirmed from the child run log or Cloud Run Finished summary forbid treating watcher host_results, grouped XML, or check-xml-files.ts alone as proof of PASS record the 2026-04-24 false-PASS incident as a run learning --- atvm/AGENTS.md | 2 ++ atvm/docs/automation/guide.md | 3 +++ atvm/docs/automation/run-learnings.md | 10 ++++++++++ 3 files changed, 15 insertions(+) diff --git a/atvm/AGENTS.md b/atvm/AGENTS.md index 9f21cad..25ef691 100644 --- a/atvm/AGENTS.md +++ b/atvm/AGENTS.md @@ -94,6 +94,8 @@ This file defines how to operate and maintain the ATVM workspace in `/home/aw/co - Before confirming a completed ATVM run status, verify in this order: live launch log, matching reporter artifacts, `Cloud Run Finished` summary / Currents URL, then compare against saved watcher state. - If saved watcher state disagrees with the launch log or a replay of the exact artifacts through the current watcher code, treat the saved state as stale and do not report from it. - Never confirm a completed ATVM run from `state.json` alone. +- For categorized runs, never report a grouped sub-run as `PASS` from watcher `host_results`, grouped XML, or a lone `check-xml-files.ts` result by itself. +- Before reporting a categorized grouped sub-run as `PASS`, confirm that the matching child batch also passed in the live launch log or the final `Cloud Run Finished` summary for that child run. - If the operator asks for ATVM run status without mentioning Mattermost, respond locally only and do not post externally. - If the operator asks to send ATVM run status to Mattermost, use `MATTERMOST_ATVM_WEBHOOK` and `MATTERMOST_ATVM_CHANNEL` from `/home/aw/code/cds/.env.credentials.local` by default and send the final status only after the run has fully completed, whether the run passed or failed. - Do not call out expected, harmless `systemctl reset-failed ... unit not loaded` output in routine run updates; mention it only if it blocks startup or matters for debugging. diff --git a/atvm/docs/automation/guide.md b/atvm/docs/automation/guide.md index 801a4d1..ef558f0 100644 --- a/atvm/docs/automation/guide.md +++ b/atvm/docs/automation/guide.md @@ -88,6 +88,8 @@ Run ATVM CMC automation tests on the designated automation VM without unintended - Do not treat the existence of a per-host reporter artifact by itself as proof that the host passed. - For categorized grouped recovery, prefer the matching per-host reporter JSON or mochawesome result and carry through the real `failures`, `pending`, and failure message instead of assuming `PASS completed`. - If grouped XML only contains `check-xml-files.ts`, cross-check the grouped result against the per-host reporter artifacts before posting or repeating status for that grouped sub-run. +- Do not report a categorized grouped sub-run as `PASS` from watcher `host_results`, grouped XML, or a lone `check-xml-files.ts` result by itself. +- Before reporting a categorized grouped sub-run as `PASS`, confirm that the matching child batch also passed in the live launch log or the final `Cloud Run Finished` summary for that child run. - Treat saved watcher state under `/var/lib/atvm-run-watcher//state.json` as cached status only. - For completed-run verification, confirm in this order: - launch log under `/tmp/.launch.log` @@ -116,6 +118,7 @@ Completed-run verification sequence: 3. Use the `Cloud Run Finished` summary and Currents URL as the final parent-run check when present. 4. Compare that result against saved watcher state. 5. If there is any disagreement, replay the exact artifacts through the current watcher code in an isolated temp state directory before confirming the result. +6. For categorized runs, do not let a `check-xml-files.ts` child result override a failing child batch shown in the launch log or `Cloud Run Finished` summary. ## Config File / Gold Disk Mapping - `cypress.atvm-config-gold.ts` -> Gold Disk 1 diff --git a/atvm/docs/automation/run-learnings.md b/atvm/docs/automation/run-learnings.md index 35842a0..1ad8aa2 100644 --- a/atvm/docs/automation/run-learnings.md +++ b/atvm/docs/automation/run-learnings.md @@ -9,6 +9,16 @@ This file stores run-specific examples only when a run produced a new learning r ## Current State - No run-learning entries recorded yet from `guide.md` source material. +## Run Learning: 2026-04-24 (Categorized watcher false-PASS guardrail) +- Observed failure mode: + - A categorized compute-migration run was incorrectly reported as `PASS` for `atvm121-ubuntu24.04` even though the actual Ubuntu grouped sub-run failed. + - The false `PASS` came from cached watcher `host_results` plus a grouped XML that only contained `check-xml-files.ts` with `failures="0"`. + - The authoritative launch log and `Cloud Run Finished` summary for that same child run showed `1 failing`. +- Action for future runs: + - Do not report a categorized grouped sub-run as `PASS` from watcher state, grouped XML, or `check-xml-files.ts` alone. + - Before reporting a categorized grouped sub-run as `PASS`, confirm that the matching child batch also passed in the live launch log or the final `Cloud Run Finished` summary for that child run. + - If watcher state or grouped XML disagrees with the launch log or child-run summary, treat the cached/grouped result as stale and report from the launch log plus per-host artifacts instead. + ## Run Learning: 2026-03-08 (E2E redhat9.7, pure/fc) - Request: - template: `cmc-e2e`