Update ATVM automation status reporting guidance

atvm/atvm-automation-guide.md

@@ -150,7 +150,12 @@ Status-report expectations:
 - Use the live automation VM state when available.
 - Derive the heading/title from the run `build_name` when available.
 - Format every machine entry as `machine-name - STATUS`.
-- Derive completed-machine status from completed spec results already written to the run log.
+- For categorized runs, reconstruct the whole run across all category batches; do not treat the current live category batch as the full run scope.
+- Derive completed-machine status from completed spec results already written during the same run.
+- Parse all same-run `test-result-*.xml` files, not only machine-named `test-result-atvm*.xml` files.
+- When XML filenames are hash-named, extract the machine name from XML contents such as `testsuite file=`, `testsuite name=`, or `testcase name=`.
+- Ignore `check-xml-files.ts` XML outputs when counting machine completion because they are bookkeeping steps, not machine runs.
+- When multiple same-run XML files exist for one machine, use the most recently written XML for that machine.
 - Include the run start time in every status response when it can be derived from the run log.
 - If the run is complete, include the end time and total run time.
 - If the run is still active, include the elapsed run time so far.

atvm/atvm-automation-runs.md

@@ -54,3 +54,23 @@ This file stores run-specific examples only when a run produced a new learning r
   - Format machine entries as `machine-name - STATUS`.
   - Keep failure reasons after the machine/status entry when a machine failed.
   - When giving ETA, explicitly state it is the estimate for completion of the full remaining run.
+
+## Run Learning: 2026-03-11 (Categorized run status must be reconstructed across batches)
+- Observed failure mode:
+  - `run-sorry-cypress.py --categorize` mutates the active config to the current category batch, so live files such as `specPattern`, `current_vm`, and the newest `/tmp` Cypress JSON only describe the current category, not the full automation run.
+  - Answering from only the current live batch underreports the run and misses already-finished machines from earlier category batches.
+- Action for future runs:
+  - Reconstruct whole-run status from the generated machine scope plus all machine result artifacts written since the run start time.
+  - Use the current batch only to identify the live `RUNNING` machine and immediate next machine(s), not as the full run scope.
+  - Do not answer status requests for categorized runs until earlier category results have been checked as part of the same run.
+
+## Run Learning: 2026-03-11 (Hash-named XML files still belong to machine runs)
+- Observed failure mode:
+  - Same-run JUnit output is not consistently named `test-result-atvm...xml`.
+  - Many machine results for the same automation run were written as hash-named files such as `test-result-01fe412894862398d06d9cc4bc7e81a0.xml`.
+  - Limiting status reconstruction to machine-named XML files causes major undercounting of completed machines.
+- Action for future runs:
+  - Parse all `test-result-*.xml` files written since the run start time, not only `test-result-atvm*.xml`.
+  - Extract the machine name from XML contents such as `testsuite file=`, `testsuite name=`, or `testcase name=` when the filename does not include the machine name.
+  - Treat `check-xml-files.ts` XML outputs as bookkeeping steps, not machine results.
+  - Prefer the most recently written same-run XML per machine when multiple XML files exist for that machine.