diff --git a/atvm/atvm-automation-guide.md b/atvm/atvm-automation-guide.md index 8924f37..28f0ddf 100644 --- a/atvm/atvm-automation-guide.md +++ b/atvm/atvm-automation-guide.md @@ -150,7 +150,12 @@ Status-report expectations: - Use the live automation VM state when available. - Derive the heading/title from the run `build_name` when available. - Format every machine entry as `machine-name - STATUS`. -- Derive completed-machine status from completed spec results already written to the run log. +- For categorized runs, reconstruct the whole run across all category batches; do not treat the current live category batch as the full run scope. +- Derive completed-machine status from completed spec results already written during the same run. +- Parse all same-run `test-result-*.xml` files, not only machine-named `test-result-atvm*.xml` files. +- When XML filenames are hash-named, extract the machine name from XML contents such as `testsuite file=`, `testsuite name=`, or `testcase name=`. +- Ignore `check-xml-files.ts` XML outputs when counting machine completion because they are bookkeeping steps, not machine runs. +- When multiple same-run XML files exist for one machine, use the most recently written XML for that machine. - Include the run start time in every status response when it can be derived from the run log. - If the run is complete, include the end time and total run time. - If the run is still active, include the elapsed run time so far. diff --git a/atvm/atvm-automation-runs.md b/atvm/atvm-automation-runs.md index 12bba05..0c4e52c 100644 --- a/atvm/atvm-automation-runs.md +++ b/atvm/atvm-automation-runs.md @@ -54,3 +54,23 @@ This file stores run-specific examples only when a run produced a new learning r - Format machine entries as `machine-name - STATUS`. - Keep failure reasons after the machine/status entry when a machine failed. - When giving ETA, explicitly state it is the estimate for completion of the full remaining run. + +## Run Learning: 2026-03-11 (Categorized run status must be reconstructed across batches) +- Observed failure mode: + - `run-sorry-cypress.py --categorize` mutates the active config to the current category batch, so live files such as `specPattern`, `current_vm`, and the newest `/tmp` Cypress JSON only describe the current category, not the full automation run. + - Answering from only the current live batch underreports the run and misses already-finished machines from earlier category batches. +- Action for future runs: + - Reconstruct whole-run status from the generated machine scope plus all machine result artifacts written since the run start time. + - Use the current batch only to identify the live `RUNNING` machine and immediate next machine(s), not as the full run scope. + - Do not answer status requests for categorized runs until earlier category results have been checked as part of the same run. + +## Run Learning: 2026-03-11 (Hash-named XML files still belong to machine runs) +- Observed failure mode: + - Same-run JUnit output is not consistently named `test-result-atvm...xml`. + - Many machine results for the same automation run were written as hash-named files such as `test-result-01fe412894862398d06d9cc4bc7e81a0.xml`. + - Limiting status reconstruction to machine-named XML files causes major undercounting of completed machines. +- Action for future runs: + - Parse all `test-result-*.xml` files written since the run start time, not only `test-result-atvm*.xml`. + - Extract the machine name from XML contents such as `testsuite file=`, `testsuite name=`, or `testcase name=` when the filename does not include the machine name. + - Treat `check-xml-files.ts` XML outputs as bookkeeping steps, not machine results. + - Prefer the most recently written same-run XML per machine when multiple XML files exist for that machine.