diff --git a/atvm/AGENTS.md b/atvm/AGENTS.md index 83576ac..2e26005 100644 --- a/atvm/AGENTS.md +++ b/atvm/AGENTS.md @@ -78,6 +78,9 @@ This file defines how to operate and maintain the ATVM workspace in `/home/aw/co - When the watcher is requested, start the watcher before `run-sorry-cypress.py`. - Do not start the runner before the watcher, because the watcher helper clears stale `/tmp/.log` and can delete the fresh live runner log if the runner starts first. - For host-level test detail and failed-test investigation, use `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter`, especially `logs/`, `xml/`, and `mochawesome/`. +- When reporting `TEST FLOW:` for an ATVM run, prefer the numbered steps extracted from the generated spec for that exact run. +- If the generated spec exists, do not rely on a static template flow list for `TEST FLOW:`. +- Only fall back to template-level or static flow definitions when the generated spec cannot be located or parsed. - Treat `/var/lib/atvm-run-watcher//state.json` as cached watcher output, not the source of truth for a completed-run confirmation. - Before confirming a completed ATVM run status, verify in this order: live launch log, matching reporter artifacts, `Cloud Run Finished` summary / Currents URL, then compare against saved watcher state. - If saved watcher state disagrees with the launch log or a replay of the exact artifacts through the current watcher code, treat the saved state as stale and do not report from it. diff --git a/atvm/docs/automation/guide.md b/atvm/docs/automation/guide.md index 1b6d919..d29af13 100644 --- a/atvm/docs/automation/guide.md +++ b/atvm/docs/automation/guide.md @@ -274,7 +274,11 @@ Status-report expectations: - If `categorize mode: enabled` is already shown in `COVERAGE:`, do not also repeat `--categorize` under `run options`. - When grouped categorized timing is reconstructed from host reporter artifacts, derive per-host quickest/longest/average durations from the sequence of recovered host timestamps and the grouped end time instead of leaving those metrics as `n/a`. - In `TEST FLOW:`, show the template-specific numbered run flow once for the whole test, not per host. -- Resolve the flow from the run template name. +- For `TEST FLOW:`, treat the generated host spec from the actual run as the source of truth whenever it exists. +- Extract the numbered flow steps from the generated `.ts` spec referenced by that run's `specPattern`. +- Do not prefer a static template flow list over a generated spec from the actual run. +- Use template-level or static fallback flow only when the generated spec cannot be found or parsed. +- If fallback is required, resolve it from the run template name before using any generic default flow. - `cmc-e2e` currently uses the 22-step migration flow documented in `/home/aw/code/cds/atvm/docs/automation/status-template.md`. - `cmc-systemOS` currently uses the 21-step boot-disk migration flow documented in `/home/aw/code/cds/atvm/docs/automation/status-template.md`. - Keep `NOTES:` behavior consistent across template types; do not add template-specific internal-source notes such as parent-log-summary recovery details. diff --git a/atvm/docs/automation/run-learnings.md b/atvm/docs/automation/run-learnings.md index 2045547..beedc8e 100644 --- a/atvm/docs/automation/run-learnings.md +++ b/atvm/docs/automation/run-learnings.md @@ -443,3 +443,11 @@ This file stores run-specific examples only when a run produced a new learning r - Action for future runs: - Add `--test_partition` to `cmc-templates.py` commands by default. - Omit it only when the operator explicitly asks not to use it. + +## Run Learning: 2026-03-30 (Use generated spec as the source of truth for `TEST FLOW:`) +- Observed operator requirement: + - The operator wants the current full workflow steps for the actual test template/run, not a stale hand-maintained flow list. +- Action for future runs: + - Resolve `TEST FLOW:` from the generated `.ts` spec for the actual run whenever that spec exists. + - Extract the numbered `it(...)` steps from the generated spec referenced by the run's `specPattern`. + - Only use template-level or static fallback flow definitions when the generated spec cannot be found or parsed.