Prefer generated specs for ATVM test flow

This commit is contained in:
2026-03-30 20:31:14 -04:00
parent b1b7e89449
commit c64a64ca14
3 changed files with 16 additions and 1 deletions

View File

@@ -78,6 +78,9 @@ This file defines how to operate and maintain the ATVM workspace in `/home/aw/co
- When the watcher is requested, start the watcher before `run-sorry-cypress.py`. - When the watcher is requested, start the watcher before `run-sorry-cypress.py`.
- Do not start the runner before the watcher, because the watcher helper clears stale `/tmp/<build-name>.log` and can delete the fresh live runner log if the runner starts first. - Do not start the runner before the watcher, because the watcher helper clears stale `/tmp/<build-name>.log` and can delete the fresh live runner log if the runner starts first.
- For host-level test detail and failed-test investigation, use `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter`, especially `logs/`, `xml/`, and `mochawesome/`. - For host-level test detail and failed-test investigation, use `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter`, especially `logs/`, `xml/`, and `mochawesome/`.
- When reporting `TEST FLOW:` for an ATVM run, prefer the numbered steps extracted from the generated spec for that exact run.
- If the generated spec exists, do not rely on a static template flow list for `TEST FLOW:`.
- Only fall back to template-level or static flow definitions when the generated spec cannot be located or parsed.
- Treat `/var/lib/atvm-run-watcher/<build>/state.json` as cached watcher output, not the source of truth for a completed-run confirmation. - Treat `/var/lib/atvm-run-watcher/<build>/state.json` as cached watcher output, not the source of truth for a completed-run confirmation.
- Before confirming a completed ATVM run status, verify in this order: live launch log, matching reporter artifacts, `Cloud Run Finished` summary / Currents URL, then compare against saved watcher state. - Before confirming a completed ATVM run status, verify in this order: live launch log, matching reporter artifacts, `Cloud Run Finished` summary / Currents URL, then compare against saved watcher state.
- If saved watcher state disagrees with the launch log or a replay of the exact artifacts through the current watcher code, treat the saved state as stale and do not report from it. - If saved watcher state disagrees with the launch log or a replay of the exact artifacts through the current watcher code, treat the saved state as stale and do not report from it.

View File

@@ -274,7 +274,11 @@ Status-report expectations:
- If `categorize mode: enabled` is already shown in `COVERAGE:`, do not also repeat `--categorize` under `run options`. - If `categorize mode: enabled` is already shown in `COVERAGE:`, do not also repeat `--categorize` under `run options`.
- When grouped categorized timing is reconstructed from host reporter artifacts, derive per-host quickest/longest/average durations from the sequence of recovered host timestamps and the grouped end time instead of leaving those metrics as `n/a`. - When grouped categorized timing is reconstructed from host reporter artifacts, derive per-host quickest/longest/average durations from the sequence of recovered host timestamps and the grouped end time instead of leaving those metrics as `n/a`.
- In `TEST FLOW:`, show the template-specific numbered run flow once for the whole test, not per host. - In `TEST FLOW:`, show the template-specific numbered run flow once for the whole test, not per host.
- Resolve the flow from the run template name. - For `TEST FLOW:`, treat the generated host spec from the actual run as the source of truth whenever it exists.
- Extract the numbered flow steps from the generated `.ts` spec referenced by that run's `specPattern`.
- Do not prefer a static template flow list over a generated spec from the actual run.
- Use template-level or static fallback flow only when the generated spec cannot be found or parsed.
- If fallback is required, resolve it from the run template name before using any generic default flow.
- `cmc-e2e` currently uses the 22-step migration flow documented in `/home/aw/code/cds/atvm/docs/automation/status-template.md`. - `cmc-e2e` currently uses the 22-step migration flow documented in `/home/aw/code/cds/atvm/docs/automation/status-template.md`.
- `cmc-systemOS` currently uses the 21-step boot-disk migration flow documented in `/home/aw/code/cds/atvm/docs/automation/status-template.md`. - `cmc-systemOS` currently uses the 21-step boot-disk migration flow documented in `/home/aw/code/cds/atvm/docs/automation/status-template.md`.
- Keep `NOTES:` behavior consistent across template types; do not add template-specific internal-source notes such as parent-log-summary recovery details. - Keep `NOTES:` behavior consistent across template types; do not add template-specific internal-source notes such as parent-log-summary recovery details.

View File

@@ -443,3 +443,11 @@ This file stores run-specific examples only when a run produced a new learning r
- Action for future runs: - Action for future runs:
- Add `--test_partition` to `cmc-templates.py` commands by default. - Add `--test_partition` to `cmc-templates.py` commands by default.
- Omit it only when the operator explicitly asks not to use it. - Omit it only when the operator explicitly asks not to use it.
## Run Learning: 2026-03-30 (Use generated spec as the source of truth for `TEST FLOW:`)
- Observed operator requirement:
- The operator wants the current full workflow steps for the actual test template/run, not a stale hand-maintained flow list.
- Action for future runs:
- Resolve `TEST FLOW:` from the generated `.ts` spec for the actual run whenever that spec exists.
- Extract the numbered `it(...)` steps from the generated spec referenced by the run's `specPattern`.
- Only use template-level or static fallback flow definitions when the generated spec cannot be found or parsed.