Prefer generated specs for ATVM test flow
This commit is contained in:
@@ -78,6 +78,9 @@ This file defines how to operate and maintain the ATVM workspace in `/home/aw/co
|
||||
- When the watcher is requested, start the watcher before `run-sorry-cypress.py`.
|
||||
- Do not start the runner before the watcher, because the watcher helper clears stale `/tmp/<build-name>.log` and can delete the fresh live runner log if the runner starts first.
|
||||
- For host-level test detail and failed-test investigation, use `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter`, especially `logs/`, `xml/`, and `mochawesome/`.
|
||||
- When reporting `TEST FLOW:` for an ATVM run, prefer the numbered steps extracted from the generated spec for that exact run.
|
||||
- If the generated spec exists, do not rely on a static template flow list for `TEST FLOW:`.
|
||||
- Only fall back to template-level or static flow definitions when the generated spec cannot be located or parsed.
|
||||
- Treat `/var/lib/atvm-run-watcher/<build>/state.json` as cached watcher output, not the source of truth for a completed-run confirmation.
|
||||
- Before confirming a completed ATVM run status, verify in this order: live launch log, matching reporter artifacts, `Cloud Run Finished` summary / Currents URL, then compare against saved watcher state.
|
||||
- If saved watcher state disagrees with the launch log or a replay of the exact artifacts through the current watcher code, treat the saved state as stale and do not report from it.
|
||||
|
||||
@@ -274,7 +274,11 @@ Status-report expectations:
|
||||
- If `categorize mode: enabled` is already shown in `COVERAGE:`, do not also repeat `--categorize` under `run options`.
|
||||
- When grouped categorized timing is reconstructed from host reporter artifacts, derive per-host quickest/longest/average durations from the sequence of recovered host timestamps and the grouped end time instead of leaving those metrics as `n/a`.
|
||||
- In `TEST FLOW:`, show the template-specific numbered run flow once for the whole test, not per host.
|
||||
- Resolve the flow from the run template name.
|
||||
- For `TEST FLOW:`, treat the generated host spec from the actual run as the source of truth whenever it exists.
|
||||
- Extract the numbered flow steps from the generated `.ts` spec referenced by that run's `specPattern`.
|
||||
- Do not prefer a static template flow list over a generated spec from the actual run.
|
||||
- Use template-level or static fallback flow only when the generated spec cannot be found or parsed.
|
||||
- If fallback is required, resolve it from the run template name before using any generic default flow.
|
||||
- `cmc-e2e` currently uses the 22-step migration flow documented in `/home/aw/code/cds/atvm/docs/automation/status-template.md`.
|
||||
- `cmc-systemOS` currently uses the 21-step boot-disk migration flow documented in `/home/aw/code/cds/atvm/docs/automation/status-template.md`.
|
||||
- Keep `NOTES:` behavior consistent across template types; do not add template-specific internal-source notes such as parent-log-summary recovery details.
|
||||
|
||||
@@ -443,3 +443,11 @@ This file stores run-specific examples only when a run produced a new learning r
|
||||
- Action for future runs:
|
||||
- Add `--test_partition` to `cmc-templates.py` commands by default.
|
||||
- Omit it only when the operator explicitly asks not to use it.
|
||||
|
||||
## Run Learning: 2026-03-30 (Use generated spec as the source of truth for `TEST FLOW:`)
|
||||
- Observed operator requirement:
|
||||
- The operator wants the current full workflow steps for the actual test template/run, not a stale hand-maintained flow list.
|
||||
- Action for future runs:
|
||||
- Resolve `TEST FLOW:` from the generated `.ts` spec for the actual run whenever that spec exists.
|
||||
- Extract the numbered `it(...)` steps from the generated spec referenced by the run's `specPattern`.
|
||||
- Only use template-level or static fallback flow definitions when the generated spec cannot be found or parsed.
|
||||
|
||||
Reference in New Issue
Block a user