anthony.wen/cds-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Improve ATVM failed-host detail recovery

Browse Source
This commit is contained in:
anthony.wen
2026-03-30 21:38:59 -04:00
parent d1a909f9ab
commit 18dcbc89f9
4 changed files with 209 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
atvm/docs/automation/run-learnings.md
View File

@@ -451,3 +451,30 @@ This file stores run-specific examples only when a run produced a new learning r
- Resolve `TEST FLOW:` from the generated `.ts` spec for the actual run whenever that spec exists.
- Extract the numbered `it(...)` steps from the generated spec referenced by the run's `specPattern`.
- Only use template-level or static fallback flow definitions when the generated spec cannot be found or parsed.
## Run Learning: 2026-03-30 (Event-log reporter JSON must not be ignored in non-categorized fallback)
- Observed failure mode:
- A failed non-categorized run still posted/saved host detail as only `1 failures` even though the per-host reporter artifacts preserved the failing step.
- The per-host `.json` artifact used an event-log format with `metadata` plus `tests`, but no top-level `stats` block.
- The watcher ignored that JSON format, fell back to the `.txt`, and lost structured test counts/detail.
- Action for future runs:
- Support the event-log JSON format directly when parsing per-host reporter artifacts.
- In non-categorized fallback, prefer the structured `.json` artifact over the matching `.txt` when they belong to the same run timestamp.
- Recover at least the failing testcase name and a nonzero test count from those artifacts even when the consolidated run log is missing.
## Run Learning: 2026-03-30 (Use `mochawesome` as the rich fallback for host failure detail)
- Observed failure mode:
- The full UI-visible Cypress error text for a failed ATVM host run existed in `cypress/cmcReporter/mochawesome/*.html`, but the lower-fidelity host-level `.json` and `.txt` reporter artifacts only preserved the failing step boundary.
- That made the host detail fall back to a thin summary even though a richer error payload was available on the controller.
- Action for future runs:
- When the consolidated run log is missing, use `mochawesome` as the rich fallback source for per-host failure text before settling for lower-fidelity reporter artifacts.
- Keep the `HOSTS` table compact by showing the failing step plus a short error summary.
- Put the longer trimmed failure excerpt in `NOTES:` instead of dumping the full raw stack trace into the host-detail column.
## Run Learning: 2026-03-30 (Apply rich failed-host detail recovery to every ATVM template)
- Observed operator requirement:
- The same failed-host recovery and formatting rules should apply across all ATVM template runs, not only reboot scenarios.
- If any ATVM test template fails, the result should still recover the best available failure detail and present it consistently.
- Action for future runs:
- Use the same failure-detail recovery order for every ATVM template: consolidated run log, `mochawesome`, structured reporter artifacts, then text reporter artifacts.
- Keep failed-host `Detail` compact and put the longer trimmed excerpt in `NOTES:` for every template type.