fix watcher failure detection for host reporter json

Handle dict-shaped reporter events when deriving watcher host failures.

- parse reporter JSON events with type/message/severity fields
- preserve existing support for list-shaped event records
- record the false-PASS failure mode in ATVM automation run learnings
This commit is contained in:
2026-04-29 12:37:48 -04:00
parent 9673d769e2
commit 2e0acb69c1
2 changed files with 18 additions and 4 deletions

View File

@@ -18,6 +18,15 @@ This file stores run-specific examples only when a run produced a new learning r
- Write the template phase output to `/tmp/<build>.launch.log` so template activity is preserved separately from the live runner log.
- If the template step fails, stop immediately and do not start the watcher or the runner.
## Run Learning: 2026-04-29 (Watcher host-artifact parser must handle dict-shaped reporter events)
- Observed failure mode:
- A non-categorized ATVM compute-migration run failed in the host reporter artifacts, but the watcher posted `PASS`.
- The watcher fell back to the per-host JSON artifact after `check-xml-files.ts`, but `extract_failure_from_reporter_events()` only recognized the older list-shaped event format.
- Current reporter JSON stores events as dicts with fields such as `type`, `message`, and `severity`, so the parser missed `severity: error` and incorrectly returned `0 failures`.
- Action for future runs:
- Treat both list-shaped and dict-shaped reporter event records as valid inputs when extracting failure details from host JSON artifacts.
- Continue treating host reporter artifacts as authoritative fallback evidence when final XML only contains `check-xml-files.ts`.
## Run Learning: 2026-04-24 (Categorized watcher false-PASS guardrail)
- Observed failure mode:
- A categorized compute-migration run was incorrectly reported as `PASS` for `atvm121-ubuntu24.04` even though the actual Ubuntu grouped sub-run failed.

View File

@@ -524,11 +524,16 @@ def extract_failure_from_reporter_events(testcase_name: str, testcase_events: ob
if not isinstance(testcase_events, list):
return None
for event in testcase_events:
if not isinstance(event, list) or len(event) < 3:
if isinstance(event, dict):
event_type = str(event.get("type", "")).lower()
message_value = str(event.get("message", ""))
status_value = str(event.get("severity", "")).lower()
elif isinstance(event, list) and len(event) >= 3:
event_type = str(event[0]).lower()
message_value = str(event[1]) if len(event) > 1 else ""
status_value = str(event[2]).lower()
else:
continue
event_type = str(event[0]).lower()
message_value = str(event[1]) if len(event) > 1 else ""
status_value = str(event[2]).lower()
if event_type in {"cy:command", "cy:task"} and status_value in {"failed", "fail", "error"}:
return compact_failure_detail(f"{concise_testcase_name(testcase_name)} - {message_value}")
return None