fix watcher failure detection for host reporter json
Handle dict-shaped reporter events when deriving watcher host failures. - parse reporter JSON events with type/message/severity fields - preserve existing support for list-shaped event records - record the false-PASS failure mode in ATVM automation run learnings
This commit is contained in:
@@ -18,6 +18,15 @@ This file stores run-specific examples only when a run produced a new learning r
|
||||
- Write the template phase output to `/tmp/<build>.launch.log` so template activity is preserved separately from the live runner log.
|
||||
- If the template step fails, stop immediately and do not start the watcher or the runner.
|
||||
|
||||
## Run Learning: 2026-04-29 (Watcher host-artifact parser must handle dict-shaped reporter events)
|
||||
- Observed failure mode:
|
||||
- A non-categorized ATVM compute-migration run failed in the host reporter artifacts, but the watcher posted `PASS`.
|
||||
- The watcher fell back to the per-host JSON artifact after `check-xml-files.ts`, but `extract_failure_from_reporter_events()` only recognized the older list-shaped event format.
|
||||
- Current reporter JSON stores events as dicts with fields such as `type`, `message`, and `severity`, so the parser missed `severity: error` and incorrectly returned `0 failures`.
|
||||
- Action for future runs:
|
||||
- Treat both list-shaped and dict-shaped reporter event records as valid inputs when extracting failure details from host JSON artifacts.
|
||||
- Continue treating host reporter artifacts as authoritative fallback evidence when final XML only contains `check-xml-files.ts`.
|
||||
|
||||
## Run Learning: 2026-04-24 (Categorized watcher false-PASS guardrail)
|
||||
- Observed failure mode:
|
||||
- A categorized compute-migration run was incorrectly reported as `PASS` for `atvm121-ubuntu24.04` even though the actual Ubuntu grouped sub-run failed.
|
||||
|
||||
@@ -524,11 +524,16 @@ def extract_failure_from_reporter_events(testcase_name: str, testcase_events: ob
|
||||
if not isinstance(testcase_events, list):
|
||||
return None
|
||||
for event in testcase_events:
|
||||
if not isinstance(event, list) or len(event) < 3:
|
||||
if isinstance(event, dict):
|
||||
event_type = str(event.get("type", "")).lower()
|
||||
message_value = str(event.get("message", ""))
|
||||
status_value = str(event.get("severity", "")).lower()
|
||||
elif isinstance(event, list) and len(event) >= 3:
|
||||
event_type = str(event[0]).lower()
|
||||
message_value = str(event[1]) if len(event) > 1 else ""
|
||||
status_value = str(event[2]).lower()
|
||||
else:
|
||||
continue
|
||||
event_type = str(event[0]).lower()
|
||||
message_value = str(event[1]) if len(event) > 1 else ""
|
||||
status_value = str(event[2]).lower()
|
||||
if event_type in {"cy:command", "cy:task"} and status_value in {"failed", "fail", "error"}:
|
||||
return compact_failure_detail(f"{concise_testcase_name(testcase_name)} - {message_value}")
|
||||
return None
|
||||
|
||||
Reference in New Issue
Block a user