57 Commits

Author SHA1 Message Date
Anthony Wen
5f7824619d Simplify ATVM host detail reporting 2026-06-13 18:41:19 -04:00
8c4985d33a fix(atvm-watcher): synthesize failed host result on hang-kill/nonzero exit; update run learning and vm inventory 2026-05-12 14:42:11 -04:00
65330ee9f8 atvm: fail runs explicitly on hang-kill and runner non-zero exit 2026-05-07 13:34:37 -04:00
2e0acb69c1 fix watcher failure detection for host reporter json
Handle dict-shaped reporter events when deriving watcher host failures.

- parse reporter JSON events with type/message/severity fields
- preserve existing support for list-shaped event records
- record the false-PASS failure mode in ATVM automation run learnings
2026-04-29 12:37:48 -04:00
2e25c2ae2b atvm: preserve authoritative host results in watcher artifacts
Prefer failing and terminal host results when merging supplemental
subrun artifacts so a later partial artifact cannot overwrite a more
authoritative outcome.

Also point ATVM agent guidance at the ATVM-specific git guide and add
that guide to document draft-only commit handling and the default SSH
command pattern.
2026-04-24 11:09:01 -04:00
86e7830d63 Fix ATVM watcher host parsing for wrapped duration rows
Handle Currents "Cloud Run Finished" table rows where the trailing
"s" in long duration values wraps onto its own continuation line.

Instead of appending that standalone continuation to the end of the
host row, drop the wrapped row and rely on the existing duration
parser to accept values without the trailing "s". This preserves all
host rows in parent summary parsing for completed non-categorized runs.

Also record the failure mode and recovery guidance in the ATVM run
learnings doc.
2026-04-22 14:03:02 -04:00
4f56ff9c4d Fix ATVM watcher install-only test flow extraction
Teach generated-spec TEST FLOW parsing to honor test-install-only runtime gates, including single-quoted Cypress.env checks, and suppress plugin branches when install-only mode disables them.

Document the 2026-04-16 install-only flow mismatch so future watcher updates keep Mattermost TEST FLOW aligned with the actual generated spec path.
2026-04-16 15:29:03 -04:00
37853e56a9 Harden ATVM watcher grouped-run summary recovery
Tighten `Cloud Run Finished` parsing so categorized watcher recovery only starts on real summary headers, stops at the grouped run's `Recorded Run` URL, and parses only actual summary-table host rows.

Also merge grouped per-host reporter artifacts into categorized recovery so completed grouped batches keep the correct host membership and Mattermost posts remain stable for both categorized and non-categorized runs.
2026-04-16 10:31:20 -04:00
70542a13a0 Fix ATVM watcher truncation after Cloud Run Finished
Adjust parent-run summary parsing so the watcher does not stop at the "Recorded Run" detection log line and miss later host rows in Mattermost status output.

Document the 2026-04-15 failure mode in ATVM run learnings so future watcher/reporting work preserves the broader Cloud Run Finished block parsing rule.
2026-04-15 21:11:38 -04:00
6334fecffa Harden ATVM watcher artifact scanning
- skip report artifacts that disappear during watcher post-processing
- use safe mtime handling for XML and mochawesome discovery
- prevent watcher crashes from transient FileNotFoundError races
2026-04-15 12:06:33 -04:00
a5e30b6acf Use actual ATVM runtime flow steps in watcher output 2026-04-14 19:05:37 -04:00
72ef15f308 Fix ATVM watcher test flow plugin filtering 2026-04-14 12:40:21 -04:00
a22ff8edf1 Improve ATVM watcher status metadata and run workflow 2026-04-14 07:56:53 -04:00
6403fa3d42 fix: preserve all hosts in non-categorized watcher summaries 2026-04-13 22:46:23 -04:00
f88f910815 Fix ATVM generated spec test flow fallback 2026-03-31 11:14:38 -04:00
7f87356832 Clarify ATVM host detail heading for failure notes 2026-03-31 11:07:17 -04:00
05d6f688f2 Clarify ATVM host detail column heading 2026-03-31 10:10:25 -04:00
da56c2668e Fix ATVM mochawesome failure extraction 2026-03-31 08:43:11 -04:00
7ab5daeca8 Split ATVM failure notes from general status notes 2026-03-30 22:31:41 -04:00
18dcbc89f9 Improve ATVM failed-host detail recovery 2026-03-30 21:38:59 -04:00
d1a909f9ab Add reboot both plugin caveat to ATVM notes 2026-03-30 21:04:01 -04:00
b45375dbbc Fix false ATVM failures from reporter txt fallback 2026-03-30 17:10:25 -04:00
6000fa0e78 Include failure detail in ATVM host status output 2026-03-30 16:07:20 -04:00
cc5eb6b7c8 Fix ATVM watcher reboot status recovery 2026-03-30 14:59:23 -04:00
1405a2e879 Fix categorized ATVM watcher host result recovery 2026-03-30 14:02:32 -04:00
63dd40faab Add detailed ATVM reboot test flow 2026-03-30 11:22:15 -04:00
d383b57ccc Align systemOS watcher output with template behavior 2026-03-27 21:30:36 -04:00
6c7ba5212b Refine categorized timing metrics and coverage output 2026-03-27 18:49:11 -04:00
06b8098499 Align ATVM coverage and notes with template command 2026-03-27 17:47:36 -04:00
708eddd7fa Remove verbose scope text from ATVM coverage 2026-03-27 17:40:32 -04:00
5432240525 Expand ATVM coverage details with run options 2026-03-27 17:32:22 -04:00
9024d5cadb Trim internal fallback details from ATVM notes 2026-03-27 16:12:14 -04:00
86e89ab9f1 Fix categorized watcher grouping and timing reconstruction 2026-03-27 16:08:13 -04:00
20c9ba7178 Fix categorized watcher subrun host recovery 2026-03-27 15:59:04 -04:00
650adb085e Keep ATVM watcher notes operator-focused 2026-03-27 14:54:39 -04:00
4186f2d0ea Persist Currents run URLs for ATVM watcher notes 2026-03-27 13:51:12 -04:00
c60542e0e5 Reorder ATVM status output and standardize push reminders 2026-03-27 11:57:19 -04:00
4b04f9b49d Fix duplicate watcher posts and restore 22-step E2E flow 2026-03-27 11:28:48 -04:00
cc551a6922 Recover watcher results when run log is missing 2026-03-27 11:00:11 -04:00
53f9e68fcd Show full template test flow in ATVM status 2026-03-27 10:19:02 -04:00
f890b4a105 Use parent run summary for single-host watcher completion
- update the non-categorized watcher path to fall back to the parent Cloud Run Finished summary when only the final check-xml-files reporter XML is available
- prevent single-host runs from being marked TERMINATED without host results when the controller log already contains the real host outcome
- allow the watcher to populate host results and post Mattermost successfully for completed single-host runs with reused build names
2026-03-27 09:01:05 -04:00
6daa83b0c5 Replace FUNCTIONALLY with TEST FLOW in ATVM status output
- update the ATVM status template to replace the FUNCTIONALLY section with a TEST FLOW section that shows the generic numbered run steps once for the whole test
- update the ATVM automation guide to describe TEST FLOW as the standard status-report section instead of FUNCTIONALLY
- update the watcher-generated status output so Mattermost and local status responses use the same TEST FLOW section
- add a 2026-03-27 run learning recording the move from FUNCTIONALLY to TEST FLOW for future ATVM reporting
2026-03-27 08:07:00 -04:00
4b3ed8cccd Add per-spec fallback for categorized watcher grouped summaries
- update the categorized watcher to fall back to earlier per-spec Run Finished entries when a grouped Cloud Run Finished section is present but does not contain host rows
- broaden duration parsing so colon-form timings like 13:59 and 15:32 from per-spec Cypress summaries are converted into seconds correctly
- make grouped Mattermost posting resilient across categorized batches whose cloud-summary output is incomplete or inconsistent, not just a single distro family
2026-03-26 21:28:27 -04:00
337a42066f Handle alternate grouped-run completion logs in ATVM watcher
- update the categorized watcher to recognize grouped Cloud Run Finished summaries that are followed by the controller log message  as well as the older literal  form
- allow grouped host summaries like the multi-host ubuntu batch to be parsed and posted even when the alternate controller log format is used
- prevent completed grouped runs from being left in RUNNING state solely because the watcher missed the alternate grouped summary terminator
2026-03-26 19:43:06 -04:00
e70ea8852d Aggregate multi-host grouped summaries in categorized watcher posts
- update the categorized watcher to parse all ATVM host rows from a grouped Cloud Run Finished block instead of only the first host row
- allow a later terminal grouped summary to replace an earlier RUNNING state for the same logical group so grouped posts are emitted with the full host set
- fix multi-host same-group reporting so batches like ubuntu22.04 plus ubuntu24.04 are posted as one grouped result with both hosts included
2026-03-26 18:49:58 -04:00
ba5dbca6f9 Close completed watcher groups when the next category starts
- update the categorized watcher to mark a grouped subrun complete once its final check-xml-files.ts artifact exists and the next active host belongs to a different inferred group
- prevent a finished group such as ubuntu from being kept in RUNNING state after execution has already advanced into the next category
- allow grouped Mattermost posts to be emitted at category boundaries instead of being blocked by stale active-group carryover
2026-03-26 17:43:02 -04:00
3b9b9eef0f Merge categorized watcher subruns by inferred group identity
- update the watcher to merge consecutive categorized child build ids into one logical grouped subrun when they resolve to the same real host-based group
- prevent one real grouped run from being fragmented across multiple watcher subruns just because the raw ci-build-id label changes between hosts
- keep grouped Mattermost posting aligned with the inferred group identity instead of the unstable raw child build id
2026-03-26 16:51:10 -04:00
35342c1f23 Post categorized watcher results from early host completion evidence
- update the categorized watcher to use the latest matching host reporter artifact written before grouped finalization so grouped results can be posted as soon as the host run is actually done
- reduce dependence on the later parent Cloud Run Finished summary block for grouped-run completion reporting
- keep grouped runs from waiting until near parent-run completion when real host-level completion evidence is already available
2026-03-26 15:19:54 -04:00
293b9bf239 Delay categorized watcher posts until host summary is available
- keep categorized grouped sub-runs in RUNNING state when only the grouped check-xml-files.ts artifact has appeared and the parent log has not yet exposed the real host summary
- prevent blank Mattermost grouped-run posts with empty host tables, zero counts, and zero/invalid timing caused by posting too early
- require the categorized watcher to wait for host-level grouped summary data before treating the grouped run as complete
2026-03-26 14:40:11 -04:00
f5eb21cccd Infer categorized watcher group names from actual host execution
- update the watcher to stop trusting misleading categorized child build labels when they do not match the host/spec actually being executed
- infer the reported categorized group name from the actual host being run, so mismatched labels like ubuntu-batch for a Red Hat host are corrected in status reporting
- document the categorized watcher workaround in the ATVM guide, watcher design, and watcher README without changing the underlying ATVM runner scripts
2026-03-26 14:20:22 -04:00