Commit Graph

88 Commits

Author SHA1 Message Date
a22ff8edf1 Improve ATVM watcher status metadata and run workflow 2026-04-14 07:56:53 -04:00
6403fa3d42 fix: preserve all hosts in non-categorized watcher summaries 2026-04-13 22:46:23 -04:00
e60bd5e9cd Default ATVM vmware compute migration options 2026-03-31 12:35:44 -04:00
f88f910815 Fix ATVM generated spec test flow fallback 2026-03-31 11:14:38 -04:00
7f87356832 Clarify ATVM host detail heading for failure notes 2026-03-31 11:07:17 -04:00
05d6f688f2 Clarify ATVM host detail column heading 2026-03-31 10:10:25 -04:00
da56c2668e Fix ATVM mochawesome failure extraction 2026-03-31 08:43:11 -04:00
7ab5daeca8 Split ATVM failure notes from general status notes 2026-03-30 22:31:41 -04:00
18dcbc89f9 Improve ATVM failed-host detail recovery 2026-03-30 21:38:59 -04:00
d1a909f9ab Add reboot both plugin caveat to ATVM notes 2026-03-30 21:04:01 -04:00
c64a64ca14 Prefer generated specs for ATVM test flow 2026-03-30 20:31:14 -04:00
b1b7e89449 Default ATVM template runs to test partition 2026-03-30 20:26:08 -04:00
e8353dc9a9 Require explicit confirmation for reboot plugin both 2026-03-30 20:20:15 -04:00
dec13a4667 Tighten ATVM completed-run status verification 2026-03-30 20:12:56 -04:00
b45375dbbc Fix false ATVM failures from reporter txt fallback 2026-03-30 17:10:25 -04:00
0b379612a8 Clarify ATVM plugin validation rules 2026-03-30 16:22:59 -04:00
6000fa0e78 Include failure detail in ATVM host status output 2026-03-30 16:07:20 -04:00
944a13ec26 Tighten ATVM git push approval gate 2026-03-30 15:05:23 -04:00
cc5eb6b7c8 Fix ATVM watcher reboot status recovery 2026-03-30 14:59:23 -04:00
1405a2e879 Fix categorized ATVM watcher host result recovery 2026-03-30 14:02:32 -04:00
89f558bd39 Document approval gate for git request variants 2026-03-30 11:28:16 -04:00
63dd40faab Add detailed ATVM reboot test flow 2026-03-30 11:22:15 -04:00
d383b57ccc Align systemOS watcher output with template behavior 2026-03-27 21:30:36 -04:00
6c7ba5212b Refine categorized timing metrics and coverage output 2026-03-27 18:49:11 -04:00
06b8098499 Align ATVM coverage and notes with template command 2026-03-27 17:47:36 -04:00
708eddd7fa Remove verbose scope text from ATVM coverage 2026-03-27 17:40:32 -04:00
5432240525 Expand ATVM coverage details with run options 2026-03-27 17:32:22 -04:00
9024d5cadb Trim internal fallback details from ATVM notes 2026-03-27 16:12:14 -04:00
86e89ab9f1 Fix categorized watcher grouping and timing reconstruction 2026-03-27 16:08:13 -04:00
20c9ba7178 Fix categorized watcher subrun host recovery 2026-03-27 15:59:04 -04:00
650adb085e Keep ATVM watcher notes operator-focused 2026-03-27 14:54:39 -04:00
4186f2d0ea Persist Currents run URLs for ATVM watcher notes 2026-03-27 13:51:12 -04:00
3b28f877ef Refine explicit-VM blacklist handling for ATVM runs 2026-03-27 12:30:31 -04:00
95eeba60cd Make watcher the default ATVM approval path 2026-03-27 12:24:21 -04:00
c60542e0e5 Reorder ATVM status output and standardize push reminders 2026-03-27 11:57:19 -04:00
4b04f9b49d Fix duplicate watcher posts and restore 22-step E2E flow 2026-03-27 11:28:48 -04:00
cc551a6922 Recover watcher results when run log is missing 2026-03-27 11:00:11 -04:00
53f9e68fcd Show full template test flow in ATVM status 2026-03-27 10:19:02 -04:00
2699651f8f Require watcher startup before ATVM runner launch
- update the ATVM automation guide to make watcher-first launch order explicit whenever the watcher is approved
- update the ATVM AGENTS rules so the runner is never started before the watcher for watcher-backed runs
- add a 2026-03-27 run learning documenting that the watcher helper can delete the live runner log if the runner starts first
2026-03-27 09:25:48 -04:00
f890b4a105 Use parent run summary for single-host watcher completion
- update the non-categorized watcher path to fall back to the parent Cloud Run Finished summary when only the final check-xml-files reporter XML is available
- prevent single-host runs from being marked TERMINATED without host results when the controller log already contains the real host outcome
- allow the watcher to populate host results and post Mattermost successfully for completed single-host runs with reused build names
2026-03-27 09:01:05 -04:00
751e0ba514 Clear stale runner logs before starting ATVM watcher
- update the watcher start helper to remove the old /tmp/<build-name>.log file before starting a fresh watcher instance for a reused build name
- prevent the watcher from inheriting stale started_at timestamps from previous runs with the same build name and immediately exiting as HUNG
- make reused-build watcher startup safer by resetting both watcher state and the matching temporary runner log
2026-03-27 08:34:41 -04:00
6daa83b0c5 Replace FUNCTIONALLY with TEST FLOW in ATVM status output
- update the ATVM status template to replace the FUNCTIONALLY section with a TEST FLOW section that shows the generic numbered run steps once for the whole test
- update the ATVM automation guide to describe TEST FLOW as the standard status-report section instead of FUNCTIONALLY
- update the watcher-generated status output so Mattermost and local status responses use the same TEST FLOW section
- add a 2026-03-27 run learning recording the move from FUNCTIONALLY to TEST FLOW for future ATVM reporting
2026-03-27 08:07:00 -04:00
833225378d Suppress harmless watcher reset-failed noise in ATVM updates
- update the ATVM AGENTS guidance so expected  output is not called out in routine run updates
- update the ATVM automation guide to mention reset-failed output only when it actually blocks watcher startup or matters for debugging
- add a 2026-03-26 run learning recording that this reset-failed output is normal low-signal noise during successful watcher startup
2026-03-26 21:37:35 -04:00
4b3ed8cccd Add per-spec fallback for categorized watcher grouped summaries
- update the categorized watcher to fall back to earlier per-spec Run Finished entries when a grouped Cloud Run Finished section is present but does not contain host rows
- broaden duration parsing so colon-form timings like 13:59 and 15:32 from per-spec Cypress summaries are converted into seconds correctly
- make grouped Mattermost posting resilient across categorized batches whose cloud-summary output is incomplete or inconsistent, not just a single distro family
2026-03-26 21:28:27 -04:00
3431c40af7 Document ATVM spec verification lesson in run learnings
- add a 2026-03-26 run learning that explains how cmc-templates.py can generate the requested spec files while a fragile verification step still misses them
- document that shell-escaped regex one-liners over SSH are not a reliable way to validate the controller specPattern
- record the preferred future workflow: verify generated .ts files and the config specPattern directly on the controller before launching run-sorry-cypress.py
2026-03-26 19:48:16 -04:00
337a42066f Handle alternate grouped-run completion logs in ATVM watcher
- update the categorized watcher to recognize grouped Cloud Run Finished summaries that are followed by the controller log message  as well as the older literal  form
- allow grouped host summaries like the multi-host ubuntu batch to be parsed and posted even when the alternate controller log format is used
- prevent completed grouped runs from being left in RUNNING state solely because the watcher missed the alternate grouped summary terminator
2026-03-26 19:43:06 -04:00
e70ea8852d Aggregate multi-host grouped summaries in categorized watcher posts
- update the categorized watcher to parse all ATVM host rows from a grouped Cloud Run Finished block instead of only the first host row
- allow a later terminal grouped summary to replace an earlier RUNNING state for the same logical group so grouped posts are emitted with the full host set
- fix multi-host same-group reporting so batches like ubuntu22.04 plus ubuntu24.04 are posted as one grouped result with both hosts included
2026-03-26 18:49:58 -04:00
ba5dbca6f9 Close completed watcher groups when the next category starts
- update the categorized watcher to mark a grouped subrun complete once its final check-xml-files.ts artifact exists and the next active host belongs to a different inferred group
- prevent a finished group such as ubuntu from being kept in RUNNING state after execution has already advanced into the next category
- allow grouped Mattermost posts to be emitted at category boundaries instead of being blocked by stale active-group carryover
2026-03-26 17:43:02 -04:00
3b9b9eef0f Merge categorized watcher subruns by inferred group identity
- update the watcher to merge consecutive categorized child build ids into one logical grouped subrun when they resolve to the same real host-based group
- prevent one real grouped run from being fragmented across multiple watcher subruns just because the raw ci-build-id label changes between hosts
- keep grouped Mattermost posting aligned with the inferred group identity instead of the unstable raw child build id
2026-03-26 16:51:10 -04:00
2eed645a85 Require ATVM spec verification before starting the runner
- update the ATVM automation guide to require a post-template verification gate before launching run-sorry-cypress.py
- require verification that both the generated .ts files and the config specPattern include every requested VM after cmc-templates.py finishes
- document that the runner must not be launched when any requested VM is missing from the generated spec set
- update the ATVM AGENTS rules so this template-to-runner verification step is part of the default automation workflow
2026-03-26 15:55:13 -04:00