Commit Graph

84 Commits

Author SHA1 Message Date
944a13ec26 Tighten ATVM git push approval gate 2026-03-30 15:05:23 -04:00
cc5eb6b7c8 Fix ATVM watcher reboot status recovery 2026-03-30 14:59:23 -04:00
1405a2e879 Fix categorized ATVM watcher host result recovery 2026-03-30 14:02:32 -04:00
89f558bd39 Document approval gate for git request variants 2026-03-30 11:28:16 -04:00
63dd40faab Add detailed ATVM reboot test flow 2026-03-30 11:22:15 -04:00
d383b57ccc Align systemOS watcher output with template behavior 2026-03-27 21:30:36 -04:00
6c7ba5212b Refine categorized timing metrics and coverage output 2026-03-27 18:49:11 -04:00
06b8098499 Align ATVM coverage and notes with template command 2026-03-27 17:47:36 -04:00
708eddd7fa Remove verbose scope text from ATVM coverage 2026-03-27 17:40:32 -04:00
5432240525 Expand ATVM coverage details with run options 2026-03-27 17:32:22 -04:00
9024d5cadb Trim internal fallback details from ATVM notes 2026-03-27 16:12:14 -04:00
86e89ab9f1 Fix categorized watcher grouping and timing reconstruction 2026-03-27 16:08:13 -04:00
20c9ba7178 Fix categorized watcher subrun host recovery 2026-03-27 15:59:04 -04:00
650adb085e Keep ATVM watcher notes operator-focused 2026-03-27 14:54:39 -04:00
4186f2d0ea Persist Currents run URLs for ATVM watcher notes 2026-03-27 13:51:12 -04:00
3b28f877ef Refine explicit-VM blacklist handling for ATVM runs 2026-03-27 12:30:31 -04:00
95eeba60cd Make watcher the default ATVM approval path 2026-03-27 12:24:21 -04:00
c60542e0e5 Reorder ATVM status output and standardize push reminders 2026-03-27 11:57:19 -04:00
4b04f9b49d Fix duplicate watcher posts and restore 22-step E2E flow 2026-03-27 11:28:48 -04:00
cc551a6922 Recover watcher results when run log is missing 2026-03-27 11:00:11 -04:00
4c0e28cc05 Clarify explicit approval requirement for git commits 2026-03-27 10:23:03 -04:00
53f9e68fcd Show full template test flow in ATVM status 2026-03-27 10:19:02 -04:00
2699651f8f Require watcher startup before ATVM runner launch
- update the ATVM automation guide to make watcher-first launch order explicit whenever the watcher is approved
- update the ATVM AGENTS rules so the runner is never started before the watcher for watcher-backed runs
- add a 2026-03-27 run learning documenting that the watcher helper can delete the live runner log if the runner starts first
2026-03-27 09:25:48 -04:00
f890b4a105 Use parent run summary for single-host watcher completion
- update the non-categorized watcher path to fall back to the parent Cloud Run Finished summary when only the final check-xml-files reporter XML is available
- prevent single-host runs from being marked TERMINATED without host results when the controller log already contains the real host outcome
- allow the watcher to populate host results and post Mattermost successfully for completed single-host runs with reused build names
2026-03-27 09:01:05 -04:00
751e0ba514 Clear stale runner logs before starting ATVM watcher
- update the watcher start helper to remove the old /tmp/<build-name>.log file before starting a fresh watcher instance for a reused build name
- prevent the watcher from inheriting stale started_at timestamps from previous runs with the same build name and immediately exiting as HUNG
- make reused-build watcher startup safer by resetting both watcher state and the matching temporary runner log
2026-03-27 08:34:41 -04:00
6daa83b0c5 Replace FUNCTIONALLY with TEST FLOW in ATVM status output
- update the ATVM status template to replace the FUNCTIONALLY section with a TEST FLOW section that shows the generic numbered run steps once for the whole test
- update the ATVM automation guide to describe TEST FLOW as the standard status-report section instead of FUNCTIONALLY
- update the watcher-generated status output so Mattermost and local status responses use the same TEST FLOW section
- add a 2026-03-27 run learning recording the move from FUNCTIONALLY to TEST FLOW for future ATVM reporting
2026-03-27 08:07:00 -04:00
833225378d Suppress harmless watcher reset-failed noise in ATVM updates
- update the ATVM AGENTS guidance so expected  output is not called out in routine run updates
- update the ATVM automation guide to mention reset-failed output only when it actually blocks watcher startup or matters for debugging
- add a 2026-03-26 run learning recording that this reset-failed output is normal low-signal noise during successful watcher startup
2026-03-26 21:37:35 -04:00
4b3ed8cccd Add per-spec fallback for categorized watcher grouped summaries
- update the categorized watcher to fall back to earlier per-spec Run Finished entries when a grouped Cloud Run Finished section is present but does not contain host rows
- broaden duration parsing so colon-form timings like 13:59 and 15:32 from per-spec Cypress summaries are converted into seconds correctly
- make grouped Mattermost posting resilient across categorized batches whose cloud-summary output is incomplete or inconsistent, not just a single distro family
2026-03-26 21:28:27 -04:00
3431c40af7 Document ATVM spec verification lesson in run learnings
- add a 2026-03-26 run learning that explains how cmc-templates.py can generate the requested spec files while a fragile verification step still misses them
- document that shell-escaped regex one-liners over SSH are not a reliable way to validate the controller specPattern
- record the preferred future workflow: verify generated .ts files and the config specPattern directly on the controller before launching run-sorry-cypress.py
2026-03-26 19:48:16 -04:00
337a42066f Handle alternate grouped-run completion logs in ATVM watcher
- update the categorized watcher to recognize grouped Cloud Run Finished summaries that are followed by the controller log message  as well as the older literal  form
- allow grouped host summaries like the multi-host ubuntu batch to be parsed and posted even when the alternate controller log format is used
- prevent completed grouped runs from being left in RUNNING state solely because the watcher missed the alternate grouped summary terminator
2026-03-26 19:43:06 -04:00
e70ea8852d Aggregate multi-host grouped summaries in categorized watcher posts
- update the categorized watcher to parse all ATVM host rows from a grouped Cloud Run Finished block instead of only the first host row
- allow a later terminal grouped summary to replace an earlier RUNNING state for the same logical group so grouped posts are emitted with the full host set
- fix multi-host same-group reporting so batches like ubuntu22.04 plus ubuntu24.04 are posted as one grouped result with both hosts included
2026-03-26 18:49:58 -04:00
ba5dbca6f9 Close completed watcher groups when the next category starts
- update the categorized watcher to mark a grouped subrun complete once its final check-xml-files.ts artifact exists and the next active host belongs to a different inferred group
- prevent a finished group such as ubuntu from being kept in RUNNING state after execution has already advanced into the next category
- allow grouped Mattermost posts to be emitted at category boundaries instead of being blocked by stale active-group carryover
2026-03-26 17:43:02 -04:00
3b9b9eef0f Merge categorized watcher subruns by inferred group identity
- update the watcher to merge consecutive categorized child build ids into one logical grouped subrun when they resolve to the same real host-based group
- prevent one real grouped run from being fragmented across multiple watcher subruns just because the raw ci-build-id label changes between hosts
- keep grouped Mattermost posting aligned with the inferred group identity instead of the unstable raw child build id
2026-03-26 16:51:10 -04:00
2eed645a85 Require ATVM spec verification before starting the runner
- update the ATVM automation guide to require a post-template verification gate before launching run-sorry-cypress.py
- require verification that both the generated .ts files and the config specPattern include every requested VM after cmc-templates.py finishes
- document that the runner must not be launched when any requested VM is missing from the generated spec set
- update the ATVM AGENTS rules so this template-to-runner verification step is part of the default automation workflow
2026-03-26 15:55:13 -04:00
35342c1f23 Post categorized watcher results from early host completion evidence
- update the categorized watcher to use the latest matching host reporter artifact written before grouped finalization so grouped results can be posted as soon as the host run is actually done
- reduce dependence on the later parent Cloud Run Finished summary block for grouped-run completion reporting
- keep grouped runs from waiting until near parent-run completion when real host-level completion evidence is already available
2026-03-26 15:19:54 -04:00
293b9bf239 Delay categorized watcher posts until host summary is available
- keep categorized grouped sub-runs in RUNNING state when only the grouped check-xml-files.ts artifact has appeared and the parent log has not yet exposed the real host summary
- prevent blank Mattermost grouped-run posts with empty host tables, zero counts, and zero/invalid timing caused by posting too early
- require the categorized watcher to wait for host-level grouped summary data before treating the grouped run as complete
2026-03-26 14:40:11 -04:00
f5eb21cccd Infer categorized watcher group names from actual host execution
- update the watcher to stop trusting misleading categorized child build labels when they do not match the host/spec actually being executed
- infer the reported categorized group name from the actual host being run, so mismatched labels like ubuntu-batch for a Red Hat host are corrected in status reporting
- document the categorized watcher workaround in the ATVM guide, watcher design, and watcher README without changing the underlying ATVM runner scripts
2026-03-26 14:20:22 -04:00
7d49896ac2 Improve categorized watcher duration parsing for grouped runs
- relax grouped-run duration parsing so Linux categorized summaries like `12m 19.4` and `15m 42.2` are converted reliably even when the trailing `s` is split awkwardly in the parent run log
- keep the categorized grouped-run summary extraction aligned with the host results already being parsed from the parent Cloud Run Finished blocks
- refresh the controller watcher copy so the next categorized run uses the improved grouped duration parser
2026-03-26 13:41:06 -04:00
c74f74bc46 Improve categorized watcher grouped-run summary extraction
- update the watcher to parse completed categorized grouped-run host summaries from the parent run log instead of relying only on grouped XML files that often contain only check-xml-files.ts
- add grouped-run duration parsing so categorized sub-run timing can be derived from the Cloud Run Finished summary when host XML details are absent
- fix the completed-summary to grouped-xml alignment so filtered-out older artifacts do not shift host-summary assignment for the current run
2026-03-26 13:22:37 -04:00
44e6e0e653 Keep categorized ATVM watcher alive until parent run finishes
- update the watcher to treat categorized parent-run activity as the authoritative signal for whether the overall request is still running
- prevent the watcher from exiting early just because one categorized grouped sub-run completed and wrote artifacts
- document that categorized watcher instances must remain alive between grouped runs until the parent request has actually gone inactive past the grace window
- update the ATVM guide, watcher design, and install docs to reflect the stricter categorized parent-run completion rule
2026-03-26 12:39:23 -04:00
1ba508169f Fix watcher timestamp handling for reused categorized runs
- interpret ATVM controller log timestamps in the controller's local timezone before converting to UTC so the watcher uses the correct current-run window
- prevent newly started categorized runs from immediately picking up older categorized artifacts just because the parent build name was reused
- keep the categorized watcher focused on artifacts from the current controller run instead of stale prior attempts
2026-03-26 12:26:23 -04:00
3ea732d63c Improve categorized ATVM watcher sub-run detection
- update the watcher to detect the active categorized sub-run from the live `--ci-build-id` process state instead of treating the parent run as one synthetic grouped run
- fix host XML parsing so the watcher prefers the real host suite over the `Root Suite` entry, avoiding `0 tests, 0 failures` summaries
- use the first timestamp inside the run log as the watcher start time so restarted watchers do not miss current-run categorized artifacts because of log file mtime drift
- improve active-host inference for categorized runs so the watcher maps the current categorized build to the correct host family while the sub-run is still in progress
2026-03-26 12:01:07 -04:00
f5849dde0c Reset reused watcher state before starting a new ATVM run
- update the watcher start helper to stop any stale watcher instance for the same requested parent build name and remove its old state directory before starting fresh
- document that reused parent build names must not inherit stale cancelled, posted, state.json, or subruns state from older runs
- update the watcher install and design docs so the controller workflow explicitly treats stale reused-build-name state as part of startup cleanup
2026-03-26 11:30:28 -04:00
dda0a0b4c0 Require verification after git commit execution
- update the git guide to require post-commit verification before reporting success
- document that commit verification must confirm git commit exited successfully, HEAD advanced, and the new commit appears in git log
- document that failed or non-created commits must be reported clearly instead of being claimed as successful
2026-03-26 11:03:20 -04:00
d60b8b9b18 Update ATVM watcher for categorized sub-run posting
- update the watcher design and automation guide to treat --categorize as sequential ATVM sub-runs rather than one parent run with internal phases
- document that categorized runs should send one Mattermost status per completed grouped sub-run instead of one parent-only final post
- add a --categorize option to the watcher start helper so categorized mode is explicit in watcher startup
- update the watcher implementation to track categorized sub-runs separately, write per-subrun state, and post each completed grouped run once
2026-03-26 11:00:39 -04:00
68cd428733 Require CDC cleanup before Windows CMC reinstall in CDS MCP workflow
- update the Windows CMC install reference to require CDC project cleanup before reinstall when stale host state may exist
- update the VMware MigrateOps guide so the cleanup-first CMC preparation rule applies to both Linux and Windows sources
- document that Windows reinstall must be followed by a fresh CDC reconnect check before creating migration sessions
- update CDS MCP AGENTS guidance so Windows sources do not skip the standard CDC cleanup plus reinstall sequence
2026-03-26 10:14:11 -04:00
b19d6a912e Document Windows SSH-based CMC workflow for CDS MCP
- update the CDS MCP CMC install reference to prefer SSH plus PowerShell for Windows guest execution instead of VMware guest operations
- document the Windows install-state check and uninstall-then-reinstall pattern so it matches the Linux CMC preparation flow
- update the VMware MigrateOps guide to treat Windows CMC preparation as an SSH-based workflow using ATVM_WINDOWS_TARGET_* credentials
- update the CDS MCP README and AGENTS guidance so the Windows guest execution default is explicit
2026-03-26 08:47:53 -04:00
c0004c1dff Document separate Linux and Windows ATVM target credentials
- add explicit Windows ATVM guest credential references alongside the existing Linux target defaults
- update the ATVM automation guide and AGENTS rules so Linux SSH uses ATVM_TARGET_* while Windows guest access uses ATVM_WINDOWS_TARGET_*
- update the CDS MCP CMC install and VMware workflow docs to distinguish Linux and Windows credential usage for the shared ATVM target IP
- update the VM lookup reference so common VM credentials list both Linux and Windows target variables
2026-03-26 08:33:41 -04:00
c9706e9702 Record cancelled watcher state on ATVM run cancellation
- update the watcher cancel helper so it writes a final CANCELLED state into state.json before stopping the service
- record cancellation timestamps and a cancellation note in the watcher state file for clearer post-run inspection
- update the watcher service docs so the documented cancel behavior matches the state-file handling
2026-03-25 18:24:17 -04:00
9caa7deb94 Stop tracking Python bytecode in watcher service
- remove the committed watcher-service __pycache__ bytecode file from git tracking
- ignore Python bytecode artifacts so generated .pyc files do not get committed again
- keep the watcher-service source files as the only tracked implementation artifacts
2026-03-25 17:44:52 -04:00