Update automation guide defaults so cmc-migrateops-compute-migration includes --set_static_ip_dest by default unless explicitly overridden. Clarify vmware-specific default as vm_platforms selection, and record the behavior change in run learnings for future runs.
Update ATVM run workflow rules to remove the default pre-run approval gate for automation requests while keeping safety checks around live running-state and spec verification. Set watcher-backed execution as the default unless explicitly overridden and require post-execution reporting of the exact template and runner commands used. Record the workflow shift in automation run learnings with a dated entry for future consistency.
Handle dict-shaped reporter events when deriving watcher host failures.
- parse reporter JSON events with type/message/severity fields
- preserve existing support for list-shaped event records
- record the false-PASS failure mode in ATVM automation run learnings
Execute the template step before starting watcher-backed ATVM runs.
- run --template-command synchronously in start-atvm-run.sh
- write template output to /tmp/<build>.launch.log
- stop before watcher/runner startup if template generation fails
- document the corrected wrapper behavior in watcher-service docs
- record the stale specPattern failure mode in automation run learnings
Update the ATVM git instructions so controller-repo git drafting and the SSH-prefixed push example only apply when the operator explicitly asks for the e2e cypress repo or a close variation.
Align AGENTS.md with git-guide.md so generic ATVM git requests no longer assume the controller repo by default.
Add an AGENTS.md rule for vCenter VM snapshot requests so snapshot names default to the format using the local America/New_York timezone unless the operator requests a different name.
Update the ATVM git guide so push examples do not hardcode origin/main.
Prefer the operator-specified remote and branch when provided, and otherwise draft a generic SSH-prefixed git push command or use the remote/branch already established in context.
Add a dedicated ESX VM guide for read-first hardware change workflows,
including disk creation, datastore VMDK attachment, and PCI passthrough
assignment.
Update the existing lookup/assignment runbook and top-level docs to
reference the new guide and require explicit reporting of VM name
mismatches and missing datastore directories before any change.
Prefer failing and terminal host results when merging supplemental
subrun artifacts so a later partial artifact cannot overwrite a more
authoritative outcome.
Also point ATVM agent guidance at the ATVM-specific git guide and add
that guide to document draft-only commit handling and the default SSH
command pattern.
require categorized PASS results to be confirmed from the child run log or Cloud Run Finished summary
forbid treating watcher host_results, grouped XML, or check-xml-files.ts alone as proof of PASS
record the 2026-04-24 false-PASS incident as a run learning
Handle Currents "Cloud Run Finished" table rows where the trailing
"s" in long duration values wraps onto its own continuation line.
Instead of appending that standalone continuation to the end of the
host row, drop the wrapped row and rely on the existing duration
parser to accept values without the trailing "s". This preserves all
host rows in parent summary parsing for completed non-categorized runs.
Also record the failure mode and recovery guidance in the ATVM run
learnings doc.
Add a small monitoring helper script for logging CPU and memory usage
for mtdi-daemon and galaxy-migrate.
Also update the cdssync workspace instructions to document how to use
the monitoring helper.
Add bulk data generation controls for folder count, files per folder,
file size range, and bulk dataset size limits.
Also update the cdssync docs to describe the new options and how
update mode applies to generated bulk files.
Add support for running content updates against an existing migration
test dataset without recreating the filesystem structure.
Also make ACL/xattr updates non-fatal on filesystems that do not
support those operations.
Add optional interval-based random content updates to the cdssync
migration test dataset generator and document the new behavior.
This allows the dataset to be created once and then updated either
continuously or every N seconds while preserving the intended
special-case file structure.
Add the cdssync migration test dataset manifest, generator script,
workspace instructions, and gitignore.
This sets the default workflow to:
- generate the dataset locally
- copy it to the test machine with metadata preserved
- verify the copied data before migration testing
Teach generated-spec TEST FLOW parsing to honor test-install-only runtime gates, including single-quoted Cypress.env checks, and suppress plugin branches when install-only mode disables them.
Document the 2026-04-16 install-only flow mismatch so future watcher updates keep Mattermost TEST FLOW aligned with the actual generated spec path.
Tighten `Cloud Run Finished` parsing so categorized watcher recovery only starts on real summary headers, stops at the grouped run's `Recorded Run` URL, and parses only actual summary-table host rows.
Also merge grouped per-host reporter artifacts into categorized recovery so completed grouped batches keep the correct host membership and Mattermost posts remain stable for both categorized and non-categorized runs.
Add the working vCenter placement-check pattern to the ATVM automation guide so explicit VM-list planning uses the correct lowercase govc JSON keys, resolves datastore refs to names, and ignores helper datastores such as install ISO attachments when deciding between gold and gold-2.
Adjust parent-run summary parsing so the watcher does not stop at the "Recorded Run" detection log line and miss later host rows in Mattermost status output.
Document the 2026-04-15 failure mode in ATVM run learnings so future watcher/reporting work preserves the broader Cloud Run Finished block parsing rule.
- skip report artifacts that disappear during watcher post-processing
- use safe mtime handling for XML and mochawesome discovery
- prevent watcher crashes from transient FileNotFoundError races
- map AutomatedTest-VMBootImg-Gold to gold and Gold-2 to gold-2
- require live vCenter placement checks for explicit VM lists
- stop mixed gold/gold-2 requests before planning ATVM runs
- return hostname-only datastore inventories by default
- tell the operator about the vCenter placement check before showing run commands
- clarify that repo inventory files are not the source of truth for current datastore membership
Remove the old issue-handling defaults and approval rules that referred
to the galaxy-complete workspace.
Keep the remaining commit approval and repository safety guidance.
- update the ATVM automation guide to make watcher-first launch order explicit whenever the watcher is approved
- update the ATVM AGENTS rules so the runner is never started before the watcher for watcher-backed runs
- add a 2026-03-27 run learning documenting that the watcher helper can delete the live runner log if the runner starts first
- update the non-categorized watcher path to fall back to the parent Cloud Run Finished summary when only the final check-xml-files reporter XML is available
- prevent single-host runs from being marked TERMINATED without host results when the controller log already contains the real host outcome
- allow the watcher to populate host results and post Mattermost successfully for completed single-host runs with reused build names
- update the watcher start helper to remove the old /tmp/<build-name>.log file before starting a fresh watcher instance for a reused build name
- prevent the watcher from inheriting stale started_at timestamps from previous runs with the same build name and immediately exiting as HUNG
- make reused-build watcher startup safer by resetting both watcher state and the matching temporary runner log
- update the ATVM status template to replace the FUNCTIONALLY section with a TEST FLOW section that shows the generic numbered run steps once for the whole test
- update the ATVM automation guide to describe TEST FLOW as the standard status-report section instead of FUNCTIONALLY
- update the watcher-generated status output so Mattermost and local status responses use the same TEST FLOW section
- add a 2026-03-27 run learning recording the move from FUNCTIONALLY to TEST FLOW for future ATVM reporting
- update the ATVM AGENTS guidance so expected output is not called out in routine run updates
- update the ATVM automation guide to mention reset-failed output only when it actually blocks watcher startup or matters for debugging
- add a 2026-03-26 run learning recording that this reset-failed output is normal low-signal noise during successful watcher startup
- update the categorized watcher to fall back to earlier per-spec Run Finished entries when a grouped Cloud Run Finished section is present but does not contain host rows
- broaden duration parsing so colon-form timings like 13:59 and 15:32 from per-spec Cypress summaries are converted into seconds correctly
- make grouped Mattermost posting resilient across categorized batches whose cloud-summary output is incomplete or inconsistent, not just a single distro family
- add a 2026-03-26 run learning that explains how cmc-templates.py can generate the requested spec files while a fragile verification step still misses them
- document that shell-escaped regex one-liners over SSH are not a reliable way to validate the controller specPattern
- record the preferred future workflow: verify generated .ts files and the config specPattern directly on the controller before launching run-sorry-cypress.py
- update the categorized watcher to recognize grouped Cloud Run Finished summaries that are followed by the controller log message as well as the older literal form
- allow grouped host summaries like the multi-host ubuntu batch to be parsed and posted even when the alternate controller log format is used
- prevent completed grouped runs from being left in RUNNING state solely because the watcher missed the alternate grouped summary terminator
- update the categorized watcher to parse all ATVM host rows from a grouped Cloud Run Finished block instead of only the first host row
- allow a later terminal grouped summary to replace an earlier RUNNING state for the same logical group so grouped posts are emitted with the full host set
- fix multi-host same-group reporting so batches like ubuntu22.04 plus ubuntu24.04 are posted as one grouped result with both hosts included
- update the categorized watcher to mark a grouped subrun complete once its final check-xml-files.ts artifact exists and the next active host belongs to a different inferred group
- prevent a finished group such as ubuntu from being kept in RUNNING state after execution has already advanced into the next category
- allow grouped Mattermost posts to be emitted at category boundaries instead of being blocked by stale active-group carryover
- update the watcher to merge consecutive categorized child build ids into one logical grouped subrun when they resolve to the same real host-based group
- prevent one real grouped run from being fragmented across multiple watcher subruns just because the raw ci-build-id label changes between hosts
- keep grouped Mattermost posting aligned with the inferred group identity instead of the unstable raw child build id
- update the ATVM automation guide to require a post-template verification gate before launching run-sorry-cypress.py
- require verification that both the generated .ts files and the config specPattern include every requested VM after cmc-templates.py finishes
- document that the runner must not be launched when any requested VM is missing from the generated spec set
- update the ATVM AGENTS rules so this template-to-runner verification step is part of the default automation workflow
- update the categorized watcher to use the latest matching host reporter artifact written before grouped finalization so grouped results can be posted as soon as the host run is actually done
- reduce dependence on the later parent Cloud Run Finished summary block for grouped-run completion reporting
- keep grouped runs from waiting until near parent-run completion when real host-level completion evidence is already available
- keep categorized grouped sub-runs in RUNNING state when only the grouped check-xml-files.ts artifact has appeared and the parent log has not yet exposed the real host summary
- prevent blank Mattermost grouped-run posts with empty host tables, zero counts, and zero/invalid timing caused by posting too early
- require the categorized watcher to wait for host-level grouped summary data before treating the grouped run as complete
- update the watcher to stop trusting misleading categorized child build labels when they do not match the host/spec actually being executed
- infer the reported categorized group name from the actual host being run, so mismatched labels like ubuntu-batch for a Red Hat host are corrected in status reporting
- document the categorized watcher workaround in the ATVM guide, watcher design, and watcher README without changing the underlying ATVM runner scripts
- relax grouped-run duration parsing so Linux categorized summaries like `12m 19.4` and `15m 42.2` are converted reliably even when the trailing `s` is split awkwardly in the parent run log
- keep the categorized grouped-run summary extraction aligned with the host results already being parsed from the parent Cloud Run Finished blocks
- refresh the controller watcher copy so the next categorized run uses the improved grouped duration parser
- update the watcher to parse completed categorized grouped-run host summaries from the parent run log instead of relying only on grouped XML files that often contain only check-xml-files.ts
- add grouped-run duration parsing so categorized sub-run timing can be derived from the Cloud Run Finished summary when host XML details are absent
- fix the completed-summary to grouped-xml alignment so filtered-out older artifacts do not shift host-summary assignment for the current run
- update the watcher to treat categorized parent-run activity as the authoritative signal for whether the overall request is still running
- prevent the watcher from exiting early just because one categorized grouped sub-run completed and wrote artifacts
- document that categorized watcher instances must remain alive between grouped runs until the parent request has actually gone inactive past the grace window
- update the ATVM guide, watcher design, and install docs to reflect the stricter categorized parent-run completion rule
- interpret ATVM controller log timestamps in the controller's local timezone before converting to UTC so the watcher uses the correct current-run window
- prevent newly started categorized runs from immediately picking up older categorized artifacts just because the parent build name was reused
- keep the categorized watcher focused on artifacts from the current controller run instead of stale prior attempts
- update the watcher to detect the active categorized sub-run from the live `--ci-build-id` process state instead of treating the parent run as one synthetic grouped run
- fix host XML parsing so the watcher prefers the real host suite over the `Root Suite` entry, avoiding `0 tests, 0 failures` summaries
- use the first timestamp inside the run log as the watcher start time so restarted watchers do not miss current-run categorized artifacts because of log file mtime drift
- improve active-host inference for categorized runs so the watcher maps the current categorized build to the correct host family while the sub-run is still in progress
- update the watcher start helper to stop any stale watcher instance for the same requested parent build name and remove its old state directory before starting fresh
- document that reused parent build names must not inherit stale cancelled, posted, state.json, or subruns state from older runs
- update the watcher install and design docs so the controller workflow explicitly treats stale reused-build-name state as part of startup cleanup
- update the git guide to require post-commit verification before reporting success
- document that commit verification must confirm git commit exited successfully, HEAD advanced, and the new commit appears in git log
- document that failed or non-created commits must be reported clearly instead of being claimed as successful
- update the watcher design and automation guide to treat --categorize as sequential ATVM sub-runs rather than one parent run with internal phases
- document that categorized runs should send one Mattermost status per completed grouped sub-run instead of one parent-only final post
- add a --categorize option to the watcher start helper so categorized mode is explicit in watcher startup
- update the watcher implementation to track categorized sub-runs separately, write per-subrun state, and post each completed grouped run once
- update the Windows CMC install reference to require CDC project cleanup before reinstall when stale host state may exist
- update the VMware MigrateOps guide so the cleanup-first CMC preparation rule applies to both Linux and Windows sources
- document that Windows reinstall must be followed by a fresh CDC reconnect check before creating migration sessions
- update CDS MCP AGENTS guidance so Windows sources do not skip the standard CDC cleanup plus reinstall sequence
anthony.wen