cds-ai/atvm/docs/automation/guide.md

# Run ATVM Automation Guide

This file is guide-only documentation for operating ATVM CMC automation.
Do not put specific run examples here.
For reusable command examples and common option combinations, use `examples.md`.
Treat `examples.md` as reference-only.
Do not assume the operator wants the extra options shown in examples unless they explicitly request them.

## Purpose
Run ATVM CMC automation tests on the designated automation VM without unintended system or file changes.

## ATVM Cypress Automation Controller Client
- Hostname: `atvm-cypres-vm-1`
- IP: `192.168.3.190`
- Credentials: source `/home/aw/code/cds/.env.credentials.local` and use `ATVM_CONTROLLER_USER` plus `ATVM_CONTROLLER_PASSWORD`

## ATVM Target Host Default
- Treat `192.168.3.191` as the default ATVM target host reference.
- For SSH to `192.168.3.191`, ignore host key mismatch by default with `-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null`.
- For Linux SSH access to `192.168.3.191`, source `/home/aw/code/cds/.env.credentials.local` and use `ATVM_TARGET_USER` plus `ATVM_TARGET_PASSWORD` unless the operator explicitly overrides them.
- `ATVM_LINUX_TARGET_HOST`, `ATVM_LINUX_TARGET_USER`, and `ATVM_LINUX_TARGET_PASSWORD` mirror the Linux default values when an OS-specific reference is clearer.
- For Windows guest access to `192.168.3.191`, source `/home/aw/code/cds/.env.credentials.local` and use `ATVM_WINDOWS_TARGET_USER` plus `ATVM_WINDOWS_TARGET_PASSWORD` unless the operator explicitly overrides them.

## Operating Constraints
- Run only scripts/commands explicitly requested.
- Do not make manual system configuration changes on the client.
- Do not edit client files unless explicitly requested.

## Operator Preferences
- Do not include Gold Disk identifiers in `--build_name`.
- `--build_name` must not contain spaces; use `-` between words.
- For multiple VMs in same distro, use distro-scoped filtering (`--containsVm`) instead of long explicit VM lists.
- Always include `--ignore_force_shutdown` on `cmc-templates.py` commands unless the operator explicitly asks not to.
- Always include `--test_partition` on `cmc-templates.py` commands unless the operator explicitly asks not to.
- Default plugin-bearing templates to `--use_specified_plugin iscsi` unless the operator explicitly requests a different plugin.
- Do not add plugin or integration-type arguments to `cmc-systemOS`; that template should be planned without `--use_specified_plugin`, without `--integration_type`, and without watcher integration/plugin metadata.
- For `cmc-migrateops-compute-migration`, default to `--set_static_ip_dest` unless the operator explicitly says otherwise.
- For `cmc-migrateops-compute-migration` to VMware, default to `--vm_platforms vmware` unless the operator explicitly says otherwise.
- For ATVM automation runs that involve Windows guests, default `run-sorry-cypress.py` to `--hang_retries 0` unless the operator explicitly says otherwise.
- For `cmc-reboot`, treat `--use_specified_plugin both` as an exception case that requires an extra confirmation.
- When `cmc-reboot` is planned with `--use_specified_plugin both`, warn that FC+iSCSI together may hit a "chicken before the egg" timing problem where iSCSI disks are not attached before mTDI / CMC services start.
- For `cmc-reboot`, prefer `--use_specified_plugin fc` or `--use_specified_plugin iscsi` unless the operator explicitly reconfirms that `both` is really intended after seeing that warning.
- Before preparing a new run, always check whether automation is already running.
- Treat a prior status check as stale once control returns to the operator or a new ATVM request arrives; perform a fresh live controller check at request time instead of relying on the immediately previous result.
- Always report whether automation is currently running.
- If running, ask whether to terminate; terminate only with explicit approval.
- After termination approval, terminate first, then execute the new run command set.
- By default, execute `cmc-templates.py` and `run-sorry-cypress.py` without a pre-run approval gate.
- If the operator explicitly asks to review planned commands first, show them before execution.
- If the operator changes any part of the request before execution, rebuild commands and execute the revised command set.
- Default to watcher-backed execution for every run unless the operator explicitly asks to run without watcher.
- When `--categorize` is used with watcher enabled, treat the watcher as a sequential grouped-run watcher:
  - it must post one final Mattermost status per completed categorized group/sub-run
  - it must stay active between grouped sub-runs while the parent categorized request is still running
  - it must not stop after the first grouped run simply because one grouped run completed
  - if the child build id label does not match the actual host/spec being executed, report the grouped run using the inferred host-based group instead of the raw child build id label
  - it must not wait and replace those with one single parent-only post
- After execution, report immediate success/failure only.
- After execution, include the exact executed `cmc-templates.py` and `run-sorry-cypress.py` commands in the response.
- Do not include expected, harmless `systemctl reset-failed ... unit not loaded` output in routine run-start confirmations.
- Mention `reset-failed` output only when it prevents watcher startup or becomes relevant to debugging.
- Do not actively monitor completion unless explicitly requested.
- If monitoring is requested, allow long runtime windows (15-30+ minutes) and continue until completion unless operator instructs otherwise.
- Report command errors immediately.
- `sshpass` may be used where password-based SSH automation is required.
- Treat runner hang-kill events (`Sending SIGKILL ... due to no change` / `Max hang retries reached`) as explicit `FAILED` outcomes, not `RUNNING` or ambiguous termination.
- For manual `run-sorry-cypress.py` execution, treat `ATVM_HANG_FAIL ...` log markers and `/tmp/atvm-runner-state-<build>.json` terminal state files as the source of truth for hang-failure terminal status.

## Core Scripts
- Template prep: `/root/cdc-e2e-cyp-12.17.4/cmc-templates.py`
- Test execution: `./run-sorry-cypress.py`
- Detailed host-level test artifacts: `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter`

## Detailed Test Artifacts
- Use `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter` on the automation controller for detailed per-host test evidence.
- Reporter subdirectories of interest:
  - `logs/`
    - per-host text and JSON logs for the executed tests
  - `xml/`
    - machine result XML files and the final `check-xml-files.ts` bookkeeping output
  - `mochawesome/`
    - per-run HTML reports
- When a machine fails, use the matching `logs/` entry first to capture the detailed failure context for that host.
- Apply the failed-host detail recovery path to every ATVM template type, not just reboot.
- For any failed host, recover detail in this order when available:
  - consolidated run log
  - matching `mochawesome` HTML
  - structured reporter artifacts such as per-host JSON or XML
  - text reporter artifacts
- When reconstructing historical status, prefer `cmcReporter` artifacts over less-specific runner output because they preserve per-host results after the live run has ended.
- Do not treat the existence of a per-host reporter artifact by itself as proof that the host passed.
- For categorized grouped recovery, prefer the matching per-host reporter JSON or mochawesome result and carry through the real `failures`, `pending`, and failure message instead of assuming `PASS completed`.
- If grouped XML only contains `check-xml-files.ts`, cross-check the grouped result against the per-host reporter artifacts before posting or repeating status for that grouped sub-run.
- Do not report a categorized grouped sub-run as `PASS` from watcher `host_results`, grouped XML, or a lone `check-xml-files.ts` result by itself.
- Before reporting a categorized grouped sub-run as `PASS`, confirm that the matching child batch also passed in the live launch log or the final `Cloud Run Finished` summary for that child run.
- Treat saved watcher state under `/var/lib/atvm-run-watcher/<build>/state.json` as cached status only.
- For completed-run verification, confirm in this order:
  - launch log under `/tmp/<build>.launch.log`
  - matching `cmcReporter` artifacts
  - `Cloud Run Finished` summary and Currents URL
  - saved watcher state only as a comparison layer
- If saved watcher state disagrees with the launch log or with a replay of the exact artifacts through the current watcher code, treat the saved state as stale and do not use it as the reported result.
- Never confirm a completed run from `state.json` alone.

Typical sequence:
1. Build the exact `cmc-templates.py` and `run-sorry-cypress.py` commands for the request.
2. Run `cmc-templates.py` with the requested options.
3. Wait for `cmc-templates.py` to fully finish and confirm success.
4. Verify the generated `.ts` files and the config `specPattern` include every requested VM before starting the runner.
5. By default, use watcher-backed execution unless the operator explicitly asks not to.
6. For watcher-backed runs, make sure the controller's deployed watcher code is the intended version before relying on its posts.
7. For watcher-backed runs, build the watcher-start command so it automatically includes the exact `cmc-templates.py` command via `--template-command` and the exact `run-sorry-cypress.py` command via `--runner-command`.
8. For watcher-backed runs, prefer the controller-local `atvm-runner@...` systemd service instead of detached SSH background launch patterns for `run-sorry-cypress.py`.
9. For watcher-backed runs, start the watcher before launching the runner service.
10. Start the runner with the matching config and build name.
11. Report immediate start success/failure and include the exact executed template and runner commands.

Completed-run verification sequence:
1. Read the launch log for the build.
2. Inspect the matching reporter artifacts for the relevant host(s).
3. Use the `Cloud Run Finished` summary and Currents URL as the final parent-run check when present.
4. Compare that result against saved watcher state.
5. If there is any disagreement, replay the exact artifacts through the current watcher code in an isolated temp state directory before confirming the result.
6. For categorized runs, do not let a `check-xml-files.ts` child result override a failing child batch shown in the launch log or `Cloud Run Finished` summary.

## Config File / Gold Disk Mapping
- `cypress.atvm-config-gold.ts` -> Gold Disk 1
- `cypress.atvm-config-gold-2.ts` -> Gold Disk 2
- Additional numbered config variants map to corresponding Gold Disks.
- Do not default to `cypress.atvm-config.ts`.
- Unless the operator explicitly requests another config, use a config file with `gold` in the filename.
- If the operator-specified config file is missing, stop immediately and report the missing file.
- Do not search for substitute ATVM config files and do not switch to another config unless the operator explicitly instructs it.
- Treat `AutomatedTest-VMBootImg-Gold` as `gold` and `AutomatedTest-VMBootImg-Gold-2` as `gold-2`.
- Use live vCenter inventory as the source of truth for current VM membership on those datastores.
- Query vCenter datastore membership at request time when selecting `gold` vs `gold-2`; do not maintain or rely on a repo-side live reference file for this decision.
- When the operator asks to inventory or show the contents of `AutomatedTest-VMBootImg-Gold` and `AutomatedTest-VMBootImg-Gold-2`, return hostname-only VM lists unless the operator explicitly asks for more detail.
- When the operator provides an explicit VM list, check vCenter placement for every requested VM before choosing the config file.
- Before presenting any ATVM run commands for an explicit VM-list request, tell the operator that the next step is a live vCenter placement check for the requested VMs and that the result will determine whether the run must use `gold` or `gold-2`.
- For vCenter inspection and placement checks, prefer `govc` and raw vCenter REST calls when they are available before reaching for alternate wrappers.
- For `govc`-based placement checks, use `govc vm.info -json <vm>` and parse the lowercase JSON keys such as `virtualMachines` and `datastore`.
- Resolve each returned datastore managed-object reference to a datastore name with `govc object.collect -s <datastore-ref> name` before deciding between `gold` and `gold-2`.
- Ignore non-boot helper datastores such as install ISO attachments when applying the `gold` vs `gold-2` rule; base the family decision on the ATVM boot datastore membership.
- If every requested VM is on `AutomatedTest-VMBootImg-Gold`, plan the run with the `gold` config.
- If every requested VM is on `AutomatedTest-VMBootImg-Gold-2`, plan the run with the `gold-2` config.
- If the requested VM set spans both `AutomatedTest-VMBootImg-Gold` and `AutomatedTest-VMBootImg-Gold-2`, stop immediately and do not prepare or run the test.
- For a mixed-datastore request, report it as a discrepancy, list which requested VMs are on `AutomatedTest-VMBootImg-Gold` and which are on `AutomatedTest-VMBootImg-Gold-2`, tell the operator they need to correct the list, and ask whether they want the full VM inventories for both datastores so they can adjust the request.
- Do not run an ATVM test against a mixed set of VMs from both `AutomatedTest-VMBootImg-Gold` and `AutomatedTest-VMBootImg-Gold-2`.

## Available Templates
- `cmc-e2e`
- `cmc-group-consistency`
- `cmc-h2h-diff-platf`
- `cmc-h2h-same-platf`
- `cmc-migrateops`
- `cmc-migrateops-compute-migration`
- `cmc-reboot`
- `cmc-systemOS`

## Command Pattern
```bash
python3 cmc-templates.py --template <template> --ignore_force_shutdown --test_partition --config_file_path ./<config-file> [template-specific plugin/integration options when that template actually uses them...]; \
python3 ./run-sorry-cypress.py --config_file <config-file> --build_name <hyphenated-description-no-spaces> [--categorize]
```

## Examples Reference
- Commonly used command examples: `examples.md`
- Keep this guide focused on run-control rules and workflow constraints.
- Use examples as reference material only, not as default intent for new operator requests.
- Keep `examples.md` limited to reusable example commands; keep workflow rules, defaults, blacklist policy, and reporting rules in this guide or `run-learnings.md`.
- When validating plugin-specific ATVM requests, do not treat the presence of plugin-specific code blocks or step labels in the generated `.ts` file as proof that those steps will execute.
- Determine plugin behavior from the template/runtime gating logic, for example `Cypress.env(...)`-driven conditionals, and only call it a mismatch when that runtime logic would execute the wrong plugin path.

## Example Option Patterns (Guide-Only)
- Distro-scoped VM selection:
  - `--containsVm redhat`
  - `--containsVm redhat9`
- Explicit VM selection:
  - `--specify_vms <vm1> <vm2> ...`
- Compute migrateops platform:
  - `--vm_platforms vmware|ovirt|openshift|proxmox`
- Default compute migrateops options:
  - `--ignore_force_shutdown`
  - `--test_partition`
  - `--set_static_ip_dest`
- Default compute migrateops vmware options:
  - `--ignore_force_shutdown --vm_platforms vmware --test_partition --set_static_ip_dest`

## Blacklisted Machines
Always exclude these machines from broad-scope ATVM automation runs by adding them to `--exclude_partial_match`.
If the operator explicitly targets one or more named VMs with `--specify_vms`, do not add the maintained `--exclude_partial_match` list unless the operator also explicitly asks for it.
Even for explicit `--specify_vms` requests, first check whether any requested VM is on the maintained blacklist and stop instead of launching the run if one is included.

Permanently blacklisted because CMC cannot compile:
- `atvm6-centos6.0`
- `atvm41-redhat6.0`
- `atvm73-oracle6.0`

Temporarily blacklisted because the run crashes when creating a migration session:
- `atvm144-suse15.0`

Temporarily blacklisted while support requests are waiting:
- `atvm113-debian9.0.0`
- `atvm115-debian9.1.0`
- `atvm116-debian9.2.0`

Temporarily blacklisted because re-creation might be needed:
- `atvm156-debian9.3.0`

Preferred exclude list:
- `--exclude_partial_match atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0`

## Running-Automation Check (Mandatory)
Before any new automation request:
1. SSH to `root@192.168.3.190`.
2. Check for active automation processes (for example `run-sorry-cypress.py`, `cmc-templates.py`, and related Cypress runners).
3. Treat every new operator request to start, replace, or block on an ATVM run as requiring a new live controller check, even if a status check was performed earlier in the same conversation.
4. Do not reuse the result of the immediately previous running-state check when deciding whether to block or allow a new ATVM run request.
5. Report:
   - `Running` with process details, or
   - `Not running`.
6. If `Running`, ask operator whether to terminate.
7. If termination is approved, terminate matching process(es), confirm termination, then proceed to planned-command approval.
8. If termination is not approved, do not start a new run.

## Execution Workflow (Mandatory)
1. Build exact command(s) for the request.
2. By default, execute without a pre-run approval gate. If the operator explicitly asks for command review first, show planned commands before running.
3. When both template generation and the Cypress runner are requested, run them sequentially, not in parallel.
4. Do not launch the ATVM runner until `cmc-templates.py` has exited successfully and finished updating the intended config/spec files.
5. After `cmc-templates.py`, always verify that the generated spec files on disk and the config `specPattern` both contain the full requested VM set before launching the ATVM runner.
6. If any requested VM is missing from the generated files or `specPattern`, stop and report the mismatch instead of launching the runner.
7. If the operator asks to change plugin, config, filters, build name, Gold Disk, or scope before execution, discard the old plan and execute only the revised command set.
8. If the planned command is `cmc-reboot` with `--use_specified_plugin both`, add the FC+iSCSI timing warning and require explicit confirmation that `both` is intended before execution.
9. Default to watcher-backed execution unless explicitly told not to.
10. When watcher-backed execution is used, the watcher-start command must automatically include the exact executed `cmc-templates.py` command via `--template-command` and the exact executed `run-sorry-cypress.py` command via `--runner-command`.
11. If the run uses `--categorize` and watcher is enabled, include `--categorize` on the watcher start command so the watcher tracks sequential categorized sub-runs correctly.
12. When watcher-backed execution is used, launch the watcher before the runner service.
13. Do not start the runner before the watcher, because the watcher helper clears stale `/tmp/<build-name>.log` and can delete the fresh live runner log if the runner starts first.
14. Prefer the combined `start-atvm-run.sh` wrapper when both services are used, so watcher and runner are not launched in parallel.
15. If monitoring was not requested, report immediate success/failure for each command.
16. If monitoring was requested, keep monitoring until completion and report final outcome.
17. After execution, always include the exact executed `cmc-templates.py` and `run-sorry-cypress.py` commands in the response.

## Requested Test Style
When asked for one VM or a VM set:
- choose requested template/options,
- if the request includes explicit VM names, determine each VM's datastore placement in vCenter before selecting the config file,
- before showing run commands, tell the operator you are about to do the live vCenter placement check and that it will decide the config family,
- use only live vCenter datastore membership when resolving that placement,
- choose correct config file for intended Gold Disk,
- if the explicit VM set is split between `AutomatedTest-VMBootImg-Gold` and `AutomatedTest-VMBootImg-Gold-2`, interrupt the request and report the per-datastore split instead of planning commands,
- if the operator asks to see the datastore inventories, show hostname-only lists by default,
- default to a config filename containing `gold` unless the operator explicitly says otherwise,
- always include `--ignore_force_shutdown` on the template-generation command unless the operator explicitly overrides that default,
- always include `--test_partition` on the template-generation command unless the operator explicitly overrides that default,
- default plugin-bearing templates to `--use_specified_plugin iscsi` unless the operator explicitly requests another plugin or the template does not use plugin selection,
- if the operator explicitly requests `cmc-group-consistency` with one VM, allow the single-VM request and plan it without asking for additional VMs,
- treat `cmc-systemOS` as a template that does not use plugin selection or integration-type selection, so omit `--use_specified_plugin`, omit `--integration_type`, and omit watcher integration/plugin metadata for that template,
- use a descriptive `--build_name` without Gold Disk IDs.

## Update Rule
- After each run, update this guide only for workflow/rule/default changes.
- Update `examples.md` for reusable command/option examples.
- Add run-specific learnings only to `run-learnings.md` when the run produced new information.

## Monitoring Policy
- Monitor only when the operator explicitly asks to monitor.
- If monitoring was not requested, run commands and report execution success/failure and any errors.
- If monitoring was requested, do not terminate processes automatically; only terminate if the operator explicitly instructs termination.

## Mattermost Status Posting
- Treat a normal ATVM status request as local-only output by default.
- When the operator asks to send ATVM automation run status to Mattermost, use the local defaults from `/home/aw/code/cds/.env.credentials.local`.
- Default Mattermost variables:
  - `MATTERMOST_ATVM_WEBHOOK`
  - `MATTERMOST_ATVM_CHANNEL`
- Treat these as the default destination for ATVM automation run-status posts unless the operator explicitly overrides them.
- Send the final ATVM run status only after the run has fully completed, regardless of whether the run passed or failed.
- Do not send interim or in-progress ATVM run status updates to Mattermost unless the operator explicitly asks for that.
- Use the same ATVM status layout that would be shown to the operator locally when posting to Mattermost.
- Default status template: `/home/aw/code/cds/atvm/docs/automation/status-template.md`
- Do not post to Mattermost unless the operator explicitly asks for the run status to be sent there.
- For categorized execution with watcher enabled, send one Mattermost status per completed categorized sub-run/group after that grouped run fully finishes.

## Status Reporting Format
When the operator asks for the status of an ATVM automation run, report in this order:
1. Heading/title using the run `build_name`.
2. `SUMMARY:` section with finished, passed, failed, and skipped counts.
3. `HOSTS:` section with the machine rows.
4. `TIMING:` section with start, end, total, quickest, longest, and average.
5. `COVERAGE:` section describing what the run was intended to cover, excluding the target-host list.
6. `TEST FLOW:` section describing the template-specific numbered run flow for the test.
7. `FAILURE NOTES:` section for detailed per-host error excerpts when failures exist.
8. `NOTES:` section for broader non-failure context and anomalies.
7. Remaining machines still to run.
8. Summary counts for finished, passed, failed, and skipped machines.
9. Timing details:
   - start time
   - end time if complete
   - total run time if complete, or elapsed run time if still running
   - quickest completed test runtime
   - longest completed test runtime
   - average completed test runtime
10. Estimated completion time.

Status-report expectations:
- Use the same display layout for every ATVM automation status response regardless of test type (`e2e`, `systemOS`, `reboot`, `migrateops`, and others).
- Use `/home/aw/code/cds/atvm/docs/automation/status-template.md` as the default template for both local status output and Mattermost status posts.
- The default ATVM status template uses flat bullet-list sections for `COVERAGE:`, `TEST FLOW:`, `FAILURE NOTES:`, and `NOTES:`, and Markdown tables for `SUMMARY:`, `HOSTS:`, and `TIMING:`.
- Order the status sections as `SUMMARY:`, `HOSTS:`, `TIMING:`, `COVERAGE:`, `TEST FLOW:`, `FAILURE NOTES:`, then `NOTES:`.
- Keep `NOTES:` focused on non-failure operator-facing value such as the Currents run URL, real anomalies unrelated to the direct failure text, or material fallback behavior.
- Include the exact `cmc-templates.py` command used to trigger the ATVM automation run in `NOTES:`, without the outer `sshpass`/`ssh` wrapper and without trimming it.
- Include the exact `run-sorry-cypress.py` command used to launch the ATVM automation run in `NOTES:`, without the outer `sshpass`/`ssh` wrapper and without trimming it.
- Do not include generic watcher bookkeeping messages in `NOTES:` such as artifact-detection confirmations.
- Do not include internal watcher fallback notes in `NOTES:` such as `check-xml-files.ts` validation confirmations or reporter-artifact recovery details.
- The `HOSTS:` table includes `Host`, `Kernel`, `Status`, and `Detail` columns in that order.
- For any failed host, keep the `Detail` column compact by showing the failing step plus a short error summary, not the full raw stack trace.
- If richer failure text is available, put the longer trimmed excerpt in `FAILURE NOTES:` so the result stays readable in Mattermost and local status output.
- In `COVERAGE:`, describe the important `cmc-templates.py` command inputs such as template, categorize mode, datastore/config family, config filename, migration style, any real plugin/integration path, and other operator-relevant run options, but do not list target hosts there or include verbose prose scope descriptions.
- Only include coverage fields that the template command actually used. Do not show empty or irrelevant fields such as an integration/plugin path for templates that did not use one.
- If `categorize mode: enabled` is already shown in `COVERAGE:`, do not also repeat `--categorize` under `run options`.
- When grouped categorized timing is reconstructed from host reporter artifacts, derive per-host quickest/longest/average durations from the sequence of recovered host timestamps and the grouped end time instead of leaving those metrics as `n/a`.
- In `TEST FLOW:`, show the template-specific numbered run flow once for the whole test, not per host.
- For `TEST FLOW:`, treat the generated host spec from the actual run as the source of truth whenever it exists.
- Extract the numbered flow steps from the generated `.ts` spec referenced by that run's `specPattern`.
- When the generated spec contains runtime-gated plugin branches such as `if(useFCPlugin)`, `if(useIscsiPlugin)`, `if(usePureFCPlugin)`, or `if(usePureIscsiPlugin)`, only include the steps for the plugin path actually selected for that run.
- Do not prefer a static template flow list over a generated spec from the actual run.
- Use template-level or static fallback flow only when the generated spec cannot be found or parsed.
- If fallback is required, resolve it from the run template name before using any generic default flow.
- `cmc-e2e` currently uses the 22-step migration flow documented in `/home/aw/code/cds/atvm/docs/automation/status-template.md`.
- `cmc-systemOS` currently uses the 21-step boot-disk migration flow documented in `/home/aw/code/cds/atvm/docs/automation/status-template.md`.
- Keep `FAILURE NOTES:` and `NOTES:` behavior consistent across template types; do not add template-specific internal-source notes such as parent-log-summary recovery details.
- For the `Kernel` column, cross-reference the host name against `/home/aw/code/cds/atvm/inventory/vm-inventory.md`.
- If the hostname is not present in `vm-inventory.md`, report the kernel value as `unknown`.
- Treat references to the "ATVM automation run" or "automation run" as referring to this ATVM folder workflow and the automation VM at `192.168.3.190`, not to Cirrus project operations such as the `atvm - cypress` project.
- Treat a status request as a request for live status by default.
- Unless the operator explicitly asks to send the status to Mattermost, print the status only in the local terminal response.
- Use the live automation VM state when available.
- If no automation is currently running, fall back to the most recent historical run artifacts and logs.
- Prefer local automation evidence in this order: active runner processes, live automation-VM files, shell history for the last launch command, then historical reporter artifacts.
- For detailed machine-level failure information, use `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter/logs/` on the automation VM.
- Derive the heading/title from the run `build_name` when available.
- Format every machine entry as `machine-name - STATUS`.
- Put each machine on its own line; never combine multiple machines into one paragraph or comma-separated line.
- Use a separate `Notes` section for failure reasons, anomalies, or operator-relevant context rather than cramming those details into the completed-machine list.
- For categorized runs, reconstruct the whole run across all category batches; do not treat the current live category batch as the full run scope.
- For categorized runs with no active automation, reconstruct the status from the full historical run across all category batches, not only the most recent category batch.
- Always report the status of the entire requested run, even when the runner split execution into multiple category batches or cloud sub-runs.
- Derive completed-machine status from completed spec results already written during the same run.
- Parse all same-run `test-result-*.xml` files, not only machine-named `test-result-atvm*.xml` files.
- When XML filenames are hash-named, extract the machine name from XML contents such as `testsuite file=`, `testsuite name=`, or `testcase name=`.
- Ignore `check-xml-files.ts` XML outputs when counting machine completion because they are bookkeeping steps, not machine runs.
- When multiple same-run XML files exist for one machine, use the most recently written XML for that machine.
- Include the run start time in every status response when it can be derived from the run log.
- If the run is complete, include the end time and total run time.
- If the run is still active, include the elapsed run time so far.
- Include quickest completed test runtime, longest completed test runtime, and average completed test runtime under timing details when they can be derived from the run log.
- Show blacklisted machines under skipped machines even if they are part of the broader machine family requested by the operator.
- For skipped machines, include the reason category:
  - `BLACKLISTED: CMC INSTALL - CAN'T COMPILE`
  - `BLACKLISTED: SUPPORT REQUEST - WAITING`
  - `BLACKLISTED: RE-CREATE MIGHT BE NEEDED`
  - `BLACKLISTED: RE-CREATE NEEDED`
- If a machine is currently in progress, show it under remaining machines as `RUNNING`.
- If a machine has not started yet, show it under remaining machines as `NOT STARTED`.
- If no failures are present in completed spec results, report those completed machines as `PASS`.
- If a completed spec result shows a failure, report that machine as `FAIL` in the completed list and append a longer same-line failure description when the extra detail is useful to the operator.
- Use `Notes` for extra context beyond the machine-specific same-line failure description.
- Base the completion estimate on the full remaining machine count and recent per-machine runtime visible in the run log.
- Make the estimate explicitly refer to completion of the entire remaining run, not only the current machine/spec.
- When the operator also asks to send the status to Mattermost, send this same final status output to the configured Mattermost destination only after the run has fully completed.