anthony.wen/cds-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add portable ATVM skill and hardware prep docs

Browse Source
This commit is contained in:
Anthony Wen
2026-07-14 20:11:15 -04:00
parent 744b28affe
commit 67886174b3
24 changed files with 3838 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/atvm/SKILL.md
+256
View File
@@ -0,0 +1,256 @@
---
name: atvm
description: Portable ATVM operations skill for setup/bootstrap, ATVM Cypress automation planning and execution, watcher-aware run control, live and historical run-status reporting, Mattermost posting on request, and maintenance of ATVM workflow docs and inventory. Use when working with an ATVM target host, the ATVM automation controller, or ATVM-specific workflow rules, defaults, and artifacts.
---
# ATVM
Use this skill for ATVM setup, ATVM Cypress automation, ATVM run monitoring, watcher-aware launch flow, run-status reporting, and maintenance of ATVM operational documentation.
This is a portable packaged skill. Prefer the bundled `references/` docs first. If a live repo copy of the ATVM workspace exists, use it as an execution source when appropriate, but do not assume it exists.
## Read First
Pick only the files needed for the current task.
### Setup/bootstrap
- `references/setup-guide.md`
- `references/setup-run-learnings.md`
- `references/atvm-prep-reference.md`
### Automation execution
- `references/automation-guide.md`
- `references/automation-examples.md`
- `references/automation-run-learnings.md`
- `references/automation-status-template.md`
### Workspace maintenance conventions
- `references/workflow-conventions.md`
### Environment reference
- `references/inventory-overview.md`
- `references/infrastructure.md`
- `references/reserved-ips.md`
- `references/specialized-reference.md`
- `references/vm-inventory.md`
### Credentials and local environment contract
- `references/credentials-contract.md`
### Historical long-form notes
- `references/imported-notes-cypress-automation-for-cmc.md`
## Operating Model
This skill is execute-by-default for ATVM automation runs.
Always:
- treat git/commit approval as separate from ATVM run execution
- never execute `git push` on the operator's behalf unless a different workspace policy explicitly overrides that rule
- default ATVM automation runs to watcher-backed execution unless the operator explicitly requests no watcher
- after run launch, report the exact executed `cmc-templates.py` and `run-sorry-cypress.py` commands
Only when explicitly requested by the operator:
- show planned ATVM commands before execution as a review step
## Core Defaults
### Setup track
Default ATVM target host values:
- target host reference: `192.168.3.191`
- static IP target: `192.168.3.191/22`
- gateway: `192.168.0.1`
- DNS: `8.8.8.8`, `8.8.4.4`
- client log file: `atvm_setup_script.log`
Default Linux target credentials:
- `ATVM_TARGET_USER`
- `ATVM_TARGET_PASSWORD`
Default Windows target credentials:
- `ATVM_WINDOWS_TARGET_USER`
- `ATVM_WINDOWS_TARGET_PASSWORD`
For SSH to the default target host, ignore host key mismatch by default:
- `-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null`
Use `references/setup-guide.md` for the hardware-prep trigger phrase:
- `finish prepping the ATVM VM hardware for <vm-name>`
That procedure adds the standard two ATVM data disks, two FC passthrough ports, and full guest-memory reservation for the exact VM name supplied by the operator.
### Automation track
Default controller values:
- controller host: `atvm-cypres-vm-1`
- controller IP: `192.168.3.190`
Default controller credentials:
- `ATVM_CONTROLLER_USER`
- `ATVM_CONTROLLER_PASSWORD`
Detailed controller artifact root:
- `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter`
Automation defaults:
- default plugin-bearing template plugin: `--use_specified_plugin iscsi`
- default config family: `gold`
- do not include Gold Disk identifiers in `--build_name`
- `--build_name` must not contain spaces; use `-` between words
- for multiple VMs in the same distro, prefer distro-scoped filtering (`--containsVm`) over long explicit VM lists
- always include `--ignore_force_shutdown` unless explicitly told not to
- always include `--test_partition` unless explicitly told not to
- for `cmc-migrateops-compute-migration` to VMware, default to:
- `--vm_platforms vmware`
- `--set_static_ip_dest`
- for vCenter inspection and placement checks, prefer `govc` and raw vCenter REST calls before alternate wrappers when available
- treat `cmc-systemOS` as not using plugin or integration-type arguments; do not add `--use_specified_plugin`, `--integration_type`, or watcher integration/plugin metadata for that template
- do not auto-add a maintained `--exclude_partial_match` blacklist when the operator explicitly targets named VMs with `--specify_vms`
- even for explicit `--specify_vms` requests, first check whether any requested VM is on the maintained blacklist and stop if it is
## Mandatory Safety Rules
### Setup
- never run setup without both `--expected-ip` and `--expected-hostname`
- keep static IP configuration as the final setup step
- do not weaken host identity validation
- for ATVM hardware prep, operate only on the exact VM name, verify the VM is powered off before hardware changes, and never delete, destroy, remove from inventory, or unregister the VM
### Automation
- before any automation run, always check whether automation is already running
- treat any previous controller running-state result as stale once a new ATVM request arrives; perform a fresh live controller check at request time instead of relying on the immediately previous result
- always report whether automation is already running
- if automation is already running, terminate only with explicit approval
- after `cmc-templates.py`, always verify that generated spec files and the config `specPattern` still include every requested VM before launching `run-sorry-cypress.py`
- if any requested VM is missing, stop and report the mismatch
- when watcher mode is in use (default), start the watcher before `run-sorry-cypress.py`
- when watcher mode is in use (default), build the watcher-start command so it automatically includes the exact executed `cmc-templates.py` command via `--template-command` and the exact executed `run-sorry-cypress.py` command via `--runner-command`; do not require the operator to restate them separately
- when watcher mode is in use (default), prefer the controller-local `atvm-runner@...` systemd service over detached SSH background launch patterns for `run-sorry-cypress.py`
- do not start the runner before the watcher, because watcher startup may clear stale `/tmp/<build>.log` files and can delete a fresh live runner log if the runner starts first
- when starting both watcher and runner, prefer the combined `start-atvm-run.sh` wrapper so they are not launched in parallel against the same `/tmp/<build>.log`
- do not infer plugin behavior from stray strings in generated specs; determine it from runtime gates such as `Cypress.env(...)`
- for `cmc-reboot`, treat `--use_specified_plugin both` as a separate confirmation gate and warn about the FC+iSCSI timing risk unless the operator explicitly reconfirms `both`
- for vCenter VM snapshot requests, default the snapshot name to `VM Snapshot [mm/dd/yyyy:hh:mm:ss AM/PM]` in the local `America/New_York` timezone unless the operator explicitly requests a different name
- never delete, destroy, remove from inventory, or unregister any ATVM VM from vCenter
- treat a vCenter VM as protected ATVM inventory if its name starts with `atvm`, if it resides on an `AutomatedTest-*` ATVM datastore, or if it is an ATVM platform/controller VM, even when it is not listed in `inventory/vm-inventory.md`
- preserve ATVM vCenter VM inventory records even when power, maintenance, datastore, cleanup, or prep work is requested; for cleanup, only power off, detach/unmount supporting resources, or report the needed manual action
### Red Hat manual tasks (non-automation)
- for manually requested tasks on Red Hat machines, run the subscription reset/register sequence before the requested task
- do not apply this step to ATVM automation runs where template/runtime scripts already handle the subscription flow
- use credentials from the local environment source:
- `REDHAT_SUBSCRIPTION_USER`
- `REDHAT_SUBSCRIPTION_PASSWORD`
- required sequence:
- `subscription-manager remove --all`
- `subscription-manager unregister`
- `subscription-manager clean`
- `subscription-manager register --username "$REDHAT_SUBSCRIPTION_USER" --password "$REDHAT_SUBSCRIPTION_PASSWORD"`
### Status confirmation
Treat watcher state as cached output only:
- `/var/lib/atvm-run-watcher/<build>/state.json`
Before confirming a completed run, verify in this order:
1. launch log
2. matching reporter artifacts
3. `Cloud Run Finished` summary / Currents URL
4. watcher state only as comparison
Never confirm a completed run from watcher state alone.
For categorized runs:
- never report a grouped sub-run as `PASS` from watcher `host_results`, grouped XML, or a lone `check-xml-files.ts` result by itself
- before reporting a grouped sub-run as `PASS`, confirm that the matching child batch also passed in the live launch log or the final `Cloud Run Finished` summary for that child run
### Mattermost
- if the operator asks for ATVM run status without mentioning Mattermost, respond locally only
- if the operator explicitly asks to send status to Mattermost, use the locally configured:
- `MATTERMOST_ATVM_WEBHOOK`
- `MATTERMOST_ATVM_CHANNEL`
- send the final status only after the run fully completes
## Credentials Contract
Do not bundle real credentials inside the skill.
Use this resolution order:
1. operator-provided env vars in the current shell
2. operator-provided local env file
3. machine-local default env file if one is known and present
Expected variables:
- `ATVM_TARGET_USER`
- `ATVM_TARGET_PASSWORD`
- `ATVM_WINDOWS_TARGET_USER`
- `ATVM_WINDOWS_TARGET_PASSWORD`
- `ATVM_CONTROLLER_USER`
- `ATVM_CONTROLLER_PASSWORD`
- `MATTERMOST_ATVM_WEBHOOK`
- `MATTERMOST_ATVM_CHANNEL`
If credentials are missing, stop and ask for the local source or the missing variable names. Do not invent fallback secrets.
## Execution Entry Points
Prefer documented scripts and commands over ad hoc rewrites.
### Setup entry points
- client setup script: `scripts/atvm-setup-script.sh` when available in the current workspace
- controller wrapper: `scripts/run-atvm-setup-and-collect-log.sh` when available in the current workspace
- controller prep script: `/root/atvm_prep/atvm_prep.py` on `192.168.3.190`; show the exact planned command and wait for explicit operator approval before execution
If the local ATVM repo is not present, use the procedural rules in the bundled references and ask the operator for the actual script location before execution.
### Automation entry points
Typical controller-side tools:
- `cmc-templates.py`
- `run-sorry-cypress.py`
Treat their exact location as environment-specific unless a local repo or controller path is available.
## Artifact Recovery and Reporting
For detailed host-level failure analysis, prefer:
- controller run log
- `cmcReporter/logs/`
- `cmcReporter/mochawesome/`
- structured reporter artifacts such as JSON and XML
- plain text reporter artifacts last
Use the status layout from `references/automation-status-template.md`.
Keep the `HOSTS` detail compact.
Put richer failure excerpts in `FAILURE NOTES:`.
Reserve `NOTES:` for non-failure context such as template command, Currents URL, and operator-facing caveats.
Apply failed-host detail recovery consistently for every ATVM template run, not just `cmc-reboot`.
For failed hosts, recover detail in this order when available: consolidated run log, `mochawesome`, structured reporter artifacts such as JSON and XML, then plain text artifacts.
When reporting `TEST FLOW:`, prefer the numbered steps extracted from the generated spec for that exact run.
If the generated spec exists, do not rely on a static template flow list.
Only fall back to template-level or static flow definitions when the generated spec cannot be located or parsed.
## Maintenance Rules
When changing workflow behavior:
- update the relevant guide content
When adding reusable command patterns:
- update the automation examples content
When a run produces a new lasting lesson:
- update the relevant run learnings content
Keep durable environment reference in the inventory reference files.
Do not remove detailed inventory or credential contract information unless explicitly instructed.
## Practical Use Pattern
1. Classify the task as setup, automation execution, status/reporting, watcher behavior, or documentation maintenance.
2. Read only the smallest relevant reference set.
3. Build exact run commands and execute by default for automation requests.
4. If the operator explicitly asks to review commands first, show planned commands before execution.
5. For any new run request, re-check live controller running state before deciding whether the previous run is still active.
6. If running automation, verify generated specs and `specPattern` before runner launch.
7. Default to watcher-backed launch unless explicitly told not to.
8. After launch, report immediate execution outcome and the exact executed template/runner commands.
9. If reporting status, use live evidence and reporter artifacts before cached watcher state.
10. If editing docs, keep rules in guide content, reusable commands in examples content, dated lessons in run learnings, and durable environment data in inventory content.
skills/atvm/agents/openai.yaml
+7
View File
@@ -0,0 +1,7 @@
interface:
display_name: "ATVM"
short_description: "Portable ATVM operations workflow"
default_prompt: "Use $atvm to plan or operate an ATVM setup, automation run, watcher flow, or status report."
policy:
allow_implicit_invocation: true
skills/atvm/references/atvm-prep-reference.md
+118
View File
@@ -0,0 +1,118 @@
# ATVM Prep Script Reference
This document summarizes the behavior of the controller-local prep script:
- Host: `atvm-cypress-vm-1` (`192.168.3.190`)
- Path: `/root/atvm_prep/atvm_prep.py`
- Execution rule: run all `atvm_prep.py` commands on `192.168.3.190` (ATVM Cypress controller).
Use this as a quick runbook when ATVM datastore/lun prep is requested.
## Purpose
`atvm_prep.py` orchestrates DGS and VMware actions around a target DGS LUN name for ATVM test environments. It handles clone lifecycle cleanup, snapshot clone assignment, datastore mount/unmount operations, and optional VM register/unregister workflows.
## Primary Modes
The script supports these operation modes:
1. `-A` / `--automated_testing`
- Prepare environment for automated testing.
- Cleans up prior assigned clone/lun state, creates a new clone from snapshot, assigns it, and force-mounts unresolved VMFS.
2. `-M` / `--maintenance`
- Prepare for maintenance against the specified DGS LUN.
- Cleans clone state, reassigns original/gold volume, rescans storage, and mounts the datastore.
3. `-R` / `--register`
- Register VMs only for the LUN currently assigned to the target client.
- Scans VMX files on datastore and registers missing VMs.
4. `-U` / `--unregister`
- Power off and unregister VMs only for the LUN currently assigned to the target client.
5. `-rvgd` / `--rollback_vm_gold_disk`
- Roll back target volume to a selected snapshot, then assign/mount for use.
## Main Inputs
Notable arguments:
- `-n` / `--dgs_lun_name` (required): target DGS LUN base name
- `-s` / `--snapshot_name`: optional snapshot name; latest used when omitted
- `-c` / `--client_name`: defaults to `CDS1-ESX165`
- `-vi` / `--vcenter_ip`: defaults to `192.168.0.201`
- `-vu` / `--vcenter_username`
- `-vp` / `--vcenter_password`
- `-e` / `--esx_host_ip`: defaults to `192.168.1.165`
## Workspace Shorthand Mapping
For requests in this workspace, operator shorthand may be expanded as follows:
1. Datastore/LUN shorthand for `-n`
- `Gold` -> `AutomatedTest-VMBootImg-Gold`
- `Gold2` -> `AutomatedTest-VMBootImg-Gold-2`
- `ComputeMigration` (or `compute migration`) -> `AutomatedTest-VMBootImgComputeMigration-Gold`
2. Client/ESXi pairing (`-c` with `-e`)
- `CDS1-ESX165` <-> `192.168.1.165`
- `CDS1-ESX166` <-> `192.168.1.166`
3. Inference rules
- If only one side of the pair is supplied (`-c` or `-e`), auto-fill the other side from this mapping.
- If both are supplied but conflict, stop and confirm before execution.
4. Approval rule
- Always show the exact planned `atvm_prep.py` command and wait for explicit operator approval before execution.
## Operational Flow
High-level behavior across modes:
1. Resolve DGS client and volume/policy IDs.
2. Resolve currently assigned volume for the target client where relevant.
3. If cleanup is required:
- Find datastore from LUN GUID.
- Power off VMs on that datastore.
- Unregister VMs.
- Unmount datastore on ESXi.
- Unassign and optionally delete clone on DGS.
4. Execute mode-specific action:
- create clone and assign (`-A`)
- assign gold and mount (`-M`)
- register only (`-R`)
- power off/unregister only (`-U`)
- rollback and reassign (`-rvgd`)
## VMware Helpers In Script
Reusable functions implemented in `atvm_prep.py` include:
- datastore discovery from DGS LUN GUID
- force-mount unresolved VMFS volume
- datastore mount/unmount
- VM power-off by datastore membership
- VM unregister by datastore membership
- VMX discovery on datastore for registration
## Safety Traits
The script intentionally exits on ambiguous or unsafe states, including:
- snapshot not found or duplicate snapshot names
- missing client/volume/datastore
- multiple datastores matching same GUID
For some operations (`register`, `unregister`, `rollback`) it prompts for explicit confirmation (`[y/n]`) before acting.
## Known Caveats
1. Default credentials are hardcoded in script arguments.
- Treat these as legacy behavior and prefer supplying credentials explicitly in runtime usage.
2. `main()` contains this condition:
- `if not is_register or not is_unregister:`
- This is effectively true for almost all normal runs and likely intended as `and`.
- Do not change behavior blindly during operations; patch only with explicit operator approval.
3. Registration steps in `-A`, `-M`, and `-rvgd` paths are currently commented out.
- Datastore prep/mount occurs, but VM registration is not automatically performed in those paths unless script is modified.
## Logging
- Script log directory: `/root/atvm_prep/log`
- Main log file: `/root/atvm_prep/log/atvm_prep.log`
- Rotating log handler is enabled in script.
## Practical Usage Notes
- Prefer dry validation of target LUN/client/snapshot before destructive modes.
- Use `-U` for scoped VM power-off/unregister tied to LUN GUID mapping.
- Use `-R` after datastore mount when VM registration is required and safe.
skills/atvm/references/automation-examples.md
+96
View File
@@ -0,0 +1,96 @@
## Examples
### E2E: Pure iscsi+fc with specific VMs
```bash
python3 cmc-templates.py --template cmc-e2e --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --test_partition --integration_type pure --use_specified_plugin both --specify_vms atvm3-ubuntu18.04 atvm109-w2k12R2; \
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-e2e-pure-plugin
```
### E2E: Infinibox fc with specific VMs
```bash
python3 cmc-templates.py --template cmc-e2e --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --test_partition --integration_type infinibox --use_specified_plugin fc --specify_vms atvm51-redhat6.10 atvm110-w2k16; \
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-e2e-infinibox-plugin
```
### E2E: Regular cutover
```bash
python3 cmc-templates.py --template cmc-e2e --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --test_partition --integration_type pure --use_specified_plugin fc --specify_vms atvm93-oracle7.9 atvm111-w2k19 --regular_cutover; \
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-e2e-regular-cutover
```
### Reboot test
```bash
python3 cmc-templates.py --template cmc-reboot --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --integration_type pure --use_specified_plugin fc --specify_vms atvm37-rocky8.8 atvm112-w2k22 --wait_for_power_on 120; \
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-reboot
```
### SystemOS test
```bash
python3 cmc-templates.py --template cmc-systemOS --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --specify_vms atvm118-oracle9.3 atvm145-w2k25; \
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-systemOS
```
### MigrateOPS test
```bash
python3 cmc-templates.py --template cmc-migrateops --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --integration_type pure --use_specified_plugin fc --specify_vms atvm139-redhat9.5 atvm112-w2k22; \
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-migrateOPS
```
### Compute MigrateOPS: vmware
```bash
python3 cmc-templates.py --template cmc-migrateops-compute-migration --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --vm_platforms vmware --test_partition --specify_vms atvm138-oracle9.4-opt atvm112-w2k22 --set_static_ip_dest; \
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-computeMigrateOPS-vmware
```
Default vmware compute-migration options:
- `--ignore_force_shutdown`
- `--vm_platforms vmware`
- `--test_partition`
- `--set_static_ip_dest`
### Compute MigrateOPS: ovirt
```bash
python3 cmc-templates.py --template cmc-migrateops-compute-migration --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --vm_platforms ovirt --test_partition --specify_vms atvm124-redhat8.8 atvm111-w2k19 --set_static_ip_dest; \
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-computeMigrateOPS-ovirt
```
### Group consistency
```bash
python3 cmc-templates.py --template cmc-group-consistency --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --integration_type pure --use_specified_plugin fc --specify_vms atvm4-ubuntu20.04 atvm112-w2k22 --enable_uuid; \
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-consistentyGroup
```
### Group consistency: single VM
```bash
python3 cmc-templates.py --template cmc-group-consistency --ignore_force_shutdown --test_partition --config_file_path ./cypress.atvm-config-gold.ts --integration_type pure --use_specified_plugin fc --specify_vms atvm71-redhat9.1 --enable_uuid; \
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-consistencyGroup-redhat9.1 --categorize
```
### H2H same platform
```bash
python3 cmc-templates.py --template cmc-h2h-same-platf --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --integration_type pure --use_specified_plugin fc --specify_vms atvm38-rocky9.0 atvm112-w2k22; \
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-h2hSamePlatform
```
### H2H different platform
```bash
python3 cmc-templates.py --template cmc-h2h-diff-platf --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --integration_type pure --use_specified_plugin fc --specify_vms atvm65-redhat8.3 atvm112-w2k22; \
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-h2hDifferentPlatform
```
### Randomized reboot sanity
```bash
python3 cmc-templates.py --template cmc-reboot --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --test_partition --integration_type pure --use_specified_plugin fc --randomize 1 --exclude_partial_match suse15.0 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 --wait_for_power_on 120; \
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name sanity-reboot-iscsi
```
### Randomized e2e sanity
```bash
python3 cmc-templates.py --template cmc-e2e --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --test_partition --integration_type pure --use_specified_plugin both --randomize 1 --exclude_partial_match suse15.0 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0; \
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name sanity-e2e
```
### Randomized systemOS sanity
```bash
python3 cmc-templates.py --template cmc-systemOS --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --randomize 1 --exclude_partial_match suse15.0 fedora34 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0; \
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name sanity-systemOS
```
skills/atvm/references/automation-guide.md
+378
View File
@@ -0,0 +1,378 @@
# Run ATVM Automation Guide
This file is guide-only documentation for operating ATVM CMC automation.
Do not put specific run examples here.
For reusable command examples and common option combinations, use `examples.md`.
Treat `examples.md` as reference-only.
Do not assume the operator wants the extra options shown in examples unless they explicitly request them.
## Purpose
Run ATVM CMC automation tests on the designated automation VM without unintended system or file changes.
## ATVM Cypress Automation Controller Client
- Hostname: `atvm-cypres-vm-1`
- IP: `192.168.3.190`
- Credentials: source `/home/aw/code/cds/.env.credentials.local` and use `ATVM_CONTROLLER_USER` plus `ATVM_CONTROLLER_PASSWORD`
## ATVM Target Host Default
- Treat `192.168.3.191` as the default ATVM target host reference.
- For SSH to `192.168.3.191`, ignore host key mismatch by default with `-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null`.
- For Linux SSH access to `192.168.3.191`, source `/home/aw/code/cds/.env.credentials.local` and use `ATVM_TARGET_USER` plus `ATVM_TARGET_PASSWORD` unless the operator explicitly overrides them.
- `ATVM_LINUX_TARGET_HOST`, `ATVM_LINUX_TARGET_USER`, and `ATVM_LINUX_TARGET_PASSWORD` mirror the Linux default values when an OS-specific reference is clearer.
- For Windows guest access to `192.168.3.191`, source `/home/aw/code/cds/.env.credentials.local` and use `ATVM_WINDOWS_TARGET_USER` plus `ATVM_WINDOWS_TARGET_PASSWORD` unless the operator explicitly overrides them.
## Operating Constraints
- Run only scripts/commands explicitly requested.
- Do not make manual system configuration changes on the client.
- Do not edit client files unless explicitly requested.
## Operator Preferences
- Do not include Gold Disk identifiers in `--build_name`.
- `--build_name` must not contain spaces; use `-` between words.
- For multiple VMs in same distro, use distro-scoped filtering (`--containsVm`) instead of long explicit VM lists.
- Always include `--ignore_force_shutdown` on `cmc-templates.py` commands unless the operator explicitly asks not to.
- Always include `--test_partition` on `cmc-templates.py` commands unless the operator explicitly asks not to.
- Default plugin-bearing templates to `--use_specified_plugin iscsi` unless the operator explicitly requests a different plugin.
- Do not add plugin or integration-type arguments to `cmc-systemOS`; that template should be planned without `--use_specified_plugin`, without `--integration_type`, and without watcher integration/plugin metadata.
- For `cmc-migrateops-compute-migration`, default to `--set_static_ip_dest` unless the operator explicitly says otherwise.
- For `cmc-migrateops-compute-migration` to VMware, default to `--vm_platforms vmware` unless the operator explicitly says otherwise.
- For ATVM automation runs that involve Windows guests, default `run-sorry-cypress.py` to `--hang_retries 0` unless the operator explicitly says otherwise.
- For `cmc-reboot`, treat `--use_specified_plugin both` as an exception case that requires an extra confirmation.
- When `cmc-reboot` is planned with `--use_specified_plugin both`, warn that FC+iSCSI together may hit a "chicken before the egg" timing problem where iSCSI disks are not attached before mTDI / CMC services start.
- For `cmc-reboot`, prefer `--use_specified_plugin fc` or `--use_specified_plugin iscsi` unless the operator explicitly reconfirms that `both` is really intended after seeing that warning.
- Before preparing a new run, always check whether automation is already running.
- Treat a prior status check as stale once control returns to the operator or a new ATVM request arrives; perform a fresh live controller check at request time instead of relying on the immediately previous result.
- Always report whether automation is currently running.
- If running, ask whether to terminate; terminate only with explicit approval.
- After termination approval, terminate first, then execute the new run command set.
- By default, execute `cmc-templates.py` and `run-sorry-cypress.py` without a pre-run approval gate.
- If the operator explicitly asks to review planned commands first, show them before execution.
- If the operator changes any part of the request before execution, rebuild commands and execute the revised command set.
- Default to watcher-backed execution for every run unless the operator explicitly asks to run without watcher.
- When `--categorize` is used with watcher enabled, treat the watcher as a sequential grouped-run watcher:
- it must post one final Mattermost status per completed categorized group/sub-run
- it must stay active between grouped sub-runs while the parent categorized request is still running
- it must not stop after the first grouped run simply because one grouped run completed
- if the child build id label does not match the actual host/spec being executed, report the grouped run using the inferred host-based group instead of the raw child build id label
- it must not wait and replace those with one single parent-only post
- After execution, report immediate success/failure only.
- After execution, include the exact executed `cmc-templates.py` and `run-sorry-cypress.py` commands in the response.
- Do not include expected, harmless `systemctl reset-failed ... unit not loaded` output in routine run-start confirmations.
- Mention `reset-failed` output only when it prevents watcher startup or becomes relevant to debugging.
- Do not actively monitor completion unless explicitly requested.
- If monitoring is requested, allow long runtime windows (15-30+ minutes) and continue until completion unless operator instructs otherwise.
- Report command errors immediately.
- `sshpass` may be used where password-based SSH automation is required.
- Treat runner hang-kill events (`Sending SIGKILL ... due to no change` / `Max hang retries reached`) as explicit `FAILED` outcomes, not `RUNNING` or ambiguous termination.
- For manual `run-sorry-cypress.py` execution, treat `ATVM_HANG_FAIL ...` log markers and `/tmp/atvm-runner-state-<build>.json` terminal state files as the source of truth for hang-failure terminal status.
## Core Scripts
- Template prep: `/root/cdc-e2e-cyp-12.17.4/cmc-templates.py`
- Test execution: `./run-sorry-cypress.py`
- Detailed host-level test artifacts: `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter`
## Detailed Test Artifacts
- Use `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter` on the automation controller for detailed per-host test evidence.
- Reporter subdirectories of interest:
- `logs/`
- per-host text and JSON logs for the executed tests
- `xml/`
- machine result XML files and the final `check-xml-files.ts` bookkeeping output
- `mochawesome/`
- per-run HTML reports
- When a machine fails, use the matching `logs/` entry first to capture the detailed failure context for that host.
- Apply the failed-host detail recovery path to every ATVM template type, not just reboot.
- For any failed host, recover detail in this order when available:
- consolidated run log
- matching `mochawesome` HTML
- structured reporter artifacts such as per-host JSON or XML
- text reporter artifacts
- When reconstructing historical status, prefer `cmcReporter` artifacts over less-specific runner output because they preserve per-host results after the live run has ended.
- Do not treat the existence of a per-host reporter artifact by itself as proof that the host passed.
- For categorized grouped recovery, prefer the matching per-host reporter JSON or mochawesome result and carry through the real `failures`, `pending`, and failure message instead of assuming `PASS completed`.
- If grouped XML only contains `check-xml-files.ts`, cross-check the grouped result against the per-host reporter artifacts before posting or repeating status for that grouped sub-run.
- Do not report a categorized grouped sub-run as `PASS` from watcher `host_results`, grouped XML, or a lone `check-xml-files.ts` result by itself.
- Before reporting a categorized grouped sub-run as `PASS`, confirm that the matching child batch also passed in the live launch log or the final `Cloud Run Finished` summary for that child run.
- Treat saved watcher state under `/var/lib/atvm-run-watcher/<build>/state.json` as cached status only.
- For completed-run verification, confirm in this order:
- launch log under `/tmp/<build>.launch.log`
- matching `cmcReporter` artifacts
- `Cloud Run Finished` summary and Currents URL
- saved watcher state only as a comparison layer
- If saved watcher state disagrees with the launch log or with a replay of the exact artifacts through the current watcher code, treat the saved state as stale and do not use it as the reported result.
- Never confirm a completed run from `state.json` alone.
Typical sequence:
1. Build the exact `cmc-templates.py` and `run-sorry-cypress.py` commands for the request.
2. Run `cmc-templates.py` with the requested options.
3. Wait for `cmc-templates.py` to fully finish and confirm success.
4. Verify the generated `.ts` files and the config `specPattern` include every requested VM before starting the runner.
5. By default, use watcher-backed execution unless the operator explicitly asks not to.
6. For watcher-backed runs, make sure the controller's deployed watcher code is the intended version before relying on its posts.
7. For watcher-backed runs, build the watcher-start command so it automatically includes the exact `cmc-templates.py` command via `--template-command` and the exact `run-sorry-cypress.py` command via `--runner-command`.
8. For watcher-backed runs, prefer the controller-local `atvm-runner@...` systemd service instead of detached SSH background launch patterns for `run-sorry-cypress.py`.
9. For watcher-backed runs, start the watcher before launching the runner service.
10. Start the runner with the matching config and build name.
11. Report immediate start success/failure and include the exact executed template and runner commands.
Completed-run verification sequence:
1. Read the launch log for the build.
2. Inspect the matching reporter artifacts for the relevant host(s).
3. Use the `Cloud Run Finished` summary and Currents URL as the final parent-run check when present.
4. Compare that result against saved watcher state.
5. If there is any disagreement, replay the exact artifacts through the current watcher code in an isolated temp state directory before confirming the result.
6. For categorized runs, do not let a `check-xml-files.ts` child result override a failing child batch shown in the launch log or `Cloud Run Finished` summary.
## Config File / Gold Disk Mapping
- `cypress.atvm-config-gold.ts` -> Gold Disk 1
- `cypress.atvm-config-gold-2.ts` -> Gold Disk 2
- Additional numbered config variants map to corresponding Gold Disks.
- Do not default to `cypress.atvm-config.ts`.
- Unless the operator explicitly requests another config, use a config file with `gold` in the filename.
- If the operator-specified config file is missing, stop immediately and report the missing file.
- Do not search for substitute ATVM config files and do not switch to another config unless the operator explicitly instructs it.
- Treat `AutomatedTest-VMBootImg-Gold` as `gold` and `AutomatedTest-VMBootImg-Gold-2` as `gold-2`.
- Use live vCenter inventory as the source of truth for current VM membership on those datastores.
- Query vCenter datastore membership at request time when selecting `gold` vs `gold-2`; do not maintain or rely on a repo-side live reference file for this decision.
- When the operator asks to inventory or show the contents of `AutomatedTest-VMBootImg-Gold` and `AutomatedTest-VMBootImg-Gold-2`, return hostname-only VM lists unless the operator explicitly asks for more detail.
- When the operator provides an explicit VM list, check vCenter placement for every requested VM before choosing the config file.
- Before presenting any ATVM run commands for an explicit VM-list request, tell the operator that the next step is a live vCenter placement check for the requested VMs and that the result will determine whether the run must use `gold` or `gold-2`.
- For vCenter inspection and placement checks, prefer `govc` and raw vCenter REST calls when they are available before reaching for alternate wrappers.
- Never delete, destroy, remove from inventory, or unregister any ATVM VM from vCenter.
- Treat a vCenter VM as protected ATVM inventory if its name starts with `atvm`, if it resides on an `AutomatedTest-*` ATVM datastore, or if it is an ATVM platform/controller VM, even when it is not listed in `inventory/vm-inventory.md`.
- Preserve ATVM vCenter VM inventory records even when power, maintenance, datastore, cleanup, or prep work is requested.
- For cleanup workflows, only power off, detach/unmount supporting resources, or report the needed manual action; do not remove ATVM VM inventory records.
- For `govc`-based placement checks, use `govc vm.info -json <vm>` and parse the lowercase JSON keys such as `virtualMachines` and `datastore`.
- Resolve each returned datastore managed-object reference to a datastore name with `govc object.collect -s <datastore-ref> name` before deciding between `gold` and `gold-2`.
- Ignore non-boot helper datastores such as install ISO attachments when applying the `gold` vs `gold-2` rule; base the family decision on the ATVM boot datastore membership.
- If every requested VM is on `AutomatedTest-VMBootImg-Gold`, plan the run with the `gold` config.
- If every requested VM is on `AutomatedTest-VMBootImg-Gold-2`, plan the run with the `gold-2` config.
- If the requested VM set spans both `AutomatedTest-VMBootImg-Gold` and `AutomatedTest-VMBootImg-Gold-2`, stop immediately and do not prepare or run the test.
- For a mixed-datastore request, report it as a discrepancy, list which requested VMs are on `AutomatedTest-VMBootImg-Gold` and which are on `AutomatedTest-VMBootImg-Gold-2`, tell the operator they need to correct the list, and ask whether they want the full VM inventories for both datastores so they can adjust the request.
- Do not run an ATVM test against a mixed set of VMs from both `AutomatedTest-VMBootImg-Gold` and `AutomatedTest-VMBootImg-Gold-2`.
## Available Templates
- `cmc-e2e`
- `cmc-group-consistency`
- `cmc-h2h-diff-platf`
- `cmc-h2h-same-platf`
- `cmc-migrateops`
- `cmc-migrateops-compute-migration`
- `cmc-reboot`
- `cmc-systemOS`
## Command Pattern
```bash
python3 cmc-templates.py --template <template> --ignore_force_shutdown --test_partition --config_file_path ./<config-file> [template-specific plugin/integration options when that template actually uses them...]; \
python3 ./run-sorry-cypress.py --config_file <config-file> --build_name <hyphenated-description-no-spaces> [--categorize]
```
## Examples Reference
- Commonly used command examples: `examples.md`
- Keep this guide focused on run-control rules and workflow constraints.
- Use examples as reference material only, not as default intent for new operator requests.
- Keep `examples.md` limited to reusable example commands; keep workflow rules, defaults, blacklist policy, and reporting rules in this guide or `run-learnings.md`.
- When validating plugin-specific ATVM requests, do not treat the presence of plugin-specific code blocks or step labels in the generated `.ts` file as proof that those steps will execute.
- Determine plugin behavior from the template/runtime gating logic, for example `Cypress.env(...)`-driven conditionals, and only call it a mismatch when that runtime logic would execute the wrong plugin path.
## Example Option Patterns (Guide-Only)
- Distro-scoped VM selection:
- `--containsVm redhat`
- `--containsVm redhat9`
- Explicit VM selection:
- `--specify_vms <vm1> <vm2> ...`
- Compute migrateops platform:
- `--vm_platforms vmware|ovirt|openshift|proxmox`
- Default compute migrateops options:
- `--ignore_force_shutdown`
- `--test_partition`
- `--set_static_ip_dest`
- Default compute migrateops vmware options:
- `--ignore_force_shutdown --vm_platforms vmware --test_partition --set_static_ip_dest`
## Blacklisted Machines
Always exclude these machines from broad-scope ATVM automation runs by adding them to `--exclude_partial_match`.
If the operator explicitly targets one or more named VMs with `--specify_vms`, do not add the maintained `--exclude_partial_match` list unless the operator also explicitly asks for it.
Even for explicit `--specify_vms` requests, first check whether any requested VM is on the maintained blacklist and stop instead of launching the run if one is included.
Permanently blacklisted because CMC cannot compile:
- `atvm6-centos6.0`
- `atvm41-redhat6.0`
- `atvm73-oracle6.0`
Temporarily blacklisted because the run crashes when creating a migration session:
- `atvm144-suse15.0`
Temporarily blacklisted while support requests are waiting:
- `atvm113-debian9.0.0`
- `atvm115-debian9.1.0`
- `atvm116-debian9.2.0`
Temporarily blacklisted because re-creation might be needed:
- `atvm156-debian9.3.0`
Preferred exclude list:
- `--exclude_partial_match atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0`
## Running-Automation Check (Mandatory)
Before any new automation request:
1. SSH to `root@192.168.3.190`.
2. Check for active automation processes (for example `run-sorry-cypress.py`, `cmc-templates.py`, and related Cypress runners).
3. Treat every new operator request to start, replace, or block on an ATVM run as requiring a new live controller check, even if a status check was performed earlier in the same conversation.
4. Do not reuse the result of the immediately previous running-state check when deciding whether to block or allow a new ATVM run request.
5. Report:
- `Running` with process details, or
- `Not running`.
6. If `Running`, ask operator whether to terminate.
7. If termination is approved, terminate matching process(es), confirm termination, then proceed to planned-command approval.
8. If termination is not approved, do not start a new run.
## Execution Workflow (Mandatory)
1. Build exact command(s) for the request.
2. By default, execute without a pre-run approval gate. If the operator explicitly asks for command review first, show planned commands before running.
3. When both template generation and the Cypress runner are requested, run them sequentially, not in parallel.
4. Do not launch the ATVM runner until `cmc-templates.py` has exited successfully and finished updating the intended config/spec files.
5. After `cmc-templates.py`, always verify that the generated spec files on disk and the config `specPattern` both contain the full requested VM set before launching the ATVM runner.
6. If any requested VM is missing from the generated files or `specPattern`, stop and report the mismatch instead of launching the runner.
7. If the operator asks to change plugin, config, filters, build name, Gold Disk, or scope before execution, discard the old plan and execute only the revised command set.
8. If the planned command is `cmc-reboot` with `--use_specified_plugin both`, add the FC+iSCSI timing warning and require explicit confirmation that `both` is intended before execution.
9. Default to watcher-backed execution unless explicitly told not to.
10. When watcher-backed execution is used, the watcher-start command must automatically include the exact executed `cmc-templates.py` command via `--template-command` and the exact executed `run-sorry-cypress.py` command via `--runner-command`.
11. If the run uses `--categorize` and watcher is enabled, include `--categorize` on the watcher start command so the watcher tracks sequential categorized sub-runs correctly.
12. When watcher-backed execution is used, launch the watcher before the runner service.
13. Do not start the runner before the watcher, because the watcher helper clears stale `/tmp/<build-name>.log` and can delete the fresh live runner log if the runner starts first.
14. Prefer the combined `start-atvm-run.sh` wrapper when both services are used, so watcher and runner are not launched in parallel.
15. If monitoring was not requested, report immediate success/failure for each command.
16. If monitoring was requested, keep monitoring until completion and report final outcome.
17. After execution, always include the exact executed `cmc-templates.py` and `run-sorry-cypress.py` commands in the response.
## Requested Test Style
When asked for one VM or a VM set:
- choose requested template/options,
- if the request includes explicit VM names, determine each VM's datastore placement in vCenter before selecting the config file,
- before showing run commands, tell the operator you are about to do the live vCenter placement check and that it will decide the config family,
- use only live vCenter datastore membership when resolving that placement,
- choose correct config file for intended Gold Disk,
- if the explicit VM set is split between `AutomatedTest-VMBootImg-Gold` and `AutomatedTest-VMBootImg-Gold-2`, interrupt the request and report the per-datastore split instead of planning commands,
- if the operator asks to see the datastore inventories, show hostname-only lists by default,
- default to a config filename containing `gold` unless the operator explicitly says otherwise,
- always include `--ignore_force_shutdown` on the template-generation command unless the operator explicitly overrides that default,
- always include `--test_partition` on the template-generation command unless the operator explicitly overrides that default,
- default plugin-bearing templates to `--use_specified_plugin iscsi` unless the operator explicitly requests another plugin or the template does not use plugin selection,
- if the operator explicitly requests `cmc-group-consistency` with one VM, allow the single-VM request and plan it without asking for additional VMs,
- treat `cmc-systemOS` as a template that does not use plugin selection or integration-type selection, so omit `--use_specified_plugin`, omit `--integration_type`, and omit watcher integration/plugin metadata for that template,
- use a descriptive `--build_name` without Gold Disk IDs.
## Update Rule
- After each run, update this guide only for workflow/rule/default changes.
- Update `examples.md` for reusable command/option examples.
- Add run-specific learnings only to `run-learnings.md` when the run produced new information.
## Monitoring Policy
- Monitor only when the operator explicitly asks to monitor.
- If monitoring was not requested, run commands and report execution success/failure and any errors.
- If monitoring was requested, do not terminate processes automatically; only terminate if the operator explicitly instructs termination.
## Mattermost Status Posting
- Treat a normal ATVM status request as local-only output by default.
- When the operator asks to send ATVM automation run status to Mattermost, use the local defaults from `/home/aw/code/cds/.env.credentials.local`.
- Default Mattermost variables:
- `MATTERMOST_ATVM_WEBHOOK`
- `MATTERMOST_ATVM_CHANNEL`
- Treat these as the default destination for ATVM automation run-status posts unless the operator explicitly overrides them.
- Send the final ATVM run status only after the run has fully completed, regardless of whether the run passed or failed.
- Do not send interim or in-progress ATVM run status updates to Mattermost unless the operator explicitly asks for that.
- Use the same ATVM status layout that would be shown to the operator locally when posting to Mattermost.
- Default status template: `/home/aw/code/cds/atvm/docs/automation/status-template.md`
- Do not post to Mattermost unless the operator explicitly asks for the run status to be sent there.
- For categorized execution with watcher enabled, send one Mattermost status per completed categorized sub-run/group after that grouped run fully finishes.
## Status Reporting Format
When the operator asks for the status of an ATVM automation run, report in this order:
1. Heading/title using the run `build_name`.
2. `SUMMARY:` section with finished, passed, failed, and skipped counts.
3. `HOSTS:` section with the machine rows.
4. `TIMING:` section with start, end, total, quickest, longest, and average.
5. `COVERAGE:` section describing what the run was intended to cover, excluding the target-host list.
6. `TEST FLOW:` section describing the template-specific numbered run flow for the test.
7. `FAILURE NOTES:` section for detailed per-host error excerpts when failures exist.
8. `NOTES:` section for broader non-failure context and anomalies.
7. Remaining machines still to run.
8. Summary counts for finished, passed, failed, and skipped machines.
9. Timing details:
- start time
- end time if complete
- total run time if complete, or elapsed run time if still running
- quickest completed test runtime
- longest completed test runtime
- average completed test runtime
10. Estimated completion time.
Status-report expectations:
- Use the same display layout for every ATVM automation status response regardless of test type (`e2e`, `systemOS`, `reboot`, `migrateops`, and others).
- Use `/home/aw/code/cds/atvm/docs/automation/status-template.md` as the default template for both local status output and Mattermost status posts.
- The default ATVM status template uses flat bullet-list sections for `COVERAGE:`, `TEST FLOW:`, `FAILURE NOTES:`, and `NOTES:`, and Markdown tables for `SUMMARY:`, `HOSTS:`, and `TIMING:`.
- Order the status sections as `SUMMARY:`, `HOSTS:`, `TIMING:`, `COVERAGE:`, `TEST FLOW:`, `FAILURE NOTES:`, then `NOTES:`.
- Keep `NOTES:` focused on non-failure operator-facing value such as the Currents run URL, real anomalies unrelated to the direct failure text, or material fallback behavior.
- Include the exact `cmc-templates.py` command used to trigger the ATVM automation run in `NOTES:`, without the outer `sshpass`/`ssh` wrapper and without trimming it.
- Include the exact `run-sorry-cypress.py` command used to launch the ATVM automation run in `NOTES:`, without the outer `sshpass`/`ssh` wrapper and without trimming it.
- Do not include generic watcher bookkeeping messages in `NOTES:` such as artifact-detection confirmations.
- Do not include internal watcher fallback notes in `NOTES:` such as `check-xml-files.ts` validation confirmations or reporter-artifact recovery details.
- The `HOSTS:` table includes `Host`, `Kernel`, `Status`, and `Detail` columns in that order.
- Do not show total test/failure counts in the `Detail` column. Keep those counts in watcher state and summary logic only.
- For passed hosts, render `Detail` as `completed`.
- For any failed host, keep the `Detail` column compact by showing the failing step plus a short error summary, not the full raw stack trace.
- If richer failure text is available, put the longer trimmed excerpt in `FAILURE NOTES:` so the result stays readable in Mattermost and local status output.
- In `COVERAGE:`, describe the important `cmc-templates.py` command inputs such as template, categorize mode, datastore/config family, config filename, migration style, any real plugin/integration path, and other operator-relevant run options, but do not list target hosts there or include verbose prose scope descriptions.
- Only include coverage fields that the template command actually used. Do not show empty or irrelevant fields such as an integration/plugin path for templates that did not use one.
- If `categorize mode: enabled` is already shown in `COVERAGE:`, do not also repeat `--categorize` under `run options`.
- When grouped categorized timing is reconstructed from host reporter artifacts, derive per-host quickest/longest/average durations from the sequence of recovered host timestamps and the grouped end time instead of leaving those metrics as `n/a`.
- In `TEST FLOW:`, show the template-specific numbered run flow once for the whole test, not per host.
- For `TEST FLOW:`, treat the generated host spec from the actual run as the source of truth whenever it exists.
- Extract the numbered flow steps from the generated `.ts` spec referenced by that run's `specPattern`.
- When the generated spec contains runtime-gated plugin branches such as `if(useFCPlugin)`, `if(useIscsiPlugin)`, `if(usePureFCPlugin)`, or `if(usePureIscsiPlugin)`, only include the steps for the plugin path actually selected for that run.
- Do not prefer a static template flow list over a generated spec from the actual run.
- Use template-level or static fallback flow only when the generated spec cannot be found or parsed.
- If fallback is required, resolve it from the run template name before using any generic default flow.
- `cmc-e2e` currently uses the 22-step migration flow documented in `/home/aw/code/cds/atvm/docs/automation/status-template.md`.
- `cmc-systemOS` currently uses the 21-step boot-disk migration flow documented in `/home/aw/code/cds/atvm/docs/automation/status-template.md`.
- Keep `FAILURE NOTES:` and `NOTES:` behavior consistent across template types; do not add template-specific internal-source notes such as parent-log-summary recovery details.
- For the `Kernel` column, cross-reference the host name against `/home/aw/code/cds/atvm/inventory/vm-inventory.md`.
- If the hostname is not present in `vm-inventory.md`, report the kernel value as `unknown`.
- Treat references to the "ATVM automation run" or "automation run" as referring to this ATVM folder workflow and the automation VM at `192.168.3.190`, not to Cirrus project operations such as the `atvm - cypress` project.
- Treat a status request as a request for live status by default.
- Unless the operator explicitly asks to send the status to Mattermost, print the status only in the local terminal response.
- Use the live automation VM state when available.
- If no automation is currently running, fall back to the most recent historical run artifacts and logs.
- Prefer local automation evidence in this order: active runner processes, live automation-VM files, shell history for the last launch command, then historical reporter artifacts.
- For detailed machine-level failure information, use `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter/logs/` on the automation VM.
- Derive the heading/title from the run `build_name` when available.
- Format every machine entry as `machine-name - STATUS`.
- Put each machine on its own line; never combine multiple machines into one paragraph or comma-separated line.
- Use a separate `Notes` section for failure reasons, anomalies, or operator-relevant context rather than cramming those details into the completed-machine list.
- For categorized runs, reconstruct the whole run across all category batches; do not treat the current live category batch as the full run scope.
- For categorized runs with no active automation, reconstruct the status from the full historical run across all category batches, not only the most recent category batch.
- Always report the status of the entire requested run, even when the runner split execution into multiple category batches or cloud sub-runs.
- Derive completed-machine status from completed spec results already written during the same run.
- Parse all same-run `test-result-*.xml` files, not only machine-named `test-result-atvm*.xml` files.
- When XML filenames are hash-named, extract the machine name from XML contents such as `testsuite file=`, `testsuite name=`, or `testcase name=`.
- Ignore `check-xml-files.ts` XML outputs when counting machine completion because they are bookkeeping steps, not machine runs.
- When multiple same-run XML files exist for one machine, use the most recently written XML for that machine.
- Include the run start time in every status response when it can be derived from the run log.
- If the run is complete, include the end time and total run time.
- If the run is still active, include the elapsed run time so far.
- Include quickest completed test runtime, longest completed test runtime, and average completed test runtime under timing details when they can be derived from the run log.
- Show blacklisted machines under skipped machines even if they are part of the broader machine family requested by the operator.
- For skipped machines, include the reason category:
- `BLACKLISTED: CMC INSTALL - CAN'T COMPILE`
- `BLACKLISTED: SUPPORT REQUEST - WAITING`
- `BLACKLISTED: RE-CREATE MIGHT BE NEEDED`
- `BLACKLISTED: RE-CREATE NEEDED`
- If a machine is currently in progress, show it under remaining machines as `RUNNING`.
- If a machine has not started yet, show it under remaining machines as `NOT STARTED`.
- If no failures are present in completed spec results, report those completed machines as `PASS`.
- If a completed spec result shows a failure, report that machine as `FAIL` in the completed list and append a longer same-line failure description when the extra detail is useful to the operator.
- Use `Notes` for extra context beyond the machine-specific same-line failure description.
- Base the completion estimate on the full remaining machine count and recent per-machine runtime visible in the run log.
- Make the estimate explicitly refer to completion of the entire remaining run, not only the current machine/spec.
- When the operator also asks to send the status to Mattermost, send this same final status output to the configured Mattermost destination only after the run has fully completed.
skills/atvm/references/automation-run-learnings.md
+676
View File
@@ -0,0 +1,676 @@
# Run ATVM Automation Runs
This file stores run-specific examples only when a run produced a new learning relevant to future automation tasks.
## Entry Rule
- Add an entry only when a run changed workflow behavior, exposed a failure mode, or confirmed a required new check.
- Do not add routine runs with no new learning.
## Current State
- No run-learning entries recorded yet from `guide.md` source material.
## Run Learning: 2026-05-06 (ATVM runs now execute by default with watcher and report executed commands after start)
- Observed requirement:
- The operator does not want a pre-run approval/review gate for standard ATVM automation run requests.
- The operator wants watcher-backed execution to be the default unless explicitly overridden.
- The operator wants the exact executed template and runner commands reported after execution starts.
- Action for future runs:
- Execute ATVM run requests by default without waiting for explicit `approve`.
- Default to watcher-backed launch using the combined `start-atvm-run.sh` wrapper unless the operator explicitly asks to run without watcher.
- After command execution, report the exact executed `cmc-templates.py` and `run-sorry-cypress.py` command strings in the response.
## Run Learning: 2026-05-07 (Default ATVM compute migrateops runs to set destination static IP)
- Observed requirement:
- ATVM compute migrateops runs should default to enabling destination static IP handling to avoid status-check/parser mismatches and keep destination addressing consistent post-cutover.
- Action for future runs:
- For `cmc-migrateops-compute-migration`, include `--set_static_ip_dest` by default unless the operator explicitly asks not to.
- Keep platform selection (`--vm_platforms ...`) based on the operator request while retaining this static-IP default.
## Run Learning: 2026-05-07 (Default Windows-involved ATVM runs to hang retries 0)
- Observed requirement:
- When an ATVM automation run involves Windows guests, runner retry loops should be disabled by default to avoid long post-failure wait cycles.
- Action for future runs:
- For Windows-involved ATVM automation runs, add `--hang_retries 0` to `run-sorry-cypress.py` by default unless the operator explicitly requests a different value.
- Keep this as an operator-default behavior even though the underlying runner option is generic and not Windows-only in code.
## Run Learning: 2026-05-07 (Treat hang-kill as explicit failure)
- Observed failure mode:
- A run can stall long enough for `run-sorry-cypress.py` to force-kill Cypress (`Sending SIGKILL ... due to no change`) and still be reported as an ambiguous terminated state.
- Action for future runs:
- When run logs contain hang-kill markers (`Sending SIGKILL ... due to no change` and `Max hang retries reached.`), classify the run as `FAILED`.
- When the runner service exits non-zero, classify the run as `FAILED` instead of generic terminated.
## Run Learning: 2026-05-07 (Manual runner emits explicit hang-fail markers and terminal state)
- Observed failure mode:
- Manual `run-sorry-cypress.py` execution can appear "still running" after hang-kill handling because failure state was not emitted in a machine-readable terminal marker.
- Action for future runs:
- `run-sorry-cypress.py` now emits `ATVM_HANG_FAIL ...` on hang-kill paths and writes terminal state JSON under `/tmp/atvm-runner-state-<build>.json`.
- Max hang-retry exhaustion now writes terminal failure state before exiting non-zero, including categorized and non-categorized flows.
## Run Learning: 2026-05-02 (Do not reuse the previous controller status check for a new ATVM request)
- Observed failure mode:
- A later ATVM run request was blocked because the assistant reused the immediately previous controller status result instead of performing a fresh live running-state check at request time.
- The earlier check had been correct when it was taken, but the prior run had already failed and exited by the time the new request arrived.
- Action for future runs:
- Treat each new ATVM request to start, replace, or block on a run as requiring a fresh live controller check on `192.168.3.190`.
- Do not decide whether to block or allow a new run from a previously reported controller state, even within the same conversation.
- Re-check live runner and watcher state before telling the operator that automation is still running.
## Run Learning: 2026-04-29 (Combined watcher wrapper must execute template generation before runner startup)
- Observed failure mode:
- A watcher-backed `start-atvm-run.sh` launch for `cmc-migrateops-compute-migration` started `run-sorry-cypress.py` without ever running the approved `cmc-templates.py` command.
- The wrapper passed `--template-command` into watcher metadata only, so the runner consumed stale controller config state and started against a previous `specPattern` pointing at `atvm121-ubuntu24.04`.
- Action for future runs:
- The combined watcher wrapper must execute `--template-command` synchronously before watcher and runner startup.
- Write the template phase output to `/tmp/<build>.launch.log` so template activity is preserved separately from the live runner log.
- If the template step fails, stop immediately and do not start the watcher or the runner.
## Run Learning: 2026-04-29 (Watcher host-artifact parser must handle dict-shaped reporter events)
- Observed failure mode:
- A non-categorized ATVM compute-migration run failed in the host reporter artifacts, but the watcher posted `PASS`.
- The watcher fell back to the per-host JSON artifact after `check-xml-files.ts`, but `extract_failure_from_reporter_events()` only recognized the older list-shaped event format.
- Current reporter JSON stores events as dicts with fields such as `type`, `message`, and `severity`, so the parser missed `severity: error` and incorrectly returned `0 failures`.
- Action for future runs:
- Treat both list-shaped and dict-shaped reporter event records as valid inputs when extracting failure details from host JSON artifacts.
- Continue treating host reporter artifacts as authoritative fallback evidence when final XML only contains `check-xml-files.ts`.
## Run Learning: 2026-04-24 (Categorized watcher false-PASS guardrail)
- Observed failure mode:
- A categorized compute-migration run was incorrectly reported as `PASS` for `atvm121-ubuntu24.04` even though the actual Ubuntu grouped sub-run failed.
- The false `PASS` came from cached watcher `host_results` plus a grouped XML that only contained `check-xml-files.ts` with `failures="0"`.
- The authoritative launch log and `Cloud Run Finished` summary for that same child run showed `1 failing`.
- Action for future runs:
- Do not report a categorized grouped sub-run as `PASS` from watcher state, grouped XML, or `check-xml-files.ts` alone.
- Before reporting a categorized grouped sub-run as `PASS`, confirm that the matching child batch also passed in the live launch log or the final `Cloud Run Finished` summary for that child run.
- If watcher state or grouped XML disagrees with the launch log or child-run summary, treat the cached/grouped result as stale and report from the launch log plus per-host artifacts instead.
## Run Learning: 2026-03-08 (E2E redhat9.7, pure/fc)
- Request:
- template: `cmc-e2e`
- filter: `--containsVm redhat9.7`
- integration: `--integration_type pure`
- plugin: `--use_specified_plugin fc`
- Observed result:
- Cypress spec execution passed (`1` test, `1` passing, `0` failing).
- Cloud run URL was produced and marked uploaded.
- `run-sorry-cypress.py` remained running afterward with a defunct `npm exec cypress-cloud` child process and did not exit cleanly on its own.
- Action for future runs:
- If pass/upload is confirmed but `run-sorry-cypress.py` does not exit, treat it as a runner hang condition.
- Capture run URL and pass/fail status first, then terminate the stuck runner process cleanly.
## Run Learning: 2026-03-09 (Blacklist handling and status format)
- Observed requirement:
- Some ATVM machines must be skipped even when a broad selector such as `--containsVm` or `--randomize` would otherwise include them.
- Machines to blacklist via `--exclude_partial_match`:
- `BLACKLISTED: CMC INSTALL - CAN'T COMPILE`:
- `atvm6-centos6.0`
- `atvm41-redhat6.0`
- `atvm73-oracle6.0`
- `BLACKLISTED: SUPPORT REQUEST - WAITING`:
- `atvm113-debian9.0.0`
- `atvm115-debian9.1.0`
- `atvm116-debian9.2.0`
- `BLACKLISTED: RE-CREATE MIGHT BE NEEDED`:
- `atvm156-debian9.3.0`
- Action for future runs:
- Add these machine names to `--exclude_partial_match` when building broad-scope automation commands.
- When reporting run status, include skipped blacklisted machines separately with their reason, in addition to completed and remaining machines.
- Use the run `build_name` as the heading/title for status responses so the test type is obvious.
- For failed machines in status responses, include the failure reason taken from the run log.
- Include timing details in status responses: start time, end time when complete, and total or elapsed runtime.
- Also include timing stats in status responses: quickest completed test runtime, longest completed test runtime, and average completed test runtime.
## Run Learning: 2026-03-11 (Machine-first status lines and whole-run ETA)
- Observed requirement:
- Status output must list each machine first and then its status, rather than leading with the status label.
- Estimated completion time must refer to the entire remaining automation run, not only the currently running machine.
- Action for future runs:
- Format machine entries as `machine-name - STATUS`.
- Keep failure reasons after the machine/status entry when a machine failed.
- When giving ETA, explicitly state it is the estimate for completion of the full remaining run.
## Run Learning: 2026-03-11 (Categorized run status must be reconstructed across batches)
- Observed failure mode:
- `run-sorry-cypress.py --categorize` mutates the active config to the current category batch, so live files such as `specPattern`, `current_vm`, and the newest `/tmp` Cypress JSON only describe the current category, not the full automation run.
- Answering from only the current live batch underreports the run and misses already-finished machines from earlier category batches.
- Action for future runs:
- Reconstruct whole-run status from the generated machine scope plus all machine result artifacts written since the run start time.
- Use the current batch only to identify the live `RUNNING` machine and immediate next machine(s), not as the full run scope.
- Do not answer status requests for categorized runs until earlier category results have been checked as part of the same run.
## Run Learning: 2026-03-11 (Hash-named XML files still belong to machine runs)
- Observed failure mode:
- Same-run JUnit output is not consistently named `test-result-atvm...xml`.
- Many machine results for the same automation run were written as hash-named files such as `test-result-01fe412894862398d06d9cc4bc7e81a0.xml`.
- Limiting status reconstruction to machine-named XML files causes major undercounting of completed machines.
- Action for future runs:
- Parse all `test-result-*.xml` files written since the run start time, not only `test-result-atvm*.xml`.
- Extract the machine name from XML contents such as `testsuite file=`, `testsuite name=`, or `testcase name=` when the filename does not include the machine name.
- Treat `check-xml-files.ts` XML outputs as bookkeeping steps, not machine results.
- Prefer the most recently written same-run XML per machine when multiple XML files exist for that machine.
## Run Learning: 2026-03-12 (Status output must be one machine per line with notes separated)
- Observed requirement:
- Listing multiple completed machines on one line makes run status harder to scan and does not meet the expected reporting format.
- Failure reasons and extra context should be separated from the machine status list so the list stays clean.
- Action for future runs:
- Under completed, skipped, and remaining sections, put exactly one machine status on each line.
- Add a `Notes` section after completed machines for failure reasons, anomalies, and other operator-relevant context.
- Keep completed machine lines in the form `machine-name - STATUS` and avoid appending long explanations inline.
## Run Learning: 2026-03-12 (Add suse15.0 machine to blacklist)
- Observed requirement:
- `atvm144-suse15.0` must be excluded from automation runs because it crashes while creating the migration session.
- Action for future runs:
- Add `atvm144-suse15.0` to the maintained blacklist.
- Record the reason as `CRASHES WHEN CREATING MIGRATION SESSION - BUG`.
- Include it in reusable `--exclude_partial_match` command examples.
## Run Learning: 2026-03-12 (Default to gold-named ATVM config files)
- Observed requirement:
- The automation VM does not reliably have `cypress.atvm-config.ts`, and defaulting to that filename can break runs before they start.
- Operator preference is to use ATVM config files with `gold` in the filename unless explicitly told otherwise.
- Action for future runs:
- Do not reference `cypress.atvm-config.ts` by default in commands or examples.
- Default to `cypress.atvm-config-gold.ts` unless the operator explicitly requests another config.
## Run Learning: 2026-03-12 (Examples are reference-only, not default intent)
- Observed requirement:
- Reusable examples may contain extra excludes or options that the operator did not ask for.
- Carrying those example details into a new run without confirmation can change the requested scope.
- Action for future runs:
- Treat `examples.md` as reference-only.
- Use only the options the operator explicitly requested, plus maintained mandatory blacklist handling.
- Do not assume extra example exclusions such as distro filters are desired unless the operator asks for them.
## Run Learning: 2026-03-12 (Use one status format for all automation run types)
- Observed requirement:
- The operator wants the same ATVM run status display every time, regardless of whether the run is `e2e`, `systemOS`, `reboot`, or another template.
- Changing the display style between run types makes the status harder to scan and compare.
- Action for future runs:
- Use one consistent ATVM status layout for all automation status responses.
- Keep the order the same: build name, completed machines, notes, skipped machines, remaining machines, summary, timing, estimated completion time.
- Keep machine entries one per line as `machine-name - STATUS` regardless of test type.
## Run Learning: 2026-03-13 (Put longer failure description on failed machine line)
- Observed requirement:
- Failed machines are easier to scan when the failure description appears directly on the same line as the machine status.
- A longer same-line description works better than a very short label when the extra detail helps explain what actually failed.
- Action for future runs:
- Format failed machine lines as `machine-name - FAIL - <failure description>`.
- Prefer the longer same-line description when it adds useful operator-facing context.
- Keep `Notes` for broader context, anomalies, and extra follow-up detail beyond the machine-specific failure description.
## Run Learning: 2026-03-14 (Missing requested ATVM config must fail fast)
- Observed requirement:
- If the operator asks for a specific ATVM config file and that file is missing on the automation VM, looking for other config files or substituting a different one creates the wrong next step.
- The operator wants to decide what to do after a missing-config failure.
- Action for future runs:
- If the requested config file is missing, stop immediately and report the missing filename.
- Do not search the automation VM for alternate config files.
- Do not switch to another config unless the operator explicitly instructs it.
## Run Learning: 2026-03-16 (Status requests default to live view with whole-run historical fallback)
- Observed requirement:
- When the operator asks for ATVM automation run status, they want live status by default.
- If no automation is currently running, the status response must fall back to the most recent historical run.
- For categorized runs, the response must still cover the entire run rather than only the latest category batch or cloud sub-run.
- Action for future runs:
- Treat every ATVM status request as a request for live run status unless the operator explicitly asks for something else.
- If no automation is active, reconstruct status from the most recent historical run artifacts and logs.
- For categorized runs, always aggregate all same-run category batches so the response covers the full run scope.
## Run Learning: 2026-03-17 (Default ignore-force-shutdown and iscsi plugin)
- Observed requirement:
- The operator wants `--ignore_force_shutdown` included on every ATVM automation run by default.
- The operator wants plugin selection to default to `--use_specified_plugin iscsi` unless a different plugin is explicitly requested.
- Action for future runs:
- Add `--ignore_force_shutdown` to every `cmc-templates.py` command unless the operator explicitly asks not to use it.
- Default plugin-bearing ATVM automation commands to `--use_specified_plugin iscsi`.
- Only switch away from `iscsi` when the operator explicitly requests `fc`, `both`, or another applicable override.
## Run Learning: 2026-03-18 (ATVM status requests must resolve from the local ATVM workflow, not Cirrus project operations)
- Observed failure mode:
- Interpreting "status of the ATVM automation run" as a request about Cirrus project operations can return the wrong source entirely.
- The operator uses "ATVM automation" to mean the automation contained in the local `atvm` folder and the corresponding automation VM workflow.
- Action for future runs:
- Resolve ATVM status requests from the local ATVM workflow first.
- Check the automation VM at `192.168.3.190` for live runner processes and live files before looking at historical artifacts.
- If no automation is active, reconstruct the most recent historical run from the automation VM shell history and reporter artifacts.
- Do not use Cirrus project operations such as `atvm - cypress` as the source for ATVM automation status unless the operator explicitly asks for project-operation status.
## Run Learning: 2026-03-20 (Display exact ATVM commands and wait for approval before any execution)
- Observed failure mode:
- ATVM run commands were executed before the operator had a chance to review and approve them.
- This happened even though the operator expects a review gate before any ATVM automation command is launched.
- Action for future runs:
- Always display the exact planned ATVM commands before execution.
- Do not run `cmc-templates.py` until the operator explicitly approves the displayed commands.
- Do not run `run-sorry-cypress.py` until the operator explicitly approves the displayed commands.
- Treat template generation as execution that also requires operator approval.
- If any requested option changes after commands are displayed, rebuild and redisplay the commands and wait for fresh approval.
## Run Learning: 2026-03-26 (Verify generated specs directly on the controller before launching the runner)
- Observed failure mode:
- `cmc-templates.py` can successfully generate the requested `.ts` files, but a subsequent run can still start with an incomplete or stale `specPattern` if the runner is launched too early or the verification step is too fragile.
- Shell-escaped regex one-liners used over SSH can fail even when the controller config is actually correct, which makes the verification gate unreliable.
- Action for future runs:
- After `cmc-templates.py`, verify both the generated `.ts` files and the controller config `specPattern` before launching `run-sorry-cypress.py`.
- Prefer direct controller-side inspection of the config block and file presence rather than fragile shell-escaped regex checks.
- If the requested VM list is not visibly present in both places, stop and report the mismatch instead of starting the runner.
## Run Learning: 2026-03-26 (Do not repeat harmless reset-failed watcher noise)
- Observed requirement:
- `systemctl reset-failed atvm-run-watcher@...` often reports that the unit was not loaded.
- In normal watcher startup this has been harmless and does not change the run outcome.
- Repeating that note in routine run confirmations adds noise without helping the operator.
- Action for future runs:
- Do not mention expected, harmless `reset-failed` output in routine run updates.
- Only mention it if it actually prevents watcher startup or becomes relevant to debugging.
## Run Learning: 2026-03-27 (Replace FUNCTIONALLY with TEST FLOW in status output)
- Observed requirement:
- The operator wants the status format to show the full numbered ATVM test flow for the active template rather than a vague high-level `FUNCTIONALLY:` summary.
- Each ATVM template can have its own test-flow step list.
- The step list should appear once for the whole run, not repeated per host.
- Action for future runs:
- Replace the `FUNCTIONALLY:` section with `TEST FLOW:` in ATVM status output.
- Resolve `TEST FLOW:` from the ATVM template name instead of hardcoding one shared list for every template.
- For `cmc-e2e`, use this numbered run flow:
- `1. Verifying set up`
- `2. Power on and obtain ip address and host name`
- `3. Uninstall CMC if still exists`
- `4. Setting up disk on the host`
- `5. Copy CMC install command from GUI`
- `6. Install CMC`
- `7. Create migration session`
- `8. Tracking Changes`
- `9. Trigger cmotion and do I/O test before actual cutover`
- `10. Verify data for cmotion`
- `11. Trigger revert cmotion and do I/O test before and during cmotion`
- `12. Verify data for revert cmotion`
- `13. Trigger cmotion again`
- `14. Finalize cutover`
- `15. Create migration report`
- `16. Delete migration session`
- `17. Verify local destination disk`
- `18. Remove enabled FC integration`
- `19. Remove host and volumes`
- `20. Uninstall CMC`
- `21. Clean up iSCSI targets`
- `22. Power off`
## Run Learning: 2026-03-27 (Template-specific coverage fields and systemOS flow)
- Observed requirement:
- `COVERAGE:` should only show fields that were actually present in the `cmc-templates.py` command for that template.
- Showing an empty integration/plugin path on a template that does not use one adds noise and misleads the reader.
- `cmc-systemOS` needs its own full numbered `TEST FLOW:` list rather than falling back to the generic short placeholder flow.
- `NOTES:` should stay consistent across templates and should not include internal parent-summary recovery notes for `cmc-systemOS`.
- Action for future runs:
- Render `COVERAGE:` from the actual template command inputs used for that run.
- Omit integration/plugin coverage lines when the template command did not use them.
- Use the 21-step `cmc-systemOS` flow from `status-template.md`.
- Keep `NOTES:` template-consistent and operator-facing, without parent-log-summary recovery notes.
## Run Learning: 2026-03-27 (Start watcher before runner when watcher is requested)
- Observed failure mode:
- Starting `run-sorry-cypress.py` before the watcher can race with the watcher helper's stale-log cleanup.
- The watcher helper clears stale `/tmp/<build-name>.log` before startup.
- If the runner has already opened the new log, the helper can delete that live log path, leaving the watcher unable to read the run by filename.
- Action for future runs:
- When the watcher is approved, start the watcher before `run-sorry-cypress.py`.
- Keep the order as: template generation, verification, watcher start, runner start.
- Do not launch the runner first when the watcher is part of the approved command set.
## Run Learning: 2026-03-27 (Watcher must recover when the consolidated run log is missing)
- Observed failure mode:
- A non-categorized watcher run can finish without posting Mattermost even when the ATVM test itself passed.
- In this case the watcher service expected `/tmp/<build-name>.log`, but that consolidated run log was never written.
- The run still produced the final `check-xml-files.ts` XML and fresh per-host reporter artifacts under `cmcReporter/logs/<host>/`.
- Action for future runs:
- Do not rely only on `/tmp/<build-name>.log` for non-categorized watcher result recovery.
- When final `check-xml-files.ts` validation is present but host XML is absent, recover host completion from the latest matching per-host reporter artifact within the run window.
- Keep non-categorized watcher notes accurate; do not describe that failure as a categorized sub-run issue.
## Run Learning: 2026-03-27 (Non-categorized watcher runs must post once and show the full 22-step E2E flow)
- Observed failure mode:
- A non-categorized watcher run for `cmc-e2e` sent two Mattermost posts for the same build.
- The posted `TEST FLOW:` list only showed 18 steps even though the current `cmc-e2e` ATVM flow has 22 steps.
- Action for future runs:
- For non-categorized runs, post only the parent run status and do not also post the single synthetic subrun.
- Keep the static `cmc-e2e` watcher flow aligned with the current 22-step ATVM E2E sequence.
## Run Learning: 2026-03-27 (Use summary-first status layout for ATVM run results)
- Observed requirement:
- The operator wants ATVM run results ordered as `SUMMARY:`, `HOSTS:`, `TIMING:`, `COVERAGE:`, `TEST FLOW:`, then `NOTES:`.
- Action for future runs:
- Render ATVM status output in that section order for both local output and Mattermost posts.
## Run Learning: 2026-03-30 (Give `cmc-reboot` a full template-specific test flow)
- Observed failure mode:
- `cmc-reboot` status output fell back to the generic 5-step placeholder flow.
- The actual reboot workflow is substantially longer and includes reboot-specific validation around cmotion, revert cmotion, and post-reboot disk verification.
- Action for future runs:
- Define a dedicated `cmc-reboot` `TEST FLOW:` in the watcher and status template.
- Keep the reboot flow aligned with the generated reboot Cypress spec rather than the generic fallback list.
## Run Learning: 2026-03-27 (Persist the Currents run URL outside the transient runner log)
- Observed failure mode:
- The watcher can include the Currents run URL in `NOTES:`, but only if it can still read the URL from live runner output or a consolidated run log.
- In practice, `/tmp/<build-name>.log` is not guaranteed to exist, and the host reporter artifacts do not preserve the final Currents run URL.
- Action for future runs:
- Persist the Currents `Recorded Run` URL as soon as `run-sorry-cypress.py` sees it.
- Store it under the watcher state directory for the parent build so it survives runner exit and missing log files.
- Prefer the persisted Currents URL store over transient log scraping when building the final `NOTES:` section.
## Run Learning: 2026-03-27 (Keep ATVM notes meaningful and remove generic artifact-detected lines)
- Observed requirement:
- Generic watcher bookkeeping notes such as "Run finished and one or more sub-run result artifacts were detected." and "Final reporting artifacts were detected." do not add operator value in ATVM status posts.
- Action for future runs:
- Reserve `NOTES:` for meaningful operator-facing content such as the Currents run URL, real anomalies, failure context, and important fallback behavior.
- Do not include generic artifact-detection confirmations in the posted `NOTES:` section.
- Do not include internal fallback notes such as "`check-xml-files.ts` validation passed" or "host details were derived from reporter artifacts" in the posted `NOTES:` section.
## Run Learning: 2026-03-27 (Categorized grouped XML may need host recovery from the subrun's per-host artifact)
- Observed failure mode:
- A categorized subrun can finish and write its grouped `test-result-<build>.xml`, but that XML may only contain `check-xml-files.ts`.
- In that case the watcher may know the grouped batch completed and even know its Currents URL, but still miss the host result unless it recovers the host from the matching per-host reporter artifact.
- Action for future runs:
- For categorized runs, when grouped XML only shows `check-xml-files.ts`, infer the subrun host from the categorized build id and recover the result from the latest matching per-host reporter artifact within the grouped completion window.
- Do not keep a completed grouped subrun in `RUNNING` just because the grouped XML lacked a host testcase entry.
## Run Learning: 2026-03-27 (Categorized batch results must aggregate all hosts in the group and use the earliest grouped host timestamp)
- Observed failure mode:
- A categorized grouped batch can post with only one host even when the batch actually ran multiple hosts of the same distro group.
- This also causes the grouped `start` and `total` timing values to collapse to the last recovered host artifact instead of the full grouped batch duration.
- Action for future runs:
- For categorized grouped batches, recover all matching per-host reporter artifacts for the distro group within the grouped completion window, not only the latest host.
- Derive the grouped `start` time from the earliest recovered host run timestamp and the grouped `end` time from the grouped finalization timestamp.
- Prefer the reporter JSON metadata timestamp or artifact filename timestamp over file write time when reconstructing grouped host timing, because file mtime reflects artifact completion rather than run start.
## Run Learning: 2026-03-27 (Default ATVM approval should include the watcher)
- Observed requirement:
- The operator wants `approve` to mean run with watcher by default.
- The explicit no-watcher override should be `approve without watcher`.
- Action for future runs:
- Treat `approve` as approval to run and start the watcher.
- Treat `approve without watcher` as approval to run without starting the watcher.
## Run Learning: 2026-03-27 (Expand coverage details with operator-relevant run options)
- Observed requirement:
- The operator wants `COVERAGE:` to include more than template and datastore family.
- Useful additions include the config filename and important flags such as `--ignore_force_shutdown`.
- Explicit VM names do not need to be repeated there because the host listing already shows them.
- Action for future runs:
- Include the ATVM config filename in `COVERAGE:`.
- Include important operator-relevant run options such as `--ignore_force_shutdown` in `COVERAGE:`.
- Keep `COVERAGE:` focused on run intent and options, not the explicit target-host list.
- Do not include verbose prose lines such as `scope of this run: ...` in `COVERAGE:`.
- Treat `COVERAGE:` as a concise reflection of the important `cmc-templates.py` command inputs.
## Run Learning: 2026-03-27 (Log the exact template command in NOTES)
- Observed requirement:
- The operator wants `NOTES:` to include the exact `cmc-templates.py` command that triggered the ATVM run.
- The outer `sshpass`/`ssh` wrapper should be omitted, but the command itself should not be trimmed even when long.
- Action for future runs:
- Store and display the exact `cmc-templates.py` command in `NOTES:`.
- Omit only the outer remote-execution wrapper.
## Run Learning: 2026-03-27 (Avoid redundant categorize flags and infer grouped timing stats)
- Observed requirement:
- When `categorize mode: enabled` is already shown in `COVERAGE:`, repeating `--categorize` under `run options` is redundant.
- Grouped categorized results should still show `quickest`, `longest`, and `average` when those values can be inferred from recovered host timing.
- Action for future runs:
- Do not repeat `--categorize` under `run options` when categorize mode is already shown separately.
- When grouped host results are reconstructed from reporter artifacts, infer per-host durations from the recovered host timestamp sequence and grouped end time so grouped timing stats do not default to `n/a` unnecessarily.
## Run Learning: 2026-03-27 (Do not auto-add blacklist excludes for explicitly specified VMs)
- Observed requirement:
- When the operator explicitly specifies the VM or VM list to run, they do not want the maintained `--exclude_partial_match` blacklist added automatically.
- Action for future runs:
- Keep the maintained `--exclude_partial_match` list for broad selectors such as `--containsVm` or `--randomize`.
- When the operator uses `--specify_vms`, do not auto-add the blacklist unless they explicitly request it.
- Even when the operator uses `--specify_vms`, first check whether any requested VM is on the maintained blacklist and stop instead of launching it if one is included.
## Run Learning: 2026-03-30 (Controller watcher deployment must match the repo watcher before trusting live posts)
- Observed failure mode:
- The repo watcher had the corrected `cmc-reboot` flow, but the controller install at `/opt/atvm-watcher-service/atvm_run_watcher.py` still had the old generic 5-step fallback.
- A live categorized reboot subrun therefore posted the stale 5-step `TEST FLOW:` even though the repo copy had already been fixed.
- Action for future runs:
- Before trusting watcher-generated live posts for new watcher behavior, verify that the controller install matches the intended repo watcher version.
- If the controller install is stale and the operator approves it, deploy the updated watcher code to `/opt/atvm-watcher-service` and restart only the watcher instance for the active build.
## Run Learning: 2026-03-30 (Categorized grouped recovery must parse real per-host reporter status, not assume pass)
- Observed failure mode:
- A categorized Red Hat reboot subrun posted both hosts as passed even though `atvm71-redhat9.1` actually failed during `1. Verifying set up`.
- The grouped XML only contained `check-xml-files.ts`, and the watcher incorrectly treated the presence of a per-host reporter artifact as `PASS completed`.
- Action for future runs:
- When grouped XML lacks explicit host testcase results, recover grouped host status from the per-host reporter JSON or equivalent detailed artifact.
- Carry through the real `failures`, `pending`, and failure message from that host artifact instead of assuming `PASS completed`.
- If a correction post is needed because stale or reconstructed state was wrong, mark it explicitly as a correction that supersedes the earlier result.
## Run Learning: 2026-03-30 (Git push must stay manual even after commit approval)
- Observed failure mode:
- After creating a requested local commit, the assistant treated a later `approve` as permission to run `git push`.
- The operator expectation was stricter: the assistant should stop at the local commit and only provide the manual push command reference.
- Action for future runs:
- Treat commit creation and push as separate gates.
- Never execute `git push` for this workspace unless the operator explicitly overrides the workspace rule.
- After creating a local commit, provide the manual push command reference only, defaulting to `git push origin main` unless the operator explicitly asks for a different remote or branch.
- Do not interpret a generic `approve` after a commit as push approval.
## Run Learning: 2026-03-30 (Do not infer plugin execution from generated spec text alone)
- Observed failure mode:
- A generated reboot spec for Pure still contained both iSCSI and FC code blocks, and that was incorrectly treated as proof that both plugin paths would run.
- In this template, the generated file includes both branches, but runtime execution is gated by `Cypress.env("pure_plugin_type")`.
- Action for future runs:
- Do not treat the presence of plugin-specific strings or code blocks in the generated `.ts` file as proof that those plugin steps will execute.
- For plugin-specific questions, determine expected behavior from the template/runtime gate and only call it a mismatch if the runtime logic would execute the wrong plugin path.
- Continue verifying that the requested VM set is present in the generated files and `specPattern`, but keep plugin-path validation separate from simple text-presence checks.
## Run Learning: 2026-03-30 (Do not classify reporter TXT logs as failed from generic `error` words)
- Observed failure mode:
- A completed `reboot-redhat8.10-iscsi` run actually passed in the launch log and `Cloud Run Finished` table, but the watcher saved it as failed.
- The TXT fallback matched generic strings such as `auth error encountered` and treated them as proof of host failure.
- Action for future runs:
- Do not classify a reporter TXT artifact as failed just because it contains the word `error`.
- For TXT fallback, require explicit terminal failure markers such as `cy:command error`, `cy:task error`, or real `Error:`/`AssertionError:`/timeout text.
- Prefer the parent run summary when available, because it is less prone to false failure signals than raw per-step console text.
## Run Learning: 2026-03-30 (Replay exact artifacts before assuming a thin closed-run detail is a current watcher bug)
- Observed failure mode:
- The saved controller state for `reboot-redhat8.10-both` still showed only `1 failures` under the host detail, even though the launch log contained the full md5sum failure text.
- Replaying the exact launch log and reporter artifacts through the currently installed watcher produced the correct host detail with `57 tests, 1 failures` and the failing testcase/error text.
- Action for future runs:
- Before patching the watcher again for a thin closed-run detail, replay the exact run artifacts through the currently installed watcher code.
- Treat a mismatch between saved state and current replay as evidence of a stale in-memory watcher instance or stale deployment, not automatically as a parser regression.
- Use an isolated temp state directory or other no-post path for that replay so historical validation does not repost results.
## Run Learning: 2026-03-30 (Red Hat 8.10 Pure both failure on step 38 was a missing FC reboot-validation artifact with concurrent storage instability)
- Observed failure mode:
- The failing testcase was `38. Verify diskname2Reboot file is the same as diskname2Reboot’s source (Reboot test)`.
- The concrete error was `md5sum: /root/tmp/fcDisk/diskname2Reboot.md5: No such file or directory`.
- On the target after the run, `/root/tmp/fcDisk` contained `diskname2Disk` and `diskname2Disk.md5`, but not `diskname2Reboot.md5`.
- Additional host findings:
- The target showed repeated iSCSI authorization failures and later `Could not log into all portals`.
- `mtdi-driver.service` started at `17:30:26 EDT`.
- `iscsid.service` / `Open-iSCSI` started at `17:30:30 EDT`.
- `iscsi.service`, `mtdi-daemon.service`, and `galaxy-migrate.service` reached active state at `17:32:45 EDT`.
- Repeated multipath reinitialization and `failed to get ... uid` messages continued through the run window.
- Action for future runs:
- If this failure recurs, treat it as a host/storage investigation first, not just a watcher-formatting issue.
- Check whether the FC reboot-validation step actually created `diskname2Reboot.md5` on `/root/tmp/fcDisk` before the md5 verification step ran.
- Check whether repeated iSCSI auth failures or multipath churn during the same boot window are interfering with the expected disk/file state.
## Run Learning: 2026-03-30 (`cmc-reboot` with Pure `both` needs an explicit warning/confirmation gate)
- Observed operator requirement:
- For reboot runs, using both FC and iSCSI together is not a normal default choice.
- There may be a "chicken before the egg" timing problem where iSCSI disks are not attached before mTDI / CMC services start.
- The operator wants `both` on `cmc-reboot` to trigger a warning and an explicit reconfirmation instead of being treated like a routine plugin selection.
- Action for future runs:
- If a planned `cmc-reboot` command includes `--use_specified_plugin both`, call out the FC+iSCSI timing risk before execution.
- Ask the operator to explicitly confirm that `both` is really intended for that reboot run.
- Otherwise prefer `fc` or `iscsi`, but not both.
## Run Learning: 2026-03-30 (Default `--test_partition` on ATVM template commands)
- Observed operator requirement:
- The operator wants `--test_partition` included on ATVM test-template commands by default unless they explicitly say otherwise.
- Action for future runs:
- Add `--test_partition` to `cmc-templates.py` commands by default.
- Omit it only when the operator explicitly asks not to use it.
## Run Learning: 2026-03-30 (Use generated spec as the source of truth for `TEST FLOW:`)
- Observed operator requirement:
- The operator wants the current full workflow steps for the actual test template/run, not a stale hand-maintained flow list.
- Action for future runs:
- Resolve `TEST FLOW:` from the generated `.ts` spec for the actual run whenever that spec exists.
- Extract the numbered `it(...)` steps from the generated spec referenced by the run's `specPattern`.
- Only use template-level or static fallback flow definitions when the generated spec cannot be found or parsed.
## Run Learning: 2026-03-30 (Event-log reporter JSON must not be ignored in non-categorized fallback)
- Observed failure mode:
- A failed non-categorized run still posted/saved host detail as only `1 failures` even though the per-host reporter artifacts preserved the failing step.
- The per-host `.json` artifact used an event-log format with `metadata` plus `tests`, but no top-level `stats` block.
- The watcher ignored that JSON format, fell back to the `.txt`, and lost structured test counts/detail.
- Action for future runs:
- Support the event-log JSON format directly when parsing per-host reporter artifacts.
- In non-categorized fallback, prefer the structured `.json` artifact over the matching `.txt` when they belong to the same run timestamp.
- Recover at least the failing testcase name and a nonzero test count from those artifacts even when the consolidated run log is missing.
## Run Learning: 2026-03-30 (Use `mochawesome` as the rich fallback for host failure detail)
- Observed failure mode:
- The full UI-visible Cypress error text for a failed ATVM host run existed in `cypress/cmcReporter/mochawesome/*.html`, but the lower-fidelity host-level `.json` and `.txt` reporter artifacts only preserved the failing step boundary.
- That made the host detail fall back to a thin summary even though a richer error payload was available on the controller.
- Action for future runs:
- When the consolidated run log is missing, use `mochawesome` as the rich fallback source for per-host failure text before settling for lower-fidelity reporter artifacts.
- Keep the `HOSTS` table compact by showing the failing step plus a short error summary.
- Put the longer trimmed failure excerpt in `NOTES:` instead of dumping the full raw stack trace into the host-detail column.
## Run Learning: 2026-03-30 (Apply rich failed-host detail recovery to every ATVM template)
- Observed operator requirement:
- The same failed-host recovery and formatting rules should apply across all ATVM template runs, not only reboot scenarios.
- If any ATVM test template fails, the result should still recover the best available failure detail and present it consistently.
- Action for future runs:
- Use the same failure-detail recovery order for every ATVM template: consolidated run log, `mochawesome`, structured reporter artifacts, then text reporter artifacts.
- Keep failed-host `Detail` compact and put the longer trimmed excerpt in `FAILURE NOTES:` for every template type.
## Run Learning: 2026-03-30 (Separate failure detail from general notes in ATVM status output)
- Observed operator requirement:
- The `HOSTS` detail column should stay short and scannable.
- Detailed per-host error text should not crowd the host table or mix with general `NOTES:`.
- Action for future runs:
- Keep `HOSTS` detail to the failing step plus a short error summary only.
- Put richer per-host error excerpts in `FAILURE NOTES:`.
- Reserve `NOTES:` for non-failure context such as template command, Currents URL, and operator-facing caveats.
## Run Learning: 2026-03-31 (Mochawesome failure parsing must stay within one testcase object)
- Observed failure mode:
- A reboot failure post showed step `36` with an empty `FAILURE NOTES:` excerpt even though the real failure remained step `38` and the mochawesome HTML contained the full `sshpass` / `md5sum` error text.
- The parser was scanning beyond the current mochawesome testcase object, so it paired one step title with another step's later failed-state/message fields.
- Empty mochawesome `message` / `estack` values must not be accepted as valid failure detail.
- Action for future runs:
- Parse mochawesome one testcase object at a time and do not cross object boundaries when matching `title`, `fullTitle`, `state`, `message`, and `estack`.
- Only use mochawesome to enrich host detail when it returns a non-empty failure payload.
- If mochawesome and structured reporter artifacts disagree on the step number, keep the structured reporter step as the safer fallback for the host detail.
## Run Learning: 2026-03-31 (Generated-spec `TEST FLOW` must not depend only on log-scoped `specPattern`)
- Observed failure mode:
- A completed `e2e-redhat8.10-both` run posted the static 22-step `cmc-e2e` flow even though the generated spec for that exact run contained a longer flow.
- The watcher only extracted generated-spec flow when it could find `Extracted specPattern:` in the available log text.
- When that log-scoped `specPattern` line was unavailable at final render time, the watcher silently fell back to the static template flow.
- Action for future runs:
- Resolve generated-spec `TEST FLOW` from the active config file's `specPattern` when the required log line is missing.
- Treat the static template flow as a last-resort fallback only after both log-derived and config-derived `specPattern` resolution fail.
## Run Learning: 2026-04-16 (Generated-spec `TEST FLOW` must honor `test-install-only` gates)
- Observed failure mode:
- An install-only `cmc-e2e` run for `atvm5-ubuntu22.04` posted the full 22-step `TEST FLOW:` to Mattermost even though the generated spec for that run only executed the shorter install-only path.
- The watcher already used the generated spec as the source of truth, but its gate evaluator did not understand `if (Cypress.env("test-install-only") == true/false)`.
- That left both the install-only branch and the normal post-install branch visible to the flow extractor.
- Action for future runs:
- When extracting `TEST FLOW:` from a generated spec, evaluate `test-install-only` gates the same way plugin and cutover gates are evaluated.
- For install-only runs, exclude the normal post-install branch and report only the actual numbered install-only steps from the generated spec.
## Run Learning: 2026-03-31 (Default vmware compute-migration options for ATVM)
- Observed operator requirement:
- For `cmc-migrateops-compute-migration` runs to VMware, the operator wants a stable default option set instead of having to restate the same platform flags each time.
- Action for future runs:
- Default VMware compute-migration runs to:
- `--ignore_force_shutdown`
- `--vm_platforms vmware`
- `--test_partition`
- `--set_static_ip_dest`
- Only omit or change those options when the operator explicitly overrides them.
## Run Learning: 2026-04-14 (Generated-spec `TEST FLOW` must honor the selected plugin branch)
- Observed failure mode:
- A Pure FC `cmc-e2e` run posted a 39-step `TEST FLOW:` even though the actual FC path for that template uses 22 steps.
- The generated spec contained both `if(useFCPlugin)` and `if(useIscsiPlugin)` blocks, and the watcher counted every `it(...)` step without applying the runtime plugin gate.
- Action for future runs:
- When extracting `TEST FLOW:` from a generated spec, include common steps plus only the runtime-gated plugin branch selected for that run.
- Use watcher metadata such as the approved integration/plugin path to decide whether to include FC steps, iSCSI steps, or both.
- Do not count every plugin-gated branch in the generated spec just because the text is present.
## Run Learning: 2026-04-14 (`cmc-systemOS` should not carry plugin or integration arguments)
- Observed operator requirement:
- `cmc-systemOS` runs should not be planned with `--use_specified_plugin`, `--integration_type`, or watcher integration/plugin metadata.
- Treating `cmc-systemOS` like a plugin-bearing template adds incorrect command arguments and misleading status metadata.
- Action for future runs:
- Plan `cmc-systemOS` template commands without plugin-selection or integration-type arguments.
- When watcher-backed execution is used for `cmc-systemOS`, omit watcher integration/plugin metadata too.
- Keep plugin defaults scoped to templates that actually use plugin selection.
## Run Learning: 2026-04-14 (Plugin-gated `TEST FLOW` filtering must match reboot and other template gate names too)
- Observed failure mode:
- A Pure FC `cmc-reboot` run still posted the combined FC+iSCSI step count even after the earlier `cmc-e2e` fix.
- The watcher only recognized `if(useFCPlugin)` / `if(useIscsiPlugin)` gates, while the reboot templates use names such as `if(usePureFCPlugin)` / `if(usePureIscsiPlugin)`.
- Action for future runs:
- Match plugin-gated generated-spec branches generically by plugin-bearing gate variable name instead of hardcoding only one template's variable names.
- Apply the same plugin-branch filtering logic across ATVM templates so new templates do not need one-off watcher fixes.
- Validate generated-spec `TEST FLOW` against the selected runtime plugin path for reboot and other templates before assuming the generic fix is complete.
## Run Learning: 2026-04-15 (Parent `Cloud Run Finished` parsing must tolerate late host rows after Recorded Run detection)
- Observed failure mode:
- A non-categorized watcher run tested three VMs, but the Mattermost status only showed two hosts.
- In the launch log, the parent `Cloud Run Finished` summary printed one host row, then logged `Detected 'Recorded Run' after 'Cloud Run Finished' - results uploaded successfully.`, then printed the remaining host rows.
- The watcher treated that detection log line as the end of the summary block, so the merged parent-run summary dropped the later host row.
- Action for future runs:
- Do not stop parent summary parsing at the Recorded Run detection log line.
- Bound each `Cloud Run Finished` block by the next run boundary such as the next `Extracted specPattern:` or the next `Cloud Run Finished`, then parse all host rows inside that block.
## Run Learning: 2026-04-16 (Categorized `Cloud Run Finished` parsing must stop at the Recorded Run URL for each grouped batch)
- Observed failure mode:
- A categorized ATVM run completed its Windows batch in the Cypress launch log, but the watcher posted only the earlier grouped results and never sent a separate Windows Mattermost status.
- The watcher let one categorized `Cloud Run Finished` block run forward into the next grouped batch because the next grouped run did not present a fresh `Extracted specPattern:` boundary before the next runner output.
- That let host-row parsing drift across grouped runs, which caused the Windows batch XML to be relabeled under the wrong subrun and left the real Windows subrun stuck in `RUNNING`.
- Action for future runs:
- For categorized grouped recovery, stop each `Cloud Run Finished` block at that grouped run's `🏁 Recorded Run:` line when it is present.
- Do not let categorized summary parsing continue into the next grouped batch's runner output.
- Keep grouped host-row parsing scoped to the actual summary table rows for that grouped run only.
## Run Learning: 2026-04-22 (Wrapped duration rows in parent `Cloud Run Finished` tables must not drop hosts)
- Observed failure mode:
- A non-categorized `cmc-migrateops` run completed four hosts, and the launch log's parent `Cloud Run Finished` table showed all four host rows.
- The saved watcher state still only kept two hosts.
- The Currents summary wrapped the trailing `s` in long duration values such as `16m 13.9s` onto its own continuation row.
- The watcher normalization appended that standalone `s` to the far end of the host row, which broke the host-row regex for those wrapped rows.
- Action for future runs:
- When parsing parent `Cloud Run Finished` tables, treat standalone wrapped `s` rows as duration-cell continuations and remove those rows instead of appending `s` to the end of the host line.
- Rely on the existing duration parser to accept wrapped values without the trailing `s`.
- Replay the exact launch log through the current watcher code after this fix before trusting a corrected host count.
## Run Learning: 2026-05-07 (Synthesize failed host row when hang-kill occurs before reporter artifacts)
- Observed failure mode:
- Some hang-killed runs exit before host-level reporter artifacts are emitted, which can leave Mattermost statuses with `FAILED` summary but no host rows.
- Action for future runs:
- When a run is marked `FAILED` from hang-kill markers or non-zero runner exit and no host results are available, synthesize one failed host row from current host/spec inference.
- Use a clear failure detail such as `hang timeout killed runner` so operator-facing status always includes a concrete host failure line.
## Run Learning: 2026-06-04 (Host detail should not show total test counts)
- Observed failure mode:
- A passing compute-migration run posted `29 tests, 0 failures` in the `HOSTS` detail column while `TEST FLOW:` listed 30 planned/generated spec steps.
- The mismatch was confusing because host detail came from actual Cypress reporter counts, while `TEST FLOW:` came from static generated-spec extraction.
- Action for future runs:
- Keep raw test/failure counts in watcher state for classification and debugging, but do not render them in the `HOSTS` detail column.
- For passing hosts, render detail as `completed`.
- For failed hosts, render only the failing step plus compact error summary; put richer excerpts in `FAILURE NOTES:`.
skills/atvm/references/automation-status-template.md
+176
View File
@@ -0,0 +1,176 @@
# ATVM Status Template
Use this as the default ATVM automation run-status template for:
- local status responses in the terminal
- Mattermost status posts after a completed run
## Layout
```md
## ATVM Run Status
### <build_name>
**SUMMARY:**
| Metric | Value |
|---|---:|
| finished | <n> |
| passed | <n> |
| failed | <n> |
| skipped | <n> |
**HOSTS:**
| Host | Kernel | Status | Detail (For failures, see Failure Notes section below for more details) |
|---|---|---|---|
| <host-name> | <kernel-version or unknown> | ✅ PASS | completed |
| <host-name> | <kernel-version or unknown> | ⚠️ FAIL | <useful failure description> |
| <host-name> | <kernel-version or unknown> | ⏳ RUN | in progress |
| <host-name> | <kernel-version or unknown> | ⏭️ SKIP | <skip reason> |
**TIMING:**
| Metric | Value |
|---|---|
| start | <start time> |
| end | <end time or n/a> |
| total | <total or elapsed runtime> |
| quickest | <host> - <runtime> or n/a |
| longest | <host> - <runtime> or n/a |
| average | <runtime> or n/a |
**COVERAGE:**
- template: `<template-name>`
- categorize mode: `enabled` or `disabled`
- datastore/config family: `<config family>`
- config file: `<config-file-name>`
- migration style: `<high-level test style>`
- integration/plugin path: `<integration/plugin>` when the template command actually uses one
- run options: `<operator-relevant template flags such as --ignore_force_shutdown>`
**TEST FLOW:**
- <template-specific numbered steps>
**FAILURE NOTES:**
- <detailed per-host failure excerpt when a host fails>
**NOTES:**
- <note>
- <note>
```
## Rules
- Keep `SUMMARY:`, `HOSTS:`, `TIMING:`, `COVERAGE:`, `TEST FLOW:`, `FAILURE NOTES:`, and `NOTES:` in that order.
- Use the title format:
- `## ATVM Run Status`
- `### <build_name>`
- Use flat bullet lists for `COVERAGE:` and `TEST FLOW:`.
- Use Markdown tables for `SUMMARY:`, `HOSTS:`, and `TIMING:`.
- Use one host per row in the `HOSTS:` section.
- Include a `Kernel` column immediately after `Host` in the `HOSTS:` table.
- For completed hosts, prefer:
- `✅ PASS`
- `⚠️ FAIL`
- For in-progress or skipped hosts, use:
- `⏳ RUN`
- `⏭️ SKIP`
- Keep `Detail` concise.
- Put richer per-host failure excerpts under `FAILURE NOTES:`, not in the host table.
- Put broader non-failure context under `NOTES:`.
- When available, put the persistent Currents run URL in `NOTES:` so operators can open the exact recorded run directly.
- Include the exact `cmc-templates.py` command used to trigger the run in `NOTES:`, without the outer `sshpass`/`ssh` wrapper.
- Include the exact `run-sorry-cypress.py` command used to launch the run in `NOTES:`, without the outer `sshpass`/`ssh` wrapper.
- Keep `FAILURE NOTES:` limited to detailed per-host error excerpts.
- Keep `NOTES:` limited to meaningful non-failure operator-facing items such as the Currents link, real anomalies, or important fallback behavior.
- Do not include generic watcher bookkeeping lines in `NOTES:` such as "run artifacts were detected" or "final reporting artifacts were detected."
- Do not include internal fallback notes in `NOTES:` such as "`check-xml-files.ts` validation passed" or "host details were derived from reporter artifacts."
- `COVERAGE:` should describe what the run was intended to cover without listing target hosts.
- `COVERAGE:` should mostly mirror the important `cmc-templates.py` command inputs such as template, categorize mode, config filename, any real integration/plugin path, and important flags like `--ignore_force_shutdown`.
- Do not render template-command fields in `COVERAGE:` when that template did not use them.
- If `categorize mode: enabled` is shown, do not also repeat `--categorize` under `run options`.
- When grouped categorized timing is reconstructed from host reporter artifacts, still populate `quickest`, `longest`, and `average` from inferred per-host durations when possible.
- `TEST FLOW:` should describe the template-specific numbered run flow once for the whole test, not per host.
- The watcher should prefer the generated spec for the actual run when it exists, and only include the plugin-gated branch that actually ran.
- `cmc-e2e` currently uses this flow:
- `1. Verifying set up`
- `2. Power on and obtain ip address and host name`
- `3. Uninstall CMC if still exists`
- `4. Setting up disk on the host`
- `5. Copy CMC install command from GUI`
- `6. Install CMC`
- `7. Create migration session`
- `8. Tracking Changes`
- `9. Trigger cmotion and do I/O test before actual cutover`
- `10. Verify data for cmotion`
- `11. Trigger revert cmotion and do I/O test before and during cmotion`
- `12. Verify data for revert cmotion`
- `13. Trigger cmotion again`
- `14. Finalize cutover`
- `15. Create migration report`
- `16. Delete migration session`
- `17. Verify local destination disk`
- `18. Remove enabled FC integration`
- `19. Remove host and volumes`
- `20. Uninstall CMC`
- `21. Clean up iSCSI targets`
- `22. Power off`
- `cmc-systemOS` currently uses this flow:
- `1. Verifying set up`
- `2. Power on and obtain ip address and host name`
- `3. Uninstall CMC if still exists`
- `4. Attach destination disk on the host`
- `5. Copy CMC install command from GUI`
- `6. Install CMC on the host`
- `7. Create migration session (Simple Migration)`
- `8. Tracking Changes (Simple Migration)`
- `9. Finalize cutover (Simple Migration)`
- `10. Create migration report (Simple Migration)`
- `11. Delete migration session (Simple Migration)`
- `12. Power off the host`
- `13. Detach original source OS disk`
- `14. Reassign destination OS disk`
- `15. Power on to verify destination disk`
- `16. Power off the host`
- `17. Detach destination OS disk`
- `18. Attach original source OS disk back`
- `19. Power on and obtain ip address and host name`
- `20. Uninstall CMC on the host`
- `21. Power off the host`
- `cmc-reboot` currently uses this flow:
- `1. Verifying set up`
- `2. Power on and obtain ip address and host name`
- `3. Uninstall CMC if still exists`
- `4. Setting up disk on the host`
- `5. Copy CMC install command from GUI`
- `6. Install CMC on the host`
- `7. Create migration session`
- `8. Tracking Changes`
- `9. Create reboot validation file on the source disk`
- `10. Trigger cmotion`
- `11. Reboot the host`
- `12. Update disk info after power on`
- `13. Mount source disk`
- `14. Verify reboot validation file after reboot`
- `15. Create second reboot validation file on the source disk`
- `16. Revert cmotion`
- `17. Reboot the host`
- `18. Update disk info after power on`
- `19. Mount source disk`
- `20. Verify second reboot validation file after reboot`
- `21. Trigger cmotion and do I/O test during cmotion`
- `22. Revert cmotion`
- `23. Verify data with md5 checksum for the I/O test`
- `24. Trigger cmotion again`
- `25. Finalize cutover`
- `26. Create migration report`
- `27. Delete migration session`
- `28. Verify local destination disk`
- `29. Remove host and disks`
- `30. Remove enabled integration`
- `31. Uninstall CMC on the host`
- `32. Clean up iSCSI targets`
- `33. Power off the host`
- See `/home/aw/code/cds/atvm/docs/automation/examples.md` for `cmc-e2e` examples.
- Resolve kernel values by cross-referencing hostnames against `/home/aw/code/cds/atvm/inventory/vm-inventory.md`.
- If no kernel value can be verified from `vm-inventory.md`, use `unknown`.
- Use the same template for Mattermost and local operator-visible status output.
skills/atvm/references/credentials-contract.md
+65
View File
@@ -0,0 +1,65 @@
# ATVM Accounts And Credentials
This file organizes the ATVM lab account and credential information that was preserved from the original long-form notes.
All secret values have been moved to `/home/aw/code/cds/.env.credentials.local`.
Use the variable names below instead of storing raw credentials in tracked files.
## CMC GCStage
- URL:
- `CMC_GCSTAGE_URL`
## CMC ATVM Test Account
- User:
- `CMC_TEST_USER`
- Password:
- `CMC_TEST_PASSWORD`
- 2FA registration code:
- `CMC_TEST_TOTP_SECRET`
- CMC API token:
- `CIRRUS_API_TOKEN`
- Xray location for failed tests:
- `../cdc-e2e/cypress/cmcXray`
## CMC User-Administration Test Account
- User:
- `CMC_USERADMIN_TEST_USER`
- Password:
- `CMC_USERADMIN_TEST_PASSWORD`
- 2FA registration code:
- `CMC_USERADMIN_TOTP_SECRET`
- CMC API token:
- `CMC_USERADMIN_API_TOKEN`
## Red Hat Free Subscription Account
- User:
- `REDHAT_SUBSCRIPTION_USER`
- Password:
- `REDHAT_SUBSCRIPTION_PASSWORD`
- Registration command:
- `subscription-manager register --username "$REDHAT_SUBSCRIPTION_USER" --password "$REDHAT_SUBSCRIPTION_PASSWORD"`
- Re-register sequence:
```none
# subscription-manager remove --all
# subscription-manager unregister
# subscription-manager clean
# subscription-manager register --username "$REDHAT_SUBSCRIPTION_USER" --password "$REDHAT_SUBSCRIPTION_PASSWORD"
```
- Renewal link:
- `https://developers.redhat.com/products/rhel/download#publicandprivatecloudreadyrhelimages`
## Related Host Credentials
- ATVM controller host:
- `ATVM_CONTROLLER_USER / ATVM_CONTROLLER_PASSWORD`
- `ATVM_CONTROLLER_ALT_USER / ATVM_CONTROLLER_ALT_PASSWORD`
- Linux repository VM:
- `ATVM_REPO_USER / ATVM_REPO_PASSWORD`
- vCenter `192.168.0.201`:
- `VCENTER_USER / VCENTER_PASSWORD`
- ESXi `192.168.1.165`:
- `ESXI_HOST_1_USER / ESXI_HOST_1_PASSWORD`
- ESXi `192.168.1.166`:
- `ESXI_HOST_2_USER / ESXI_HOST_2_PASSWORD`
## Preserved Source
- Full original notes remain in:
- `archive/imported-notes/cypress-automation-for-cmc.md`
skills/atvm/references/imported-notes-cypress-automation-for-cmc.md
+1327
View File
File diff suppressed because it is too large Load Diff
skills/atvm/references/infrastructure.md
+71
View File
@@ -0,0 +1,71 @@
# ATVM Infrastructure
This file organizes the main infrastructure reference that was previously embedded in the long-form ATVM notes.
Tracked docs should reference `/home/aw/code/cds/.env.credentials.local` for secret values instead of storing them inline.
## Storage / Appliance
- Primary DGS Phoenix Server: `192.168.1.172`
- Replica DGS Phoenix Server: `192.168.1.89`
- Primary DGS web login:
- `DGS_PRIMARY_USER / DGS_PRIMARY_PASSWORD`
- The preserved detailed storage and appliance notes remain in:
- `archive/imported-notes/cypress-automation-for-cmc.md`
## VMware
- Active vCenter Server Appliance: `192.168.0.201`
- vCenter login:
- `VCENTER_USER / VCENTER_PASSWORD`
- Primary ESXi hosts:
- `192.168.1.165`
- `192.168.1.166`
- Primary ATVM boot datastores:
- `AutomatedTest-VMBootImg-Gold` (3 TB) -> `gold` -> `cypress.atvm-config-gold.ts`
- `AutomatedTest-VMBootImg-Gold-2` (3 TB) -> `gold-2` -> `cypress.atvm-config-gold-2.ts`
- `AutomatedTest-Cypress` (1 TB)
- `AutomatedTest-VMBootImgComputeMigration-Gold` (3 TB)
- Current VM membership for those datastores should be queried from live vCenter when needed, not stored as a maintained repo-side live reference list.
- ATVM datastore placement rule for explicit VM-list test requests:
- all requested VMs must resolve to one datastore family only
- do not run a mixed request spanning both `AutomatedTest-VMBootImg-Gold` and `AutomatedTest-VMBootImg-Gold-2`
- if the request is mixed, stop and report the per-datastore VM split so the operator can correct the list
- ESXi `192.168.1.165`:
- SSH: `ESXI_HOST_1_USER / ESXI_HOST_1_PASSWORD`
- IPMI: `ESXI_HOST_1_IPMI_USER / ESXI_HOST_1_IPMI_PASSWORD`
- ESXi `192.168.1.166`:
- SSH: `ESXI_HOST_2_USER / ESXI_HOST_2_PASSWORD`
- IPMI: `ESXI_HOST_2_IPMI_USER / ESXI_HOST_2_IPMI_PASSWORD`
- Legacy VMware environment details are preserved in the archived notes.
## ATVM Controller Host
- Controller host reference:
- `atvm-cypress-vm`
- `atvm-cypress-vm-1`
- Controller IP:
- `192.168.3.190`
- Controller credentials:
- `ATVM_CONTROLLER_USER / ATVM_CONTROLLER_PASSWORD`
- `ATVM_CONTROLLER_ALT_USER / ATVM_CONTROLLER_ALT_PASSWORD`
- Current noted controller variants:
- `atvm-cypress-vm`
- `atvm-cypress-vm-1`
## Offline Repository Host
- Repository host:
- `linux-repo-vm`
- Repository IP:
- `192.168.3.199`
- Repository credentials:
- `ATVM_REPO_USER / ATVM_REPO_PASSWORD`
- Repository content families include:
- RedHat
- Debian
- Suse
## Detailed Source Sections
- Storage Array / Appliance
- VMware Hosts
- ATVM Cypress Host
- Offline DVD Linux Repository
These sections are preserved in full at:
- `archive/imported-notes/cypress-automation-for-cmc.md`
skills/atvm/references/inventory-overview.md
+21
View File
@@ -0,0 +1,21 @@
# ATVM Inventory Overview
This directory contains the durable environment reference for the ATVM workspace.
## Inventory Files
- `infrastructure.md`
- environment overview, storage, VMware, ATVM controller host, and offline repo summary
- `accounts-and-credentials.md`
- test accounts, credentials, tokens, and subscription details used in this environment
- `reserved-ips.md`
- reserved IP ranges and per-IP role assignments
- `specialized-reference.md`
- map of the preserved detailed sections for zoning, ISO inventory, guest image inventory, guest configuration, gold images, and temporary clients
- `vm-inventory.md`
- broad VM reference snapshot; do not use it for current `AutomatedTest-VMBootImg-Gold` / `AutomatedTest-VMBootImg-Gold-2` membership during ATVM run planning
## Preserved Source
- Full original notes used by tracked inventory references are preserved at:
- `archive/imported-notes/cypress-automation-for-cmc.md`
Use the inventory files first for organized access, then use the archived source when you need the original full narrative and detailed listings in their original order.
skills/atvm/references/reserved-ips.md
+66
View File
@@ -0,0 +1,66 @@
# ATVM Reserved IPs
This file indexes the reserved IP ranges and role assignments used in the ATVM environment.
## Key Reserved Range
- `192.168.3.176 - 179`
- ATVM infrastructure for compute migration platforms
- `192.168.3.190 - 199`
- ATVM infrastructure
- `192.168.3.180 - 189`
- unit testing ATVM clones
## High-Use Addresses
- `192.168.3.190`
- ATVM Cypress controller host
- `192.168.3.191`
- ATVM source machine static IP target
- `192.168.3.192`
- H2H destination hosts
- `192.168.3.193`
- second source-machine static IP target set
- `192.168.3.194`
- second H2H destination host set
- `192.168.3.195`
- CMC helper for compute migration ops
- `192.168.3.196`
- ATVM OpenStack host
- `192.168.3.197`
- ATVM OpenStack secondary address / host note
- `192.168.3.198`
- currently unassigned in the archived notes
- `192.168.3.199`
- Linux repository VM
## Compute Migration Infrastructure
- `192.168.3.176`
- ATVM oVirt engine
- `192.168.3.177`
- ATVM KVM host
- `192.168.3.178`
- ATVM Proxmox host
- `192.168.3.179`
- currently unassigned in the archived notes
## Unit Test Clone Notes
- `192.168.3.180`
- currently unassigned in the archived notes
- `192.168.3.181`
- `aw999-redhat9.6`, unstructured migration client
- `192.168.3.182`
- currently unassigned in the archived notes
- `192.168.3.183`
- `atvmxxx-w2k22-uefi`
- `192.168.3.184`
- currently unassigned in the archived notes
- `192.168.3.185`
- `atvm999-oracle8.10`, unstructured migration migrator
- `192.168.3.186 - 189`
- currently unassigned in the archived notes
## Primary Source
The full reserved IP listing and per-IP notes are preserved in:
- `archive/imported-notes/cypress-automation-for-cmc.md`
## Relevant Archived Section
- `## __Reserved IP Address__`
skills/atvm/references/setup-guide.md
+222
View File
@@ -0,0 +1,222 @@
# ATVM Setup Script Guide
This file is guide-only documentation for running and maintaining the ATVM setup workflow.
Do not put dated run examples here.
## Scope
- Client setup script: `/home/aw/code/cds/atvm/scripts/atvm-setup-script.sh`
- Controller wrapper: `/home/aw/code/cds/atvm/scripts/run-atvm-setup-and-collect-log.sh`
- Run-learnings log: `/home/aw/code/cds/atvm/docs/setup/run-learnings.md`
## Purpose
The setup flow performs a controlled bootstrap across supported Linux distributions:
1. Validate target host identity using expected IP + expected hostname before any configuration.
2. Fix repositories (especially CD/DVD media repo entries).
3. On Ubuntu, configure root SSH password-login workflow using `ATVM_TARGET_PASSWORD` for follow-on root operations.
4. On Oracle Linux, set default boot kernel to non-UEK when available.
5. Disable unattended auto-upgrades on Ubuntu.
6. Remove specific storage-related packages and install base tooling.
7. Disable SELinux on Red Hat-family systems.
8. Configure static IP as the final step.
9. Print final summary and write logs to `atvm_setup_script.log`.
10. On SELinux-capable distros, reboot and verify runtime SELinux status post-reboot.
11. Keep client powered on after successful setup so controller-side log collection + SHA256 verification can complete.
12. Power off from controller only after successful verification and no setup errors.
## Execution Model
- Shell safety flags: `set -euo pipefail`
- Logging: colorized console + plain text log file
- Entry point: `main "$@"`
- Default operator assumption for setup access: source `/home/aw/code/cds/.env.credentials.local` and use `ATVM_TARGET_USER` plus `ATVM_TARGET_PASSWORD` unless explicitly overridden.
- When the operator refers to `192.168.3.191`, treat it as the default ATVM target host.
- For SSH to `192.168.3.191`, ignore host key mismatch by default with `-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null`.
- For SSH to `192.168.3.191`, source `/home/aw/code/cds/.env.credentials.local` and use `ATVM_TARGET_USER` plus `ATVM_TARGET_PASSWORD` unless the operator explicitly provides different credentials.
## Mandatory Identity Gate
Setup must not start unless operator explicitly provides both values:
- `--expected-ip <ip>`
- `--expected-hostname <hostname>`
Rules:
- Connect to the operator-provided target IP directly.
- Do not pre-scan alternate candidate IPs.
- Do not infer hostname from target.
- If hostname is missing from request, stop and ask for it.
- If detected hostname does not exactly match expected hostname, stop immediately.
- If expected IP is not assigned on target, stop immediately.
## Finish ATVM VM Hardware Prep
When the operator says `finish prepping the ATVM VM hardware for <vm-name>`, perform only the standard vCenter hardware prep for that exact VM name.
Safety rules:
- Operate only on the exact VM name supplied by the operator.
- Verify the VM exists and is powered off before changing hardware.
- Do not delete, destroy, remove from inventory, or unregister the ATVM VM.
- Do not create replacement disks when the standard shared ATVM data disks are missing; stop and report the missing datastore paths.
- If the VM already has conflicting data disks, FC passthrough devices, or occupied unit numbers, stop and report instead of adding duplicates.
Standard hardware:
- Attach the two existing data disks from the VM's current ATVM boot datastore under `atvm-DISKS/`:
- `atvm-DISK_1.vmdk` as the first added data disk, normally Hard disk 2 / SCSI unit 1.
- `atvm-DISK_2.vmdk` as the second added data disk, normally Hard disk 3 / SCSI unit 2.
- Add the two FC passthrough ports:
- `0000:85:00.0`
- `0000:85:00.1`
- Reserve all guest memory after adding passthrough devices.
Preferred `govc` flow:
```bash
source /home/aw/code/cds/.env.credentials.local
export GOVC_URL="https://${VCENTER_HOST}/sdk"
export GOVC_USERNAME="$VCENTER_USER"
export GOVC_PASSWORD="$VCENTER_PASSWORD"
export GOVC_INSECURE=1
vm="/CDSHQ-Eng/vm/<vm-name>"
ds="<current-atvm-boot-datastore>"
govc vm.info "$vm"
govc device.info -vm "$vm"
govc datastore.ls -ds "$ds" atvm-DISKS
govc vm.disk.attach -vm "$vm" -ds "$ds" -disk atvm-DISKS/atvm-DISK_1.vmdk -link=false
govc vm.disk.attach -vm "$vm" -ds "$ds" -disk atvm-DISKS/atvm-DISK_2.vmdk -link=false
govc device.pci.ls -vm "$vm"
govc device.pci.add -vm "$vm" 0000:85:00.0
govc device.pci.add -vm "$vm" 0000:85:00.1
memory_mb="$(govc object.collect -s "$vm" config.hardware.memoryMB)"
govc vm.change -vm "$vm" -memory-pin=true -mem.reservation "$memory_mb"
```
Operational notes:
- Determine `<current-atvm-boot-datastore>` from live vCenter inventory for the VM.
- Use `-link=false` for the standard ATVM data disks so the existing datastore disks are attached directly, matching the reference ATVM inventory pattern.
- Add FC passthrough ports one at a time. Adding both addresses in one `govc device.pci.add` call can fail with duplicate device-key errors.
- Verify memory reservation by confirming `config.memoryAllocation.reservation` equals `config.hardware.memoryMB`.
## Canonical Run Order
1. `parse_args`
2. `validate_target_host_identity`
3. `check_sudo`
4. `fix_repositories`
5. `configure_ubuntu_root_ssh_access` (Ubuntu only)
6. `install_sudo_if_needed`
7. `configure_oracle_non_uek_kernel` (Oracle Linux only)
8. `disable_ubuntu_auto_upgrades` (Ubuntu only)
9. `run_package_installation`
10. `disable_selinux` (RHEL-family only)
11. `configure_static_ip` (final configuration step)
12. `print_final_summary`
13. `reboot_and_verify_selinux_if_needed`
14. `poweroff_client_if_successful` (controller-driven after verification)
## Core Behavior By Step
### Repository Fix
- Debian/Ubuntu: comment `cdrom` entries in apt lists and run `apt-get update`.
- RHEL-family/Oracle: disable media/cdrom/dvd repo entries and run `yum clean all && yum makecache`.
- Fedora: same model via `dnf clean all && dnf makecache`.
- openSUSE/SLES: disable CD/DVD repos with `zypper mr -d` and refresh.
### Oracle Linux Kernel Handling
- Oracle Linux only.
- Select first non-UEK kernel via `grubby --info=ALL` and set GRUB default.
- Track whether default changed and whether reboot is required.
### Ubuntu Root SSH Workflow
- Ubuntu only.
- Require `ATVM_TARGET_PASSWORD` in the environment, then set the root password to that value and unlock the root account.
- Write `/etc/ssh/sshd_config.d/99-atvm-root-login.conf` enabling root + password auth.
- Validate config and restart SSH service.
### Ubuntu Auto-Upgrade Disable
- Ubuntu only.
- Update `/etc/apt/apt.conf.d/20auto-upgrades` to disable periodic update/upgrade actions.
### Package Installation
- Package manager detection order: `apt-get`, `dnf`, `yum`, `zypper`, `pacman`, `apk`.
- Pre-cleanup removes multipath/iSCSI packages where applicable.
- Installs kernel headers per distro.
- Base package set includes:
`curl wget git vim perl gdb scsitools net-tools parted fio ca-certificates python3 elfutils-libelf-devel`
### SELinux Disable
- RHEL-family only.
- If enforcing/permissive, backup and rewrite `/etc/selinux/config` to disabled.
- Marks reboot recommendation/requirement in summary.
### Static IP Configuration (Final Step)
Hardcoded target values:
- IP: `192.168.3.191`
- Prefix: `22`
- Gateway: `192.168.0.1`
- DNS: `8.8.8.8`, `8.8.4.4`
Interface detection priority:
1. default-route interface
2. first non-loopback interface with IPv4
3. first non-loopback interface from link list
Network-stack handling includes `netplan`, `NetworkManager`/`nmcli`, `wicked`, and legacy `ifcfg` fallback patterns.
### SELinux Reboot Verification
- Applies to `rhel`, `centos`, `rocky`, `almalinux`, `fedora`, `ol` when SELinux changed.
- Creates one-time systemd verifier service before reboot.
- Post-reboot service records runtime `getenforce` and self-removes.
- On success/no real errors, keeps client on for controller log copy/hash verification before controller power-off.
- On errors, leaves client on for manual inspection.
## Power-State Rules
- After successful setup, keep client powered on until controller log collection + SHA256 verification completes.
- If verification succeeds and no real error lines exist (`^\[ERROR\]`), controller powers off client.
- If any real error lines exist, keep client powered on.
## Logging and Verification
- Client log filename: `atvm_setup_script.log`
- Common client log path when run as root: `/root/atvm_setup_script.log`
- Controller collected log naming: `atvm_configuration_<hostname>_<yyyymmdd_hhmmss>.log`
Required post-run validation:
1. Copy client log to controller `atvm/log/` path.
2. Compare SHA256 between client and copied controller log.
3. Require exact match.
## Preferred Execution Commands
Direct client execution:
```bash
source /home/aw/code/cds/.env.credentials.local
sudo bash /home/cirrususer/atvm-setup-script.sh \
--expected-ip <current-client-ip> \
--expected-hostname <exact-hostname>
```
Controller run + collect:
```bash
source /home/aw/code/cds/.env.credentials.local
EXPECTED_IP_ARG=<current-client-ip> EXPECTED_HOSTNAME_ARG=<exact-hostname> \
/home/aw/code/cds/atvm/scripts/run-atvm-setup-and-collect-log.sh
```
Controller collect-only after client run:
```bash
source /home/aw/code/cds/.env.credentials.local
/home/aw/code/cds/atvm/scripts/run-atvm-setup-and-collect-log.sh --collect-after-complete
```
## Troubleshooting
- If local collected log is missing, do not rerun full setup just for log recovery.
- Use collect-only mode and verify SHA256 after copy.
- If wrapper appears stuck after IP/reboot transition, stop older wrapper sessions and run one fresh collect-only session.
- If `sshpass` is missing on controller, wrapper can still run but may require repeated interactive password prompts.
## Operational Caveats
- Not fully idempotent for all paths; repeated runs may rewrite network configs and create multiple backups.
- Static IP values are hardcoded; adjust before use in other environments.
- Run in maintenance windows because network changes can interrupt active sessions.
- Preserve host identity gating; do not weaken expected IP/hostname checks.
## Update Rule
- After each run, update this file only for guide/rule/checklist/default behavior changes.
- Put run-specific outcomes in `run-learnings.md` only when the run produced a new learning.
skills/atvm/references/setup-run-learnings.md
+40
View File
@@ -0,0 +1,40 @@
# ATVM Setup Script Runs
This file stores run-specific examples only when a run produced a new learning relevant to future tasks.
## Entry Rule
- Add an entry only when the run changed workflow behavior, exposed a new failure mode, or confirmed a new required check.
- Do not add routine runs with no new learning.
## Run Learning: 2026-03-03 (Ubuntu 24.04)
- Environment:
- Initial IP: `192.168.0.89`
- Final static IP: `192.168.3.191`
- Hostname: `atvm-codextest-vm-1`
- Learning:
- Root SSH password workflow backed by `ATVM_TARGET_PASSWORD` and the log copy/hash verification path are valid end-to-end.
- Wrapper must enforce identity arguments for run-and-collect mode.
- Action for future runs:
- Require `EXPECTED_IP_ARG` and `EXPECTED_HOSTNAME_ARG` for wrapper run-and-collect.
## Run Learning: 2026-03-05 (RHEL 9)
- Environment:
- Initial IP: `192.168.3.212`
- Final static IP: `192.168.3.191`
- Hostname: `atvm-codextest-vm-2`
- Learning:
- SELinux disable path with reboot + post-reboot verifier worked.
- Auto power-off can race controller-side log collection if done too early.
- Action for future runs:
- Keep client powered on until controller log copy + SHA256 verification completes.
- Only then perform controller-side power-off when no real error lines are present.
## Run Learning: 2026-03-06 (Oracle Linux 9)
- Environment:
- Initial IP: `192.168.0.121`
- Final static IP: `192.168.3.191`
- Hostname: `atvm-codextest-vm`
- Learning:
- Wrapper auto power-off was blocked by false-positive error detection from instructional text.
- Action for future runs:
- Match only real error log lines using `^\[ERROR\]` for power-off gating.
skills/atvm/references/specialized-reference.md
+29
View File
@@ -0,0 +1,29 @@
# ATVM Specialized Reference Map
The archived ATVM source file still contains several specialized detailed sections that were preserved intact.
## Archived Source
- `archive/imported-notes/cypress-automation-for-cmc.md`
## Section Map
- `## __Zoning__`
- zoning details and SAN-related reference material
- `## VM Guest OS ISO Inventory`
- ISO inventory and media reference
- `## List of Available VM Guest OS Images:`
- guest image listing
- `## VM Guest OS Configuration - Linux`
- Linux guest setup and configuration reference
- `## Switch from SEL=disabled to SEL=enforcing`
- SELinux mode transition notes
- `## __VM Guest OS Gold Images__`
- gold image listings and notes
- `## Temporary Unit Test Clients`
- temporary client inventory
## Notable Current Archive Updates
- Guest image references include newer entries for RHEL 9.8, RHEL 10.2, and Ubuntu 26.04.
- Gold-image notes include recent codex/script preparation notes for Oracle 9.7, Oracle 10.1, Rocky 9.7, Rocky 10.1, Ubuntu 26.04, Redhat 9.8, and Redhat 10.2.
- Temporary unit test client notes now list `atvmxxx-w2k25`, `aw999-oracle10-1-tj`, `cdcpe-va-aw-2`, and `cdcpe-va-aw`.
Use this file as a map when you need those detailed sections in the preserved archived notes.
skills/atvm/references/vm-inventory.md
+173
View File
@@ -0,0 +1,173 @@
This file is a broad ATVM VM reference snapshot.
For current membership on `AutomatedTest-VMBootImg-Gold` and `AutomatedTest-VMBootImg-Gold-2`, query live vCenter when selecting `gold` vs `gold-2` for an ATVM run.
| OS | Hostname | Kernel Version | Notes |
| --- | --- | --- | --- |
| Amazon Linux 2 | atvm1-amazonlinux2 | 4.14.301-224.520.amzn2.x86_64 | |
| Amazon Linux 2023 | atvm114-amazonlinux2023 | 6.1.61-85.141.amzn2023.x86_64 | |
| CentOS 6.0 | atvm6-centos6.0 | 2.6.32-71.el6.x86_64 | BLACKLISTED: CMC INSTALL - CAN'T COMPILE |
| CentOS 6.1 | atvm7-centos6.1 | 2.6.32-131.0.15.el6.x86_64 | |
| CentOS 6.2 | atvm8-centos6.2 | 2.6.32-220.el6.x86_64 | |
| CentOS 6.3 | atvm9-centos6.3 | 2.6.32-279.el6.x86_64 | |
| CentOS 6.4 | atvm10-centos6.4 | 2.6.32-358.el6.x86_64 | |
| CentOS 6.5 | atvm11-centos6.5 | 2.6.32-431.el6.x86_64 | |
| CentOS 6.6 | atvm12-centos6.6 | 2.6.32-504.el6.x86_64 | |
| CentOS 6.7 | atvm13-centos6.7 | 2.6.32-573.el6.x86_64 | |
| CentOS 6.8 | atvm14-centos6.8 | 2.6.32-642.el6.x86_64 | |
| CentOS 6.9 | atvm15-centos6.9 | 2.6.32-696.el6.x86_64 | |
| CentOS 6.10 | atvm16-centos6.10 | 2.6.32-754.el6.x86_64 | |
| CentOS 7.0 | atvm17-centos7.0 | 3.10.0-123.el7.x86_64 | |
| CentOS 7.1 | atvm18-centos7.1 | 3.10.0-229.el7.x86_64 | |
| CentOS 7.2 | atvm19-centos7.2 | 3.10.0-327.el7.x86_64 | |
| CentOS 7.3 | atvm20-centos7.3 | 3.10.0-514.el7.x86_64 | |
| CentOS 7.4 | atvm21-centos7.4 | 3.10.0-693.el7.x86_64 | |
| CentOS 7.5 | atvm22-centos7.5 | 3.10.0-862.el7.x86_64 | |
| CentOS 7.6 | atvm23-centos7.6 | 3.10.0-957.el7.x86_64 | |
| CentOS 7.7 | atvm24-centos7.7 | 3.10.0-1062.el7.x86_64 | |
| CentOS 7.8 | atvm25-centos7.8 | 3.10.0-1127.el7.x86_64 | |
| CentOS 7.9 | atvm26-centos7.9 | 3.10.0-1160.el7.x86_64 | |
| CentOS 8.0 | atvm27-centos8.0 | 4.18.0-80.el8.x86_64 | |
| CentOS 8.1 | atvm28-centos8.1 | 4.18.0-147.el8.x86_64 | |
| CentOS 8.2 | atvm29-centos8.2 | 4.18.0-193.el8.x86_64 | |
| CentOS 8.3 | atvm30-centos8.3 | 4.18.0-240.el8.x86_64 | |
| CentOS 8.4 | atvm31-centos8.4 | 4.18.0-305.3.1.el8.x86_64 | |
| CentOS 8.5 | atvm32-centos8.5 | 4.18.0-348.el8.x86_64 | |
| Debian 9.0.0 | atvm113-debian9.0.0 | 4.9.0-3-amd64 | BLACKLISTED: SUPPORT REQUEST - WAITING |
| Debian 9.1.0 | atvm115-debian9.1.0 | 4.9.0-3-amd64 | BLACKLISTED: SUPPORT REQUEST - WAITING |
| Debian 9.2.0 | atvm116-debian9.2.0 | 4.9.0-4-amd64 | BLACKLISTED: SUPPORT REQUEST - WAITING |
| Debian 9.3 | atvm156-debian9.3.0 | 4.9.0-4-amd64 | BLACKLISTED: RE-CREATE MIGHT BE NEEDED |
| Debian 12.10.0 | atvm146-debian12.10.0 | 6.1.0-34-amd64 | |
| Debian 13.0.0 | atvm157-debian13.0.0 | 6.12.38+deb13-amd64 | |
| Debian 13.1.0 | atvm158-debian13.1.0 | 6.12.43+deb13-amd64 | |
| Debian 13.2.0 | atvm159-debian13.2.0 | 6.12.57+deb13-amd64 | |
| Fedora 27 | atvm104-fedora27 | 4.13.9-300.fc27.x86_64 | |
| Fedora 28 | atvm105-fedora28 | 4.16.3-301.fc28.x86_64 | |
| Fedora 33 | atvm106-fedora33 | 5.8.15-301.fc33.x86_64 (default kernel); 5.14.18-100.fc33.x86_64 | |
| Fedora 34 | atvm107-fedora34 | 5.11.12-300.fc34.x86_64 | |
| Oracle Linux 6.0 | atvm73-oracle6.0 | 2.6.32-71.el6.x86_64 | BLACKLISTED: CMC INSTALL - CAN'T COMPILE |
| Oracle Linux 6.1 | atvm74-oracle6.1 | 2.6.32-131.0.15.el6.x86_64 | |
| Oracle Linux 6.2 | atvm75-oracle6.2 | 2.6.32-220.el6.x86_64 | |
| Oracle Linux 6.3 | atvm76-oracle6.3 | 2.6.32-279.el6.x86_64 | |
| Oracle Linux 6.4 | atvm77-oracle6.4 | 2.6.32-358.el6.x86_64 | |
| Oracle Linux 6.5 | atvm78-oracle6.5 | 2.6.32-431.el6.x86_64 | |
| Oracle Linux 6.6 | atvm79-oracle6.6 | 2.6.32-504.el6.x86_64 | |
| Oracle Linux 6.7 | atvm80-oracle6.7 | 2.6.32-573.el6.x86_64 | |
| Oracle Linux 6.8 | atvm81-oracle6.8 | 2.6.32-642.el6.x86_64 | |
| Oracle Linux 6.9 | atvm82-oracle6.9 | 2.6.32-696.el6.x86_64 | |
| Oracle Linux 6.10 | atvm83-oracle6.10 | 2.6.32-754.el6.x86_64 | |
| Oracle Linux 7.0 | atvm84-oracle7.0 | 3.10.0-123.el7.x86_64 | |
| Oracle Linux 7.1 | atvm85-oracle7.1 | 3.10.0-229.el7.x86_64 | |
| Oracle Linux 7.2 | atvm86-oracle7.2 | 3.10.0-327.el7.x86_64 | |
| Oracle Linux 7.3 | atvm87-oracle7.3 | 3.10.0-514.el7.x86_64 | |
| Oracle Linux 7.4 | atvm88-oracle7.4 | 3.10.0-693.el7.x86_64 | |
| Oracle Linux 7.5 | atvm89-oracle7.5 | 3.10.0-862.el7.x86_64fc | |
| Oracle Linux 7.6 | atvm90-oracle7.6 | 3.10.0-957.el7.x86_64 | |
| Oracle Linux 7.7 | atvm91-oracle7.7 | 3.10.0-1062.el7.x86_64 | |
| Oracle Linux 7.8 | atvm92-oracle7.8 | 3.10.0-1127.el7.x86_64 | |
| Oracle Linux 7.9 | atvm93-oracle7.9 | 3.10.0-1160.el7.x86_64 | |
| Oracle Linux 7.9 UEK / SEL | atvm133-oracle7.9-us | 5.4.17-2102.201.3.el7uek.x86_64 | |
| Oracle Linux 8.0 | atvm94-oracle8.0 | 4.18.0-80.el8.x86_64 | |
| Oracle Linux 8.1 | atvm95-oracle8.1 | 4.18.0-147.el8.x86_64 | |
| Oracle Linux 8.2 | atvm96-oracle8.2 | 4.18.0-193.el8.x86_64 | |
| Oracle Linux 8.3 | atvm97-oracle8.3 | 4.18.0-240.el8.x86_64 | |
| Oracle Linux 8.4 | atvm98-oracle8.4 | 4.18.0-305.el8.x86_64 | |
| Oracle Linux 8.5 | atvm99-oracle8.5 | 4.18.0-348.el8.x86_64 | |
| Oracle Linux 8.6 | atvm100-oracle8.6 | 4.18.0-372.9.1.el8.x86_64 | |
| Oracle Linux 8.7 | atvm101-oracle8.7 | 4.18.0-425.3.1.el8.x86_64 | |
| Oracle Linux 8.8 | atvm129-oracle8.8 | 4.18.0-477.10.1.el8_8.x86_64 | |
| Oracle Linux 8.9 | atvm130-oracle8.9 | 4.18.0-513.5.1.el8_9.x86_64 | |
| Oracle Linux 8.10 | atvm131-oracle8.10 | 4.18.0-553.el8_10.x86_64 | |
| Oracle Linux 8.10 UEK / SEL | atvm134-oracle8.10-us | 5.15.0-206.153.7.1.el8uek.x86_64 | |
| Oracle Linux 9.0 | atvm102-oracle9.0 | 5.14.0-70.13.1.0.3.el9_0.x86_64 | |
| Oracle Linux 9.1 | atvm103-oracle9.1 | 5.14.0-162.6.1.el9_1.x86_64 | |
| Oracle Linux 9.2 | atvm117-oracle9.2 | 5.14.0-284.11.1.el9_2.x86_64 | |
| Oracle Linux 9.3 | atvm118-oracle9.3 | 5.14.0-362.8.1.el9_3.x86_64 | |
| Oracle Linux 9.3 UEK / SEL | atvm135-oracle9.3-us | 5.15.0-200.131.27.el9uek.x86_64 | |
| Oracle Linux 9.4 | atvm136-oracle9.4 | 5.14.0-427.13.1.el9_4.x86_64 | |
| Oracle Linux 9.4 OPT | atvm138-oracle9.4-opt | 5.14.0-427.13.1.el9_4.x86_64 | |
| Oracle Linux 9.5 | atvm150-oracle9.5 | 5.14.0-503.11.1.el9_5.x86_64 | |
| Oracle Linux 9.6 | atvm149-oracle9.6 | 5.14.0-570.12.1.0.1.el9_6.x86_64 | |
| Oracle Linux 9.7 | atvm162-oracle9.7 | 5.14.0-611.5.1.el9_7.x86_64 | |
| Oracle Linux 10.0 | atvm154-oracle10.0 | 6.12.0-55.9.1.0.1.el10_0.x86_64 | |
| Oracle Linux 10.1 | atvm163-oracle10.1 | 6.12.0-124.8.1.el10_1.x86_64 | |
| Redhat Linux 6.0 | atvm41-redhat6.0 | 2.6.32-71.el6.x86_64 | BLACKLISTED: CMC INSTALL - CAN'T COMPILE |
| Redhat Linux 6.1 | atvm42-redhat6.1 | 2.6.32-131.0.15.el6.x86_64 | |
| Redhat Linux 6.2 | atvm43-redhat6.2 | 2.6.32-220.el6.x86_64 | |
| Redhat Linux 6.3 | atvm44-redhat6.3 | 2.6.32-279.el6.x86_64 | |
| Redhat Linux 6.4 | atvm45-redhat6.4 | 2.6.32-358.el6.x86_64 | |
| Redhat Linux 6.5 | atvm46-redhat6.5 | 2.6.32-431.el6.x86_64 | |
| Redhat Linux 6.6 | atvm47-redhat6.6 | 2.6.32-504.el6.x86_64 | |
| Redhat Linux 6.7 | atvm48-redhat6.7 | 2.6.32-573.el6.x86_64 | |
| Redhat Linux 6.8 | atvm49-redhat6.8 | 2.6.32-642.el6.x86_64 | |
| Redhat Linux 6.9 | atvm50-redhat6.9 | 2.6.32-696.el6.x86_64 | |
| Redhat Linux 6.10 | atvm51-redhat6.10 | 2.6.32-754.el6.x86_64 | |
| Redhat Linux 7.0 | atvm52-redhat7.0 | 3.10.0-123.el7.x86_64 | |
| Redhat Linux 7.1 | atvm53-redhat7.1 | 3.10.0-229.el7.x86_64 | |
| Redhat Linux 7.2 | atvm54-redhat7.2 | 3.10.0-327.el7.x86_64 | |
| Redhat Linux 7.3 | atvm55-redhat7.3 | 3.10.0-514.el7.x86_64 | |
| Redhat Linux 7.4 | atvm56-redhat7.4 | 3.10.0-693.el7.x86_64 | |
| Redhat Linux 7.5 | atvm57-redhat7.5 | 3.10.0-862.el7.x86_64 | |
| Redhat Linux 7.6 | atvm58-redhat7.6 | 3.10.0-957.el7.x86_64 | |
| Redhat Linux 7.7 | atvm59-redhat7.7 | 3.10.0-1062.el7.x86_64 | |
| Redhat Linux 7.8 | atvm60-redhat7.8 | 3.10.0-1127.el7.x86_64 | |
| Redhat Linux 7.9 | atvm61-redhat7.9 | 3.10.0-1160.el7.x86_64 | |
| Redhat Linux 8.0 | atvm62-redhat8.0 | 4.18.0-80.el8.x86_64 | |
| Redhat Linux 8.1 | atvm63-redhat8.1 | 4.18.0-147.el8.x86_64 | |
| Redhat Linux 8.2 | atvm64-redhat8.2 | 4.18.0-193.el8.x86_64 | |
| Redhat Linux 8.3 | atvm65-redhat8.3 | 4.18.0-240.el8.x86_64 | |
| Redhat Linux 8.4 | atvm66-redhat8.4 | 4.18.0-305.el8.x86_64 | |
| Redhat Linux 8.5 | atvm67-redhat8.5 | 4.18.0-348.el8.x86_64 | |
| Redhat Linux 8.6 | atvm68-redhat8.6 | 4.18.0-372.9.1.el8.x86_64 | |
| Redhat Linux 8.7 | atvm69-redhat8.7 | 4.18.0-425.3.1.el8.x86_64 | |
| Redhat Linux 8.8 | atvm124-redhat8.8 | 4.18.0-477.27.1.el8_8.x86_64 | |
| Redhat Linux 8.9 | atvm125-redhat8.9 | 4.18.0-513.24.1.el8_9.x86_64 | |
| Redhat Linux 8.10 | atvm126-redhat8.10 | 4.18.0-553.16.1.el8_10.x86_64 | |
| Redhat Linux 9.0 | atvm70-redhat9.0 | 5.14.0-70.22.1.el9_0.x86_64 | |
| Redhat Linux 9.1 | atvm71-redhat9.1 | 5.14.0-162.6.1.el9_1.x86_64 | |
| Redhat Linux 9.2 | atvm72-redhat9.2 | 5.14.0-284.11.1.el9_2.x86_64 | |
| Redhat Linux 9.3 | atvm120-redhat9.3 | 5.14.0-362.8.1.el9_3.x86_64 | |
| Redhat Linux 9.4 | atvm122-redhat9.4 | 5.14.0-427.33.1.el9_4.x86_64 | |
| Redhat Linux 9.5 | atvm139-redhat9.5 | 5.14.0-503.14.1.el9_5.x86_64 | |
| Redhat Linux 9.6 | atvm147-redhat9.6 | 5.14.0-570.12.1.el9_6.x86_64 | |
| Redhat Linux 9.7 | atvm160-redhat9.7 | 5.14.0-611.16.1.el9_7.x86_64 | |
| Redhat Linux 9.8 | atvm167-redhat9.8 | 5.14.0-70.22.1.el9_0.x86_64 | used codex / script |
| Redhat Linux 10.0 | atvm148-redhat10.0 | 6.12.0-55.9.1.el10_0.x86_64 | |
| Redhat Linux 10.1 | atvm161-redhat10.1 | 6.12.0-124.8.1.el10_1.x86_64 | |
| Redhat Linux 10.2 | atvm168-redhat10.2 | 6.12.0-211.7.3.el10_2.x86_64 | used codex / script |
| Rocky Linux 8.4 | atvm33-rocky8.4 | 4.18.0-305.3.1.el8_4.x86_64 | |
| Rocky Linux 8.5 | atvm34-rocky8-5 | 4.18.0-348.el8.0.2.x86_64 | |
| Rocky Linux 8.6 | atvm35-rocky8.6 | 4.18.0-372.9.1.el8.x86_64 | |
| Rocky Linux 8.7 | atvm36-rocky8.7 | 4.18.0-425.3.1.el8.x86_64 | |
| Rocky Linux 8.8 | atvm37-rocky8.8 | 4.18.0-477.10.1.el8_8.x86_64 | |
| Rocky Linux 8.9 | atvm127-rocky8.9 | 4.18.0-513.5.1.el8_9.x86_64 | |
| Rocky Linux 8.10 | atvm128-rock8.10 | 4.18.0-553.el8_10.x86_64 | |
| Rocky Linux 9.0 | atvm38-rocky9.0 | 5.14.0-70.13.1.el9_0.x86_64 | |
| Rocky Linux 9.1 | atvm39-rocky9.1 | 5.14.0-162.6.1.el9_1.x86_64 | |
| Rocky Linux 9.2 | atvm40-rocky9.2 | 5.14.0-284.11.1.el9_2.x86_64 | |
| Rocky Linux 9.3 | atvm119-rocky9.3 | 5.14.0-362.8.1.el9_3.x86_64 | |
| Rocky Linux 9.4 | atvm123-rocky9.4 | 5.14.0-427.13.1.el9_4.x86_64 | |
| Rocky Linux 9.5 | atvm151-rocky9.5 | 5.14.0-503.14.1.el9_5.x86_64 | |
| Rocky Linux 9.6 | atvm152-rocky9.6 | 5.14.0-570.17.1.el9_6.x86_64 | |
| Rocky Linux 9.7 | atvm164-rocky9.7 | 5.14.0-611.5.1.el9_7.x86_64 | |
| Rocky Linux 10.0 | atvm153-rocky10.0 | 6.12.0-55.12.1.el10_0.x86_64 | |
| Rocky Linux 10.1 | atvm165-rocky10.1 | 6.12.0-124.8.1.el10_1.x86_64 | |
| Suse Linux 15.0 | atvm144-suse15.0 | 4.12.14-23-default | BLACKLISTED: CRASHES WHEN CREATING MIGRATION SESSION - BUG |
| Suse Linux 15.1 | atvm143-suse15.1 | 4.12.14-195-default | |
| Suse Linux 15.2 | atvm142-suse15.2 | 5.3.18-22-default | |
| Suse Linux 15.3 | atvm141-suse15.3 | 5.14.21-150400.22-default | |
| Suse Linux 15.4 | atvm137-suse15.4 | 5.14.21-150400.22-default | |
| Suse Linux 15.5 | atvm132-suse15.5 | 5.14.21-150500.53-default | |
| Suse Linux 15.6 | atvm140-suse15.6 | 6.4.0-150600.21-default | |
| Suse Linux 15.7 | atvm155-suse15.7 | 6.4.0-150700.51-default | |
| Ubuntu 16.04 | atvm2-ubuntu16.04 | 4.4.0-21-generic | |
| Ubuntu 18.04 | atvm3-ubuntu18.04 | 4.15.0-20-generic | |
| Ubuntu 20.04 | atvm4-ubuntu20.04 | 5.4.0-144-generic | |
| Ubuntu 22.04 | atvm5-ubuntu22.04 | 5.15.0-25-generic | |
| Ubuntu 24.04 | atvm121-ubuntu24.04 | 6.8.0-31-generic | |
| Ubuntu 26.04 | atvm166-ubuntu26.04 | 7.0.0-15-generic | |
| Windows Server 2008R2 | atvm108-w2k8r2 | 6.1.7601 Service Pack 1 Build 7601 | |
| Windows Server 2012 | atvm109-w2k12R2 | 6.3.9600 Build 9600 | |
| Windows Server 2016 | atvm110-w2k16 | 10.0.14393 Build 14393 | |
| Windows Server 2019 | atvm111-w2k19 | 10.0.17763 Build 17763 | |
| Windows Server 2022 | atvm112-w2k22 | 10.0.20438 Build 20348 | |
| Windows Server 2025 | atvm145-w2k25 | 10.0.26100 Build 26100 | |
skills/atvm/references/workflow-conventions.md
+30
View File
@@ -0,0 +1,30 @@
# ATVM Workspace Conventions
## File Roles
- `guide.md`
- authoritative workflow rules, defaults, and checklists
- `examples.md`
- reusable command examples only
- `run-learnings.md`
- dated lessons captured only when a run produced a new lasting insight
- `inventory/*.md`
- durable environment reference and listings
- `archive/imported-notes/*.md`
- preserved source material kept for completeness and traceability
## Update Policy
- Update `docs/setup/guide.md` when setup/bootstrap workflow behavior changes.
- Update `docs/automation/guide.md` when automation workflow behavior changes.
- Update `docs/automation/examples.md` when a reusable example pattern changes.
- Update `run-learnings.md` files only when a run created a new lesson worth preserving.
- Keep inventory details in `inventory/` rather than mixing them into workflow guides.
## Path Conventions
- Prefer actual repo paths under `/home/aw/code/cds/atvm/...` in documentation.
- Keep top-level navigation in `README.md` and `AGENTS.md`.
- Keep executable assets under `scripts/`.
## Archive Policy
- Preserve imported long-form notes under `archive/imported-notes/`.
- Do not rely on archived notes as the primary operational runbook when a current guide exists.
- Keep detailed listings available; reorganization should improve navigation, not remove information.