diff --git a/atvm/AGENTS.md b/atvm/AGENTS.md index f34c9a4..65889f9 100644 --- a/atvm/AGENTS.md +++ b/atvm/AGENTS.md @@ -42,6 +42,8 @@ This file defines how to operate and maintain the ATVM workspace in `/home/aw/co - For SSH to `192.168.3.191`, ignore host key mismatch by default with `-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null`. - For Linux SSH to `192.168.3.191`, source `/home/aw/code/cds/.env.credentials.local` and use `ATVM_TARGET_USER` plus `ATVM_TARGET_PASSWORD` unless explicitly overridden. - Use `ATVM_WINDOWS_TARGET_USER` plus `ATVM_WINDOWS_TARGET_PASSWORD` for Windows guest access to the same ATVM target IP unless explicitly overridden. +- For `finish prepping the ATVM VM hardware for `, follow `docs/setup/guide.md` section `Finish ATVM VM Hardware Prep`. +- Standard ATVM VM hardware prep means the two shared `atvm-DISKS` data disks, FC passthrough ports `0000:85:00.0` and `0000:85:00.1`, and full guest-memory reservation. ## Automation Track Defaults - Controller host: `atvm-cypres-vm-1` diff --git a/atvm/README.md b/atvm/README.md index 1d5e79a..e5e41f2 100644 --- a/atvm/README.md +++ b/atvm/README.md @@ -11,6 +11,8 @@ This folder contains the ATVM setup workflow, ATVM Cypress automation workflow, - `docs/workflow/conventions.md` - Environment inventory and credentials: - `inventory/overview.md` +- Portable Codex skill: + - `../skills/atvm/` ## Layout - `scripts/` @@ -25,6 +27,7 @@ This folder contains the ATVM setup workflow, ATVM Cypress automation workflow, - environment inventory, credentials, reserved IPs, and reference maps - `archive/imported-notes/` - preserved original long-form notes that were reorganized into the current structure + - includes `README.md` identifying the canonical imported note ## Practical Guidance - Use `guide.md` files for authoritative workflow behavior. @@ -33,3 +36,4 @@ This folder contains the ATVM setup workflow, ATVM Cypress automation workflow, - For detailed ATVM automation test results, especially per-host failures, use `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter` on the automation controller. - Use `inventory/` for persistent environment information such as hosts, credentials, and IP assignments. - Use `archive/imported-notes/` when you need the original full historical notes in their original layout. +- Keep `../skills/atvm/` aligned when ATVM workflow rules need to be portable outside this workspace. diff --git a/atvm/archive/imported-notes/README.md b/atvm/archive/imported-notes/README.md new file mode 100644 index 0000000..1aba19a --- /dev/null +++ b/atvm/archive/imported-notes/README.md @@ -0,0 +1,13 @@ +# Imported Notes + +This directory preserves long-form ATVM source notes in their imported form. + +## Files + +- `cypress-automation-for-cmc.md` + - canonical imported note used by tracked ATVM inventory references + - updated from the 2026-07-13 import + +## Usage + +Use the organized files under `../../inventory/` and `../../docs/` first. Use these imported notes only when the original narrative or detailed historical listing is needed. diff --git a/atvm/docs/automation/guide.md b/atvm/docs/automation/guide.md index 974e38b..c6b98cc 100644 --- a/atvm/docs/automation/guide.md +++ b/atvm/docs/automation/guide.md @@ -138,6 +138,10 @@ Completed-run verification sequence: - When the operator provides an explicit VM list, check vCenter placement for every requested VM before choosing the config file. - Before presenting any ATVM run commands for an explicit VM-list request, tell the operator that the next step is a live vCenter placement check for the requested VMs and that the result will determine whether the run must use `gold` or `gold-2`. - For vCenter inspection and placement checks, prefer `govc` and raw vCenter REST calls when they are available before reaching for alternate wrappers. +- Never delete, destroy, remove from inventory, or unregister any ATVM VM from vCenter. +- Treat a vCenter VM as protected ATVM inventory if its name starts with `atvm`, if it resides on an `AutomatedTest-*` ATVM datastore, or if it is an ATVM platform/controller VM, even when it is not listed in `inventory/vm-inventory.md`. +- Preserve ATVM vCenter VM inventory records even when power, maintenance, datastore, cleanup, or prep work is requested. +- For cleanup workflows, only power off, detach/unmount supporting resources, or report the needed manual action; do not remove ATVM VM inventory records. - For `govc`-based placement checks, use `govc vm.info -json ` and parse the lowercase JSON keys such as `virtualMachines` and `datastore`. - Resolve each returned datastore managed-object reference to a datastore name with `govc object.collect -s name` before deciding between `gold` and `gold-2`. - Ignore non-boot helper datastores such as install ISO attachments when applying the `gold` vs `gold-2` rule; base the family decision on the ATVM boot datastore membership. diff --git a/atvm/docs/setup/guide.md b/atvm/docs/setup/guide.md index d6c5da3..ccb6a6b 100644 --- a/atvm/docs/setup/guide.md +++ b/atvm/docs/setup/guide.md @@ -45,6 +45,57 @@ Rules: - If detected hostname does not exactly match expected hostname, stop immediately. - If expected IP is not assigned on target, stop immediately. +## Finish ATVM VM Hardware Prep +When the operator says `finish prepping the ATVM VM hardware for `, perform only the standard vCenter hardware prep for that exact VM name. + +Safety rules: +- Operate only on the exact VM name supplied by the operator. +- Verify the VM exists and is powered off before changing hardware. +- Do not delete, destroy, remove from inventory, or unregister the ATVM VM. +- Do not create replacement disks when the standard shared ATVM data disks are missing; stop and report the missing datastore paths. +- If the VM already has conflicting data disks, FC passthrough devices, or occupied unit numbers, stop and report instead of adding duplicates. + +Standard hardware: +- Attach the two existing data disks from the VM's current ATVM boot datastore under `atvm-DISKS/`: + - `atvm-DISK_1.vmdk` as the first added data disk, normally Hard disk 2 / SCSI unit 1. + - `atvm-DISK_2.vmdk` as the second added data disk, normally Hard disk 3 / SCSI unit 2. +- Add the two FC passthrough ports: + - `0000:85:00.0` + - `0000:85:00.1` +- Reserve all guest memory after adding passthrough devices. + +Preferred `govc` flow: +```bash +source /home/aw/code/cds/.env.credentials.local +export GOVC_URL="https://${VCENTER_HOST}/sdk" +export GOVC_USERNAME="$VCENTER_USER" +export GOVC_PASSWORD="$VCENTER_PASSWORD" +export GOVC_INSECURE=1 + +vm="/CDSHQ-Eng/vm/" +ds="" + +govc vm.info "$vm" +govc device.info -vm "$vm" +govc datastore.ls -ds "$ds" atvm-DISKS + +govc vm.disk.attach -vm "$vm" -ds "$ds" -disk atvm-DISKS/atvm-DISK_1.vmdk -link=false +govc vm.disk.attach -vm "$vm" -ds "$ds" -disk atvm-DISKS/atvm-DISK_2.vmdk -link=false + +govc device.pci.ls -vm "$vm" +govc device.pci.add -vm "$vm" 0000:85:00.0 +govc device.pci.add -vm "$vm" 0000:85:00.1 + +memory_mb="$(govc object.collect -s "$vm" config.hardware.memoryMB)" +govc vm.change -vm "$vm" -memory-pin=true -mem.reservation "$memory_mb" +``` + +Operational notes: +- Determine `` from live vCenter inventory for the VM. +- Use `-link=false` for the standard ATVM data disks so the existing datastore disks are attached directly, matching the reference ATVM inventory pattern. +- Add FC passthrough ports one at a time. Adding both addresses in one `govc device.pci.add` call can fail with duplicate device-key errors. +- Verify memory reservation by confirming `config.memoryAllocation.reservation` equals `config.hardware.memoryMB`. + ## Canonical Run Order 1. `parse_args` 2. `validate_target_host_identity` diff --git a/atvm/inventory/overview.md b/atvm/inventory/overview.md index 85ce383..bc45e71 100644 --- a/atvm/inventory/overview.md +++ b/atvm/inventory/overview.md @@ -15,7 +15,7 @@ This directory contains the durable environment reference for the ATVM workspace - broad VM reference snapshot; do not use it for current `AutomatedTest-VMBootImg-Gold` / `AutomatedTest-VMBootImg-Gold-2` membership during ATVM run planning ## Preserved Source -- Full original notes are preserved at: +- Full original notes used by tracked inventory references are preserved at: - `archive/imported-notes/cypress-automation-for-cmc.md` Use the inventory files first for organized access, then use the archived source when you need the original full narrative and detailed listings in their original order. diff --git a/skills/README.md b/skills/README.md new file mode 100644 index 0000000..4eb4b76 --- /dev/null +++ b/skills/README.md @@ -0,0 +1,12 @@ +# Codex Skills + +This directory stores portable Codex skills that should be versioned with this repo. + +## ATVM +- Path: `skills/atvm/` +- Purpose: portable ATVM operational behavior for setup/bootstrap, ATVM Cypress automation, watcher-backed run control, status reporting, vCenter safety rules, and ATVM documentation maintenance. +- Source of truth for live ATVM workspace docs remains `atvm/`; the skill bundles matching reference snapshots so it can be installed or copied to another machine without requiring the full workspace. + +When ATVM procedures change, update both: +- the live workspace docs under `atvm/` +- the portable skill under `skills/atvm/` diff --git a/skills/atvm/SKILL.md b/skills/atvm/SKILL.md new file mode 100644 index 0000000..9b78e2d --- /dev/null +++ b/skills/atvm/SKILL.md @@ -0,0 +1,256 @@ +--- +name: atvm +description: Portable ATVM operations skill for setup/bootstrap, ATVM Cypress automation planning and execution, watcher-aware run control, live and historical run-status reporting, Mattermost posting on request, and maintenance of ATVM workflow docs and inventory. Use when working with an ATVM target host, the ATVM automation controller, or ATVM-specific workflow rules, defaults, and artifacts. +--- + +# ATVM + +Use this skill for ATVM setup, ATVM Cypress automation, ATVM run monitoring, watcher-aware launch flow, run-status reporting, and maintenance of ATVM operational documentation. + +This is a portable packaged skill. Prefer the bundled `references/` docs first. If a live repo copy of the ATVM workspace exists, use it as an execution source when appropriate, but do not assume it exists. + +## Read First + +Pick only the files needed for the current task. + +### Setup/bootstrap +- `references/setup-guide.md` +- `references/setup-run-learnings.md` +- `references/atvm-prep-reference.md` + +### Automation execution +- `references/automation-guide.md` +- `references/automation-examples.md` +- `references/automation-run-learnings.md` +- `references/automation-status-template.md` + +### Workspace maintenance conventions +- `references/workflow-conventions.md` + +### Environment reference +- `references/inventory-overview.md` +- `references/infrastructure.md` +- `references/reserved-ips.md` +- `references/specialized-reference.md` +- `references/vm-inventory.md` + +### Credentials and local environment contract +- `references/credentials-contract.md` + +### Historical long-form notes +- `references/imported-notes-cypress-automation-for-cmc.md` + +## Operating Model + +This skill is execute-by-default for ATVM automation runs. + +Always: +- treat git/commit approval as separate from ATVM run execution +- never execute `git push` on the operator's behalf unless a different workspace policy explicitly overrides that rule +- default ATVM automation runs to watcher-backed execution unless the operator explicitly requests no watcher +- after run launch, report the exact executed `cmc-templates.py` and `run-sorry-cypress.py` commands + +Only when explicitly requested by the operator: +- show planned ATVM commands before execution as a review step + +## Core Defaults + +### Setup track +Default ATVM target host values: +- target host reference: `192.168.3.191` +- static IP target: `192.168.3.191/22` +- gateway: `192.168.0.1` +- DNS: `8.8.8.8`, `8.8.4.4` +- client log file: `atvm_setup_script.log` + +Default Linux target credentials: +- `ATVM_TARGET_USER` +- `ATVM_TARGET_PASSWORD` + +Default Windows target credentials: +- `ATVM_WINDOWS_TARGET_USER` +- `ATVM_WINDOWS_TARGET_PASSWORD` + +For SSH to the default target host, ignore host key mismatch by default: +- `-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null` + +Use `references/setup-guide.md` for the hardware-prep trigger phrase: +- `finish prepping the ATVM VM hardware for ` + +That procedure adds the standard two ATVM data disks, two FC passthrough ports, and full guest-memory reservation for the exact VM name supplied by the operator. + +### Automation track +Default controller values: +- controller host: `atvm-cypres-vm-1` +- controller IP: `192.168.3.190` + +Default controller credentials: +- `ATVM_CONTROLLER_USER` +- `ATVM_CONTROLLER_PASSWORD` + +Detailed controller artifact root: +- `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter` + +Automation defaults: +- default plugin-bearing template plugin: `--use_specified_plugin iscsi` +- default config family: `gold` +- do not include Gold Disk identifiers in `--build_name` +- `--build_name` must not contain spaces; use `-` between words +- for multiple VMs in the same distro, prefer distro-scoped filtering (`--containsVm`) over long explicit VM lists +- always include `--ignore_force_shutdown` unless explicitly told not to +- always include `--test_partition` unless explicitly told not to +- for `cmc-migrateops-compute-migration` to VMware, default to: + - `--vm_platforms vmware` + - `--set_static_ip_dest` +- for vCenter inspection and placement checks, prefer `govc` and raw vCenter REST calls before alternate wrappers when available +- treat `cmc-systemOS` as not using plugin or integration-type arguments; do not add `--use_specified_plugin`, `--integration_type`, or watcher integration/plugin metadata for that template +- do not auto-add a maintained `--exclude_partial_match` blacklist when the operator explicitly targets named VMs with `--specify_vms` +- even for explicit `--specify_vms` requests, first check whether any requested VM is on the maintained blacklist and stop if it is + +## Mandatory Safety Rules + +### Setup +- never run setup without both `--expected-ip` and `--expected-hostname` +- keep static IP configuration as the final setup step +- do not weaken host identity validation +- for ATVM hardware prep, operate only on the exact VM name, verify the VM is powered off before hardware changes, and never delete, destroy, remove from inventory, or unregister the VM + +### Automation +- before any automation run, always check whether automation is already running +- treat any previous controller running-state result as stale once a new ATVM request arrives; perform a fresh live controller check at request time instead of relying on the immediately previous result +- always report whether automation is already running +- if automation is already running, terminate only with explicit approval +- after `cmc-templates.py`, always verify that generated spec files and the config `specPattern` still include every requested VM before launching `run-sorry-cypress.py` +- if any requested VM is missing, stop and report the mismatch +- when watcher mode is in use (default), start the watcher before `run-sorry-cypress.py` +- when watcher mode is in use (default), build the watcher-start command so it automatically includes the exact executed `cmc-templates.py` command via `--template-command` and the exact executed `run-sorry-cypress.py` command via `--runner-command`; do not require the operator to restate them separately +- when watcher mode is in use (default), prefer the controller-local `atvm-runner@...` systemd service over detached SSH background launch patterns for `run-sorry-cypress.py` +- do not start the runner before the watcher, because watcher startup may clear stale `/tmp/.log` files and can delete a fresh live runner log if the runner starts first +- when starting both watcher and runner, prefer the combined `start-atvm-run.sh` wrapper so they are not launched in parallel against the same `/tmp/.log` +- do not infer plugin behavior from stray strings in generated specs; determine it from runtime gates such as `Cypress.env(...)` +- for `cmc-reboot`, treat `--use_specified_plugin both` as a separate confirmation gate and warn about the FC+iSCSI timing risk unless the operator explicitly reconfirms `both` +- for vCenter VM snapshot requests, default the snapshot name to `VM Snapshot [mm/dd/yyyy:hh:mm:ss AM/PM]` in the local `America/New_York` timezone unless the operator explicitly requests a different name +- never delete, destroy, remove from inventory, or unregister any ATVM VM from vCenter +- treat a vCenter VM as protected ATVM inventory if its name starts with `atvm`, if it resides on an `AutomatedTest-*` ATVM datastore, or if it is an ATVM platform/controller VM, even when it is not listed in `inventory/vm-inventory.md` +- preserve ATVM vCenter VM inventory records even when power, maintenance, datastore, cleanup, or prep work is requested; for cleanup, only power off, detach/unmount supporting resources, or report the needed manual action + +### Red Hat manual tasks (non-automation) +- for manually requested tasks on Red Hat machines, run the subscription reset/register sequence before the requested task +- do not apply this step to ATVM automation runs where template/runtime scripts already handle the subscription flow +- use credentials from the local environment source: + - `REDHAT_SUBSCRIPTION_USER` + - `REDHAT_SUBSCRIPTION_PASSWORD` +- required sequence: + - `subscription-manager remove --all` + - `subscription-manager unregister` + - `subscription-manager clean` + - `subscription-manager register --username "$REDHAT_SUBSCRIPTION_USER" --password "$REDHAT_SUBSCRIPTION_PASSWORD"` + +### Status confirmation +Treat watcher state as cached output only: +- `/var/lib/atvm-run-watcher//state.json` + +Before confirming a completed run, verify in this order: +1. launch log +2. matching reporter artifacts +3. `Cloud Run Finished` summary / Currents URL +4. watcher state only as comparison + +Never confirm a completed run from watcher state alone. + +For categorized runs: +- never report a grouped sub-run as `PASS` from watcher `host_results`, grouped XML, or a lone `check-xml-files.ts` result by itself +- before reporting a grouped sub-run as `PASS`, confirm that the matching child batch also passed in the live launch log or the final `Cloud Run Finished` summary for that child run + +### Mattermost +- if the operator asks for ATVM run status without mentioning Mattermost, respond locally only +- if the operator explicitly asks to send status to Mattermost, use the locally configured: + - `MATTERMOST_ATVM_WEBHOOK` + - `MATTERMOST_ATVM_CHANNEL` +- send the final status only after the run fully completes + +## Credentials Contract + +Do not bundle real credentials inside the skill. + +Use this resolution order: +1. operator-provided env vars in the current shell +2. operator-provided local env file +3. machine-local default env file if one is known and present + +Expected variables: +- `ATVM_TARGET_USER` +- `ATVM_TARGET_PASSWORD` +- `ATVM_WINDOWS_TARGET_USER` +- `ATVM_WINDOWS_TARGET_PASSWORD` +- `ATVM_CONTROLLER_USER` +- `ATVM_CONTROLLER_PASSWORD` +- `MATTERMOST_ATVM_WEBHOOK` +- `MATTERMOST_ATVM_CHANNEL` + +If credentials are missing, stop and ask for the local source or the missing variable names. Do not invent fallback secrets. + +## Execution Entry Points + +Prefer documented scripts and commands over ad hoc rewrites. + +### Setup entry points +- client setup script: `scripts/atvm-setup-script.sh` when available in the current workspace +- controller wrapper: `scripts/run-atvm-setup-and-collect-log.sh` when available in the current workspace +- controller prep script: `/root/atvm_prep/atvm_prep.py` on `192.168.3.190`; show the exact planned command and wait for explicit operator approval before execution + +If the local ATVM repo is not present, use the procedural rules in the bundled references and ask the operator for the actual script location before execution. + +### Automation entry points +Typical controller-side tools: +- `cmc-templates.py` +- `run-sorry-cypress.py` + +Treat their exact location as environment-specific unless a local repo or controller path is available. + +## Artifact Recovery and Reporting + +For detailed host-level failure analysis, prefer: +- controller run log +- `cmcReporter/logs/` +- `cmcReporter/mochawesome/` +- structured reporter artifacts such as JSON and XML +- plain text reporter artifacts last + +Use the status layout from `references/automation-status-template.md`. + +Keep the `HOSTS` detail compact. +Put richer failure excerpts in `FAILURE NOTES:`. +Reserve `NOTES:` for non-failure context such as template command, Currents URL, and operator-facing caveats. +Apply failed-host detail recovery consistently for every ATVM template run, not just `cmc-reboot`. +For failed hosts, recover detail in this order when available: consolidated run log, `mochawesome`, structured reporter artifacts such as JSON and XML, then plain text artifacts. +When reporting `TEST FLOW:`, prefer the numbered steps extracted from the generated spec for that exact run. +If the generated spec exists, do not rely on a static template flow list. +Only fall back to template-level or static flow definitions when the generated spec cannot be located or parsed. + +## Maintenance Rules + +When changing workflow behavior: +- update the relevant guide content + +When adding reusable command patterns: +- update the automation examples content + +When a run produces a new lasting lesson: +- update the relevant run learnings content + +Keep durable environment reference in the inventory reference files. +Do not remove detailed inventory or credential contract information unless explicitly instructed. + +## Practical Use Pattern + +1. Classify the task as setup, automation execution, status/reporting, watcher behavior, or documentation maintenance. +2. Read only the smallest relevant reference set. +3. Build exact run commands and execute by default for automation requests. +4. If the operator explicitly asks to review commands first, show planned commands before execution. +5. For any new run request, re-check live controller running state before deciding whether the previous run is still active. +6. If running automation, verify generated specs and `specPattern` before runner launch. +7. Default to watcher-backed launch unless explicitly told not to. +8. After launch, report immediate execution outcome and the exact executed template/runner commands. +9. If reporting status, use live evidence and reporter artifacts before cached watcher state. +10. If editing docs, keep rules in guide content, reusable commands in examples content, dated lessons in run learnings, and durable environment data in inventory content. diff --git a/skills/atvm/agents/openai.yaml b/skills/atvm/agents/openai.yaml new file mode 100644 index 0000000..a72c3e2 --- /dev/null +++ b/skills/atvm/agents/openai.yaml @@ -0,0 +1,7 @@ +interface: + display_name: "ATVM" + short_description: "Portable ATVM operations workflow" + default_prompt: "Use $atvm to plan or operate an ATVM setup, automation run, watcher flow, or status report." + +policy: + allow_implicit_invocation: true diff --git a/skills/atvm/references/atvm-prep-reference.md b/skills/atvm/references/atvm-prep-reference.md new file mode 100644 index 0000000..78ac09a --- /dev/null +++ b/skills/atvm/references/atvm-prep-reference.md @@ -0,0 +1,118 @@ +# ATVM Prep Script Reference + +This document summarizes the behavior of the controller-local prep script: +- Host: `atvm-cypress-vm-1` (`192.168.3.190`) +- Path: `/root/atvm_prep/atvm_prep.py` +- Execution rule: run all `atvm_prep.py` commands on `192.168.3.190` (ATVM Cypress controller). + +Use this as a quick runbook when ATVM datastore/lun prep is requested. + +## Purpose +`atvm_prep.py` orchestrates DGS and VMware actions around a target DGS LUN name for ATVM test environments. It handles clone lifecycle cleanup, snapshot clone assignment, datastore mount/unmount operations, and optional VM register/unregister workflows. + +## Primary Modes +The script supports these operation modes: + +1. `-A` / `--automated_testing` +- Prepare environment for automated testing. +- Cleans up prior assigned clone/lun state, creates a new clone from snapshot, assigns it, and force-mounts unresolved VMFS. + +2. `-M` / `--maintenance` +- Prepare for maintenance against the specified DGS LUN. +- Cleans clone state, reassigns original/gold volume, rescans storage, and mounts the datastore. + +3. `-R` / `--register` +- Register VMs only for the LUN currently assigned to the target client. +- Scans VMX files on datastore and registers missing VMs. + +4. `-U` / `--unregister` +- Power off and unregister VMs only for the LUN currently assigned to the target client. + +5. `-rvgd` / `--rollback_vm_gold_disk` +- Roll back target volume to a selected snapshot, then assign/mount for use. + +## Main Inputs +Notable arguments: +- `-n` / `--dgs_lun_name` (required): target DGS LUN base name +- `-s` / `--snapshot_name`: optional snapshot name; latest used when omitted +- `-c` / `--client_name`: defaults to `CDS1-ESX165` +- `-vi` / `--vcenter_ip`: defaults to `192.168.0.201` +- `-vu` / `--vcenter_username` +- `-vp` / `--vcenter_password` +- `-e` / `--esx_host_ip`: defaults to `192.168.1.165` + +## Workspace Shorthand Mapping +For requests in this workspace, operator shorthand may be expanded as follows: + +1. Datastore/LUN shorthand for `-n` +- `Gold` -> `AutomatedTest-VMBootImg-Gold` +- `Gold2` -> `AutomatedTest-VMBootImg-Gold-2` +- `ComputeMigration` (or `compute migration`) -> `AutomatedTest-VMBootImgComputeMigration-Gold` + +2. Client/ESXi pairing (`-c` with `-e`) +- `CDS1-ESX165` <-> `192.168.1.165` +- `CDS1-ESX166` <-> `192.168.1.166` + +3. Inference rules +- If only one side of the pair is supplied (`-c` or `-e`), auto-fill the other side from this mapping. +- If both are supplied but conflict, stop and confirm before execution. + +4. Approval rule +- Always show the exact planned `atvm_prep.py` command and wait for explicit operator approval before execution. + +## Operational Flow +High-level behavior across modes: + +1. Resolve DGS client and volume/policy IDs. +2. Resolve currently assigned volume for the target client where relevant. +3. If cleanup is required: +- Find datastore from LUN GUID. +- Power off VMs on that datastore. +- Unregister VMs. +- Unmount datastore on ESXi. +- Unassign and optionally delete clone on DGS. +4. Execute mode-specific action: +- create clone and assign (`-A`) +- assign gold and mount (`-M`) +- register only (`-R`) +- power off/unregister only (`-U`) +- rollback and reassign (`-rvgd`) + +## VMware Helpers In Script +Reusable functions implemented in `atvm_prep.py` include: +- datastore discovery from DGS LUN GUID +- force-mount unresolved VMFS volume +- datastore mount/unmount +- VM power-off by datastore membership +- VM unregister by datastore membership +- VMX discovery on datastore for registration + +## Safety Traits +The script intentionally exits on ambiguous or unsafe states, including: +- snapshot not found or duplicate snapshot names +- missing client/volume/datastore +- multiple datastores matching same GUID + +For some operations (`register`, `unregister`, `rollback`) it prompts for explicit confirmation (`[y/n]`) before acting. + +## Known Caveats +1. Default credentials are hardcoded in script arguments. +- Treat these as legacy behavior and prefer supplying credentials explicitly in runtime usage. + +2. `main()` contains this condition: +- `if not is_register or not is_unregister:` +- This is effectively true for almost all normal runs and likely intended as `and`. +- Do not change behavior blindly during operations; patch only with explicit operator approval. + +3. Registration steps in `-A`, `-M`, and `-rvgd` paths are currently commented out. +- Datastore prep/mount occurs, but VM registration is not automatically performed in those paths unless script is modified. + +## Logging +- Script log directory: `/root/atvm_prep/log` +- Main log file: `/root/atvm_prep/log/atvm_prep.log` +- Rotating log handler is enabled in script. + +## Practical Usage Notes +- Prefer dry validation of target LUN/client/snapshot before destructive modes. +- Use `-U` for scoped VM power-off/unregister tied to LUN GUID mapping. +- Use `-R` after datastore mount when VM registration is required and safe. diff --git a/skills/atvm/references/automation-examples.md b/skills/atvm/references/automation-examples.md new file mode 100644 index 0000000..9f33d7a --- /dev/null +++ b/skills/atvm/references/automation-examples.md @@ -0,0 +1,96 @@ +## Examples + +### E2E: Pure iscsi+fc with specific VMs +```bash +python3 cmc-templates.py --template cmc-e2e --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --test_partition --integration_type pure --use_specified_plugin both --specify_vms atvm3-ubuntu18.04 atvm109-w2k12R2; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-e2e-pure-plugin +``` + +### E2E: Infinibox fc with specific VMs +```bash +python3 cmc-templates.py --template cmc-e2e --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --test_partition --integration_type infinibox --use_specified_plugin fc --specify_vms atvm51-redhat6.10 atvm110-w2k16; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-e2e-infinibox-plugin +``` + +### E2E: Regular cutover +```bash +python3 cmc-templates.py --template cmc-e2e --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --test_partition --integration_type pure --use_specified_plugin fc --specify_vms atvm93-oracle7.9 atvm111-w2k19 --regular_cutover; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-e2e-regular-cutover +``` + +### Reboot test +```bash +python3 cmc-templates.py --template cmc-reboot --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --integration_type pure --use_specified_plugin fc --specify_vms atvm37-rocky8.8 atvm112-w2k22 --wait_for_power_on 120; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-reboot +``` + +### SystemOS test +```bash +python3 cmc-templates.py --template cmc-systemOS --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --specify_vms atvm118-oracle9.3 atvm145-w2k25; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-systemOS +``` + +### MigrateOPS test +```bash +python3 cmc-templates.py --template cmc-migrateops --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --integration_type pure --use_specified_plugin fc --specify_vms atvm139-redhat9.5 atvm112-w2k22; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-migrateOPS +``` + +### Compute MigrateOPS: vmware +```bash +python3 cmc-templates.py --template cmc-migrateops-compute-migration --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --vm_platforms vmware --test_partition --specify_vms atvm138-oracle9.4-opt atvm112-w2k22 --set_static_ip_dest; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-computeMigrateOPS-vmware +``` +Default vmware compute-migration options: +- `--ignore_force_shutdown` +- `--vm_platforms vmware` +- `--test_partition` +- `--set_static_ip_dest` + +### Compute MigrateOPS: ovirt +```bash +python3 cmc-templates.py --template cmc-migrateops-compute-migration --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --vm_platforms ovirt --test_partition --specify_vms atvm124-redhat8.8 atvm111-w2k19 --set_static_ip_dest; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-computeMigrateOPS-ovirt +``` + +### Group consistency +```bash +python3 cmc-templates.py --template cmc-group-consistency --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --integration_type pure --use_specified_plugin fc --specify_vms atvm4-ubuntu20.04 atvm112-w2k22 --enable_uuid; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-consistentyGroup +``` + +### Group consistency: single VM +```bash +python3 cmc-templates.py --template cmc-group-consistency --ignore_force_shutdown --test_partition --config_file_path ./cypress.atvm-config-gold.ts --integration_type pure --use_specified_plugin fc --specify_vms atvm71-redhat9.1 --enable_uuid; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-consistencyGroup-redhat9.1 --categorize +``` + +### H2H same platform +```bash +python3 cmc-templates.py --template cmc-h2h-same-platf --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --integration_type pure --use_specified_plugin fc --specify_vms atvm38-rocky9.0 atvm112-w2k22; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-h2hSamePlatform +``` + +### H2H different platform +```bash +python3 cmc-templates.py --template cmc-h2h-diff-platf --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --integration_type pure --use_specified_plugin fc --specify_vms atvm65-redhat8.3 atvm112-w2k22; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name nightly-h2hDifferentPlatform +``` + +### Randomized reboot sanity +```bash +python3 cmc-templates.py --template cmc-reboot --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --test_partition --integration_type pure --use_specified_plugin fc --randomize 1 --exclude_partial_match suse15.0 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 --wait_for_power_on 120; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name sanity-reboot-iscsi +``` + +### Randomized e2e sanity +```bash +python3 cmc-templates.py --template cmc-e2e --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --test_partition --integration_type pure --use_specified_plugin both --randomize 1 --exclude_partial_match suse15.0 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name sanity-e2e +``` + +### Randomized systemOS sanity +```bash +python3 cmc-templates.py --template cmc-systemOS --ignore_force_shutdown --config_file_path ./cypress.atvm-config-gold.ts --randomize 1 --exclude_partial_match suse15.0 fedora34 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name sanity-systemOS +``` diff --git a/skills/atvm/references/automation-guide.md b/skills/atvm/references/automation-guide.md new file mode 100644 index 0000000..c6b98cc --- /dev/null +++ b/skills/atvm/references/automation-guide.md @@ -0,0 +1,378 @@ +# Run ATVM Automation Guide + +This file is guide-only documentation for operating ATVM CMC automation. +Do not put specific run examples here. +For reusable command examples and common option combinations, use `examples.md`. +Treat `examples.md` as reference-only. +Do not assume the operator wants the extra options shown in examples unless they explicitly request them. + +## Purpose +Run ATVM CMC automation tests on the designated automation VM without unintended system or file changes. + +## ATVM Cypress Automation Controller Client +- Hostname: `atvm-cypres-vm-1` +- IP: `192.168.3.190` +- Credentials: source `/home/aw/code/cds/.env.credentials.local` and use `ATVM_CONTROLLER_USER` plus `ATVM_CONTROLLER_PASSWORD` + +## ATVM Target Host Default +- Treat `192.168.3.191` as the default ATVM target host reference. +- For SSH to `192.168.3.191`, ignore host key mismatch by default with `-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null`. +- For Linux SSH access to `192.168.3.191`, source `/home/aw/code/cds/.env.credentials.local` and use `ATVM_TARGET_USER` plus `ATVM_TARGET_PASSWORD` unless the operator explicitly overrides them. +- `ATVM_LINUX_TARGET_HOST`, `ATVM_LINUX_TARGET_USER`, and `ATVM_LINUX_TARGET_PASSWORD` mirror the Linux default values when an OS-specific reference is clearer. +- For Windows guest access to `192.168.3.191`, source `/home/aw/code/cds/.env.credentials.local` and use `ATVM_WINDOWS_TARGET_USER` plus `ATVM_WINDOWS_TARGET_PASSWORD` unless the operator explicitly overrides them. + +## Operating Constraints +- Run only scripts/commands explicitly requested. +- Do not make manual system configuration changes on the client. +- Do not edit client files unless explicitly requested. + +## Operator Preferences +- Do not include Gold Disk identifiers in `--build_name`. +- `--build_name` must not contain spaces; use `-` between words. +- For multiple VMs in same distro, use distro-scoped filtering (`--containsVm`) instead of long explicit VM lists. +- Always include `--ignore_force_shutdown` on `cmc-templates.py` commands unless the operator explicitly asks not to. +- Always include `--test_partition` on `cmc-templates.py` commands unless the operator explicitly asks not to. +- Default plugin-bearing templates to `--use_specified_plugin iscsi` unless the operator explicitly requests a different plugin. +- Do not add plugin or integration-type arguments to `cmc-systemOS`; that template should be planned without `--use_specified_plugin`, without `--integration_type`, and without watcher integration/plugin metadata. +- For `cmc-migrateops-compute-migration`, default to `--set_static_ip_dest` unless the operator explicitly says otherwise. +- For `cmc-migrateops-compute-migration` to VMware, default to `--vm_platforms vmware` unless the operator explicitly says otherwise. +- For ATVM automation runs that involve Windows guests, default `run-sorry-cypress.py` to `--hang_retries 0` unless the operator explicitly says otherwise. +- For `cmc-reboot`, treat `--use_specified_plugin both` as an exception case that requires an extra confirmation. +- When `cmc-reboot` is planned with `--use_specified_plugin both`, warn that FC+iSCSI together may hit a "chicken before the egg" timing problem where iSCSI disks are not attached before mTDI / CMC services start. +- For `cmc-reboot`, prefer `--use_specified_plugin fc` or `--use_specified_plugin iscsi` unless the operator explicitly reconfirms that `both` is really intended after seeing that warning. +- Before preparing a new run, always check whether automation is already running. +- Treat a prior status check as stale once control returns to the operator or a new ATVM request arrives; perform a fresh live controller check at request time instead of relying on the immediately previous result. +- Always report whether automation is currently running. +- If running, ask whether to terminate; terminate only with explicit approval. +- After termination approval, terminate first, then execute the new run command set. +- By default, execute `cmc-templates.py` and `run-sorry-cypress.py` without a pre-run approval gate. +- If the operator explicitly asks to review planned commands first, show them before execution. +- If the operator changes any part of the request before execution, rebuild commands and execute the revised command set. +- Default to watcher-backed execution for every run unless the operator explicitly asks to run without watcher. +- When `--categorize` is used with watcher enabled, treat the watcher as a sequential grouped-run watcher: + - it must post one final Mattermost status per completed categorized group/sub-run + - it must stay active between grouped sub-runs while the parent categorized request is still running + - it must not stop after the first grouped run simply because one grouped run completed + - if the child build id label does not match the actual host/spec being executed, report the grouped run using the inferred host-based group instead of the raw child build id label + - it must not wait and replace those with one single parent-only post +- After execution, report immediate success/failure only. +- After execution, include the exact executed `cmc-templates.py` and `run-sorry-cypress.py` commands in the response. +- Do not include expected, harmless `systemctl reset-failed ... unit not loaded` output in routine run-start confirmations. +- Mention `reset-failed` output only when it prevents watcher startup or becomes relevant to debugging. +- Do not actively monitor completion unless explicitly requested. +- If monitoring is requested, allow long runtime windows (15-30+ minutes) and continue until completion unless operator instructs otherwise. +- Report command errors immediately. +- `sshpass` may be used where password-based SSH automation is required. +- Treat runner hang-kill events (`Sending SIGKILL ... due to no change` / `Max hang retries reached`) as explicit `FAILED` outcomes, not `RUNNING` or ambiguous termination. +- For manual `run-sorry-cypress.py` execution, treat `ATVM_HANG_FAIL ...` log markers and `/tmp/atvm-runner-state-.json` terminal state files as the source of truth for hang-failure terminal status. + +## Core Scripts +- Template prep: `/root/cdc-e2e-cyp-12.17.4/cmc-templates.py` +- Test execution: `./run-sorry-cypress.py` +- Detailed host-level test artifacts: `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter` + +## Detailed Test Artifacts +- Use `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter` on the automation controller for detailed per-host test evidence. +- Reporter subdirectories of interest: + - `logs/` + - per-host text and JSON logs for the executed tests + - `xml/` + - machine result XML files and the final `check-xml-files.ts` bookkeeping output + - `mochawesome/` + - per-run HTML reports +- When a machine fails, use the matching `logs/` entry first to capture the detailed failure context for that host. +- Apply the failed-host detail recovery path to every ATVM template type, not just reboot. +- For any failed host, recover detail in this order when available: + - consolidated run log + - matching `mochawesome` HTML + - structured reporter artifacts such as per-host JSON or XML + - text reporter artifacts +- When reconstructing historical status, prefer `cmcReporter` artifacts over less-specific runner output because they preserve per-host results after the live run has ended. +- Do not treat the existence of a per-host reporter artifact by itself as proof that the host passed. +- For categorized grouped recovery, prefer the matching per-host reporter JSON or mochawesome result and carry through the real `failures`, `pending`, and failure message instead of assuming `PASS completed`. +- If grouped XML only contains `check-xml-files.ts`, cross-check the grouped result against the per-host reporter artifacts before posting or repeating status for that grouped sub-run. +- Do not report a categorized grouped sub-run as `PASS` from watcher `host_results`, grouped XML, or a lone `check-xml-files.ts` result by itself. +- Before reporting a categorized grouped sub-run as `PASS`, confirm that the matching child batch also passed in the live launch log or the final `Cloud Run Finished` summary for that child run. +- Treat saved watcher state under `/var/lib/atvm-run-watcher//state.json` as cached status only. +- For completed-run verification, confirm in this order: + - launch log under `/tmp/.launch.log` + - matching `cmcReporter` artifacts + - `Cloud Run Finished` summary and Currents URL + - saved watcher state only as a comparison layer +- If saved watcher state disagrees with the launch log or with a replay of the exact artifacts through the current watcher code, treat the saved state as stale and do not use it as the reported result. +- Never confirm a completed run from `state.json` alone. + +Typical sequence: +1. Build the exact `cmc-templates.py` and `run-sorry-cypress.py` commands for the request. +2. Run `cmc-templates.py` with the requested options. +3. Wait for `cmc-templates.py` to fully finish and confirm success. +4. Verify the generated `.ts` files and the config `specPattern` include every requested VM before starting the runner. +5. By default, use watcher-backed execution unless the operator explicitly asks not to. +6. For watcher-backed runs, make sure the controller's deployed watcher code is the intended version before relying on its posts. +7. For watcher-backed runs, build the watcher-start command so it automatically includes the exact `cmc-templates.py` command via `--template-command` and the exact `run-sorry-cypress.py` command via `--runner-command`. +8. For watcher-backed runs, prefer the controller-local `atvm-runner@...` systemd service instead of detached SSH background launch patterns for `run-sorry-cypress.py`. +9. For watcher-backed runs, start the watcher before launching the runner service. +10. Start the runner with the matching config and build name. +11. Report immediate start success/failure and include the exact executed template and runner commands. + +Completed-run verification sequence: +1. Read the launch log for the build. +2. Inspect the matching reporter artifacts for the relevant host(s). +3. Use the `Cloud Run Finished` summary and Currents URL as the final parent-run check when present. +4. Compare that result against saved watcher state. +5. If there is any disagreement, replay the exact artifacts through the current watcher code in an isolated temp state directory before confirming the result. +6. For categorized runs, do not let a `check-xml-files.ts` child result override a failing child batch shown in the launch log or `Cloud Run Finished` summary. + +## Config File / Gold Disk Mapping +- `cypress.atvm-config-gold.ts` -> Gold Disk 1 +- `cypress.atvm-config-gold-2.ts` -> Gold Disk 2 +- Additional numbered config variants map to corresponding Gold Disks. +- Do not default to `cypress.atvm-config.ts`. +- Unless the operator explicitly requests another config, use a config file with `gold` in the filename. +- If the operator-specified config file is missing, stop immediately and report the missing file. +- Do not search for substitute ATVM config files and do not switch to another config unless the operator explicitly instructs it. +- Treat `AutomatedTest-VMBootImg-Gold` as `gold` and `AutomatedTest-VMBootImg-Gold-2` as `gold-2`. +- Use live vCenter inventory as the source of truth for current VM membership on those datastores. +- Query vCenter datastore membership at request time when selecting `gold` vs `gold-2`; do not maintain or rely on a repo-side live reference file for this decision. +- When the operator asks to inventory or show the contents of `AutomatedTest-VMBootImg-Gold` and `AutomatedTest-VMBootImg-Gold-2`, return hostname-only VM lists unless the operator explicitly asks for more detail. +- When the operator provides an explicit VM list, check vCenter placement for every requested VM before choosing the config file. +- Before presenting any ATVM run commands for an explicit VM-list request, tell the operator that the next step is a live vCenter placement check for the requested VMs and that the result will determine whether the run must use `gold` or `gold-2`. +- For vCenter inspection and placement checks, prefer `govc` and raw vCenter REST calls when they are available before reaching for alternate wrappers. +- Never delete, destroy, remove from inventory, or unregister any ATVM VM from vCenter. +- Treat a vCenter VM as protected ATVM inventory if its name starts with `atvm`, if it resides on an `AutomatedTest-*` ATVM datastore, or if it is an ATVM platform/controller VM, even when it is not listed in `inventory/vm-inventory.md`. +- Preserve ATVM vCenter VM inventory records even when power, maintenance, datastore, cleanup, or prep work is requested. +- For cleanup workflows, only power off, detach/unmount supporting resources, or report the needed manual action; do not remove ATVM VM inventory records. +- For `govc`-based placement checks, use `govc vm.info -json ` and parse the lowercase JSON keys such as `virtualMachines` and `datastore`. +- Resolve each returned datastore managed-object reference to a datastore name with `govc object.collect -s name` before deciding between `gold` and `gold-2`. +- Ignore non-boot helper datastores such as install ISO attachments when applying the `gold` vs `gold-2` rule; base the family decision on the ATVM boot datastore membership. +- If every requested VM is on `AutomatedTest-VMBootImg-Gold`, plan the run with the `gold` config. +- If every requested VM is on `AutomatedTest-VMBootImg-Gold-2`, plan the run with the `gold-2` config. +- If the requested VM set spans both `AutomatedTest-VMBootImg-Gold` and `AutomatedTest-VMBootImg-Gold-2`, stop immediately and do not prepare or run the test. +- For a mixed-datastore request, report it as a discrepancy, list which requested VMs are on `AutomatedTest-VMBootImg-Gold` and which are on `AutomatedTest-VMBootImg-Gold-2`, tell the operator they need to correct the list, and ask whether they want the full VM inventories for both datastores so they can adjust the request. +- Do not run an ATVM test against a mixed set of VMs from both `AutomatedTest-VMBootImg-Gold` and `AutomatedTest-VMBootImg-Gold-2`. + +## Available Templates +- `cmc-e2e` +- `cmc-group-consistency` +- `cmc-h2h-diff-platf` +- `cmc-h2h-same-platf` +- `cmc-migrateops` +- `cmc-migrateops-compute-migration` +- `cmc-reboot` +- `cmc-systemOS` + +## Command Pattern +```bash +python3 cmc-templates.py --template