cds-ai/atvm/AGENTS.md

# ATVM AGENTS Guide

This file defines how to operate and maintain the ATVM workspace in `/home/aw/code/cds/atvm`.

## Workspace Layout
- `README.md`
  - human entry point for the folder
- `scripts/`
  - executable workflow assets
- `docs/setup/`
  - setup/bootstrap procedure and run learnings
- `docs/automation/`
  - ATVM Cypress automation procedure, examples, and run learnings
- `docs/workflow/`
  - shared conventions for how the docs are maintained
- `inventory/`
  - environment reference, credentials, IP allocations, and inventory indexes
- `archive/imported-notes/`
  - preserved original long-form source material

## Authoritative Sources
- Setup/bootstrap procedure:
  - `docs/setup/guide.md`
- Setup/bootstrap learnings:
  - `docs/setup/run-learnings.md`
- Automation execution procedure:
  - `docs/automation/guide.md`
- Automation command examples:
  - `docs/automation/examples.md`
- Automation run learnings:
  - `docs/automation/run-learnings.md`
- Workspace conventions:
  - `docs/workflow/conventions.md`

## Setup Track Defaults
- ATVM static IP target: `192.168.3.191/22`
- Gateway: `192.168.0.1`
- DNS: `8.8.8.8`, `8.8.4.4`
- Default Linux setup credential source: `/home/aw/code/cds/.env.credentials.local` via `ATVM_TARGET_USER` and `ATVM_TARGET_PASSWORD`
- Client log file: `atvm_setup_script.log`
- Treat `192.168.3.191` as the default ATVM target host reference.
- For SSH to `192.168.3.191`, ignore host key mismatch by default with `-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null`.
- For Linux SSH to `192.168.3.191`, source `/home/aw/code/cds/.env.credentials.local` and use `ATVM_TARGET_USER` plus `ATVM_TARGET_PASSWORD` unless explicitly overridden.
- Use `ATVM_WINDOWS_TARGET_USER` plus `ATVM_WINDOWS_TARGET_PASSWORD` for Windows guest access to the same ATVM target IP unless explicitly overridden.

## Automation Track Defaults
- Controller host: `atvm-cypres-vm-1`
- Controller IP: `192.168.3.190`
- Controller credential source: `/home/aw/code/cds/.env.credentials.local` via `ATVM_CONTROLLER_USER` and `ATVM_CONTROLLER_PASSWORD`
- Detailed test artifact root on controller: `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter`
- Default Mattermost status destination config: `/home/aw/code/cds/.env.credentials.local`
- Default plugin-bearing template plugin: `--use_specified_plugin iscsi`
- Always include `--ignore_force_shutdown` unless explicitly told not to.
- Always include `--test_partition` unless explicitly told not to.
- For `cmc-migrateops-compute-migration` to VMware, default to `--vm_platforms vmware` and `--set_static_ip_dest` unless explicitly told otherwise.
- Default config family: `gold`
- Treat `cmc-systemOS` as not using plugin or integration-type arguments. Do not auto-add `--use_specified_plugin`, `--integration_type`, or watcher integration/plugin metadata for that template.
- Do not auto-add the maintained `--exclude_partial_match` blacklist when the operator explicitly targets named VMs with `--specify_vms`.
- Even for explicit `--specify_vms` requests, first check whether any requested VM is on the maintained blacklist and stop if it is.

## Required Operating Rules
- Never run setup without operator-provided `--expected-ip` and `--expected-hostname`.
- Keep static IP configuration as the final setup step.
- Before any automation run, always check whether automation is already running.
- Always show exact planned ATVM commands before execution.
- Never execute setup or automation commands that require approval until the operator explicitly approves them.
- For ATVM run approvals, treat `approve` as run-with-watcher and `approve without watcher` as run-without-watcher.
- Treat git/commit requests as a separate approval gate.
- Follow `/home/aw/code/cds/atvm/git-guide.md` for ATVM git command drafting and commit-request handling, including the rule that the controller `e2e cypress` repo behavior only applies when the operator explicitly asks for the `e2e cypress` repo or a close variation, the rule to draft plain git commands rather than SSH-wrapped controller login commands unless explicitly requested, the SSH-prefixed push example requirement for that repo, and the rule that phrases such as `create me a git`, `create a git`, `create a git description`, `make me a git`, `make a git`, `make me a git description`, `create me a git description`, and close variations are prepare-only until the operator explicitly approves the displayed commit command.
- Never execute `git push` from the assistant for this workspace.
- After creating a local commit for the explicitly requested `e2e cypress` controller repo, stop and give the operator the exact manual SSH-prefixed push command reference from `git-guide.md`, unless they explicitly ask for a different remote or branch.
- Do not treat `approve` after a commit as permission to push; pushing requires separate explicit wording and still remains manual-reference-only unless the operator explicitly overrides this workspace rule.
- After `cmc-templates.py`, always verify that the generated spec files and the config `specPattern` still include every requested VM before starting `run-sorry-cypress.py`.
- If any requested VM is missing after template generation, stop and report the mismatch instead of launching the runner.
- Do not infer plugin behavior from the mere presence of plugin-specific strings or code blocks in a generated spec file.
- For plugin selection questions, determine expected behavior from the template/runtime gate such as `Cypress.env(...)` conditions or other execution guards, and only treat it as a mismatch if the runtime logic would execute the wrong plugin path.
- For `cmc-reboot`, treat `--use_specified_plugin both` as a separate confirmation gate, not a normal plugin selection.
- When a planned reboot command uses `--use_specified_plugin both`, warn that FC+iSCSI together may have a "chicken before the egg" timing issue where iSCSI disks are not attached before mTDI / CMC services start, and ask the operator to confirm that `both` is really intended.
- Unless the operator explicitly reconfirms `both` for `cmc-reboot`, prefer only `fc` or only `iscsi`, not both.
- When the watcher is requested, start the watcher before `run-sorry-cypress.py`.
- When the watcher is requested, build the watcher-start command so it automatically includes the exact approved `cmc-templates.py` command via `--template-command` and the exact approved `run-sorry-cypress.py` command via `--runner-command`; the operator should not need to restate them separately.
- When watcher-backed execution is used, prefer the controller-local `atvm-runner@...` systemd service over detached SSH background launch patterns for `run-sorry-cypress.py`.
- Do not start the runner before the watcher, because the watcher helper clears stale `/tmp/<build-name>.log` and can delete the fresh live runner log if the runner starts first.
- Prefer the combined `start-atvm-run.sh` wrapper when starting both services so watcher and runner are never launched in parallel against the same `/tmp/<build-name>.log`.
- For host-level test detail and failed-test investigation, use `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter`, especially `logs/`, `xml/`, and `mochawesome/`.
- Apply failed-host detail recovery consistently for every ATVM template run, not just `cmc-reboot`.
- For any failed ATVM host, recover failure detail in this order when available: consolidated run log, `mochawesome`, structured reporter artifacts (`json`/`xml`), then text reporter artifacts.
- Keep the `HOSTS` detail column compact with the failing step plus a short error summary only.
- Put richer per-host error excerpts in a dedicated `FAILURE NOTES:` section, and reserve `NOTES:` for non-failure context such as the template command, Currents URL, and operator-facing caveats.
- When reporting `TEST FLOW:` for an ATVM run, prefer the numbered steps extracted from the generated spec for that exact run.
- If the generated spec exists, do not rely on a static template flow list for `TEST FLOW:`.
- Only fall back to template-level or static flow definitions when the generated spec cannot be located or parsed.
- Treat `/var/lib/atvm-run-watcher/<build>/state.json` as cached watcher output, not the source of truth for a completed-run confirmation.
- Before confirming a completed ATVM run status, verify in this order: live launch log, matching reporter artifacts, `Cloud Run Finished` summary / Currents URL, then compare against saved watcher state.
- If saved watcher state disagrees with the launch log or a replay of the exact artifacts through the current watcher code, treat the saved state as stale and do not report from it.
- Never confirm a completed ATVM run from `state.json` alone.
- For categorized runs, never report a grouped sub-run as `PASS` from watcher `host_results`, grouped XML, or a lone `check-xml-files.ts` result by itself.
- Before reporting a categorized grouped sub-run as `PASS`, confirm that the matching child batch also passed in the live launch log or the final `Cloud Run Finished` summary for that child run.
- If the operator asks for ATVM run status without mentioning Mattermost, respond locally only and do not post externally.
- If the operator asks to send ATVM run status to Mattermost, use `MATTERMOST_ATVM_WEBHOOK` and `MATTERMOST_ATVM_CHANNEL` from `/home/aw/code/cds/.env.credentials.local` by default and send the final status only after the run has fully completed, whether the run passed or failed.
- For vCenter VM snapshot requests, default the snapshot name to `VM Snapshot [m/d/yyyy], [h:mm:ss AM/PM]` in the local `America/New_York` timezone unless the operator explicitly requests a different name.
- Do not call out expected, harmless `systemctl reset-failed ... unit not loaded` output in routine run updates; mention it only if it blocks startup or matters for debugging.
- Treat `docs/automation/examples.md` as reference-only, not default operator intent.
- Put reusable workflow rules in `guide.md` files.
- Put dated lessons only in `run-learnings.md` files.
- Keep durable environment reference in `inventory/`.
- Preserve imported long-form notes in `archive/imported-notes/`; do not treat them as the primary runbook.

## Maintenance Rules
- When changing workflow behavior, update the corresponding `guide.md`.
- When adding a reusable command pattern, update `docs/automation/examples.md`.
- When a run produces a new lesson, update the appropriate `run-learnings.md`.
- Keep filesystem paths in docs aligned with the actual repo layout.
- Do not remove detailed inventory or credential information from this workspace unless explicitly instructed.