cds-ai/atvm/AGENTS.md

ATVM AGENTS Guide

This file defines how to operate and maintain the ATVM workspace in /home/aw/code/cds/atvm.

Workspace Layout

• README.md
  • human entry point for the folder
• scripts/
  • executable workflow assets
• docs/setup/
  • setup/bootstrap procedure and run learnings
• docs/automation/
  • ATVM Cypress automation procedure, examples, and run learnings
• docs/workflow/
  • shared conventions for how the docs are maintained
• inventory/
  • environment reference, credentials, IP allocations, and inventory indexes
• archive/imported-notes/
  • preserved original long-form source material

Authoritative Sources

• Setup/bootstrap procedure:
  • docs/setup/guide.md
• Setup/bootstrap learnings:
  • docs/setup/run-learnings.md
• Automation execution procedure:
  • docs/automation/guide.md
• Automation command examples:
  • docs/automation/examples.md
• Automation run learnings:
  • docs/automation/run-learnings.md
• Workspace conventions:
  • docs/workflow/conventions.md

Setup Track Defaults

• ATVM static IP target: 192.168.3.191/22
• Gateway: 192.168.0.1
• DNS: 8.8.8.8, 8.8.4.4
• Default Linux setup credential source: /home/aw/code/cds/.env.credentials.local via ATVM_TARGET_USER and ATVM_TARGET_PASSWORD
• Client log file: atvm_setup_script.log
• Treat 192.168.3.191 as the default ATVM target host reference.
• For SSH to 192.168.3.191, ignore host key mismatch by default with -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null.
• For Linux SSH to 192.168.3.191, source /home/aw/code/cds/.env.credentials.local and use ATVM_TARGET_USER plus ATVM_TARGET_PASSWORD unless explicitly overridden.
• Use ATVM_WINDOWS_TARGET_USER plus ATVM_WINDOWS_TARGET_PASSWORD for Windows guest access to the same ATVM target IP unless explicitly overridden.

Automation Track Defaults

• Controller host: atvm-cypres-vm-1
• Controller IP: 192.168.3.190
• Controller credential source: /home/aw/code/cds/.env.credentials.local via ATVM_CONTROLLER_USER and ATVM_CONTROLLER_PASSWORD
• Detailed test artifact root on controller: /root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter
• For vCenter inspection, prefer govc and raw vCenter REST calls before inventing or depending on higher-level wrappers.
• Default Mattermost status destination config: /home/aw/code/cds/.env.credentials.local
• Default plugin-bearing template plugin: --use_specified_plugin iscsi
• Always include --ignore_force_shutdown unless explicitly told not to.
• Always include --test_partition unless explicitly told not to.
• For cmc-migrateops-compute-migration to VMware, default to --vm_platforms vmware and --set_static_ip_dest unless explicitly told otherwise.
• For ATVM automation runs that involve Windows guests, default the runner command to --hang_retries 0 unless explicitly told otherwise.
• Default config family: gold
• Treat cmc-systemOS as not using plugin or integration-type arguments. Do not auto-add --use_specified_plugin, --integration_type, or watcher integration/plugin metadata for that template.
• Do not auto-add the maintained --exclude_partial_match blacklist when the operator explicitly targets named VMs with --specify_vms.
• Even for explicit --specify_vms requests, first check whether any requested VM is on the maintained blacklist and stop if it is.

Required Operating Rules

• Never run setup without operator-provided --expected-ip and --expected-hostname.
• Keep static IP configuration as the final setup step.
• Before any automation run, always check whether automation is already running.
• Treat every new ATVM run request as requiring a fresh live controller running-state check; never rely on the immediately previous status check when deciding whether a new run may start.
• For ATVM automation runs, execute without a pre-run approval gate unless the operator explicitly asks to review commands first.
• Default all ATVM automation runs to watcher-backed execution unless the operator explicitly says to run without watcher.
• After starting an ATVM automation run, report the exact executed cmc-templates.py and run-sorry-cypress.py commands.
• Treat git/commit requests as a separate approval gate.
• Follow /home/aw/code/cds/atvm/git-guide.md for ATVM git command drafting and commit-request handling, including the rule that the controller e2e cypress repo behavior only applies when the operator explicitly asks for the e2e cypress repo or a close variation, the rule to draft plain git commands rather than SSH-wrapped controller login commands unless explicitly requested, the SSH-prefixed push example requirement for that repo, and the rule that phrases such as create me a git, create a git, create a git description, make me a git, make a git, make me a git description, create me a git description, and close variations are prepare-only until the operator explicitly approves the displayed commit command.
• Never execute git push from the assistant for this workspace.
• After creating a local commit for the explicitly requested e2e cypress controller repo, stop and give the operator the exact manual SSH-prefixed push command reference from git-guide.md, unless they explicitly ask for a different remote or branch.
• Do not treat approve after a commit as permission to push; pushing requires separate explicit wording and still remains manual-reference-only unless the operator explicitly overrides this workspace rule.
• After cmc-templates.py, always verify that the generated spec files and the config specPattern still include every requested VM before starting run-sorry-cypress.py.
• If any requested VM is missing after template generation, stop and report the mismatch instead of launching the runner.
• Do not infer plugin behavior from the mere presence of plugin-specific strings or code blocks in a generated spec file.
• For plugin selection questions, determine expected behavior from the template/runtime gate such as Cypress.env(...) conditions or other execution guards, and only treat it as a mismatch if the runtime logic would execute the wrong plugin path.
• For cmc-reboot, treat --use_specified_plugin both as a separate confirmation gate, not a normal plugin selection.
• When a planned reboot command uses --use_specified_plugin both, warn that FC+iSCSI together may have a "chicken before the egg" timing issue where iSCSI disks are not attached before mTDI / CMC services start, and ask the operator to confirm that both is really intended.
• Unless the operator explicitly reconfirms both for cmc-reboot, prefer only fc or only iscsi, not both.
• For default watcher-backed runs, start the watcher before run-sorry-cypress.py.
• For default watcher-backed runs, build the watcher-start command so it automatically includes the exact cmc-templates.py command via --template-command and the exact run-sorry-cypress.py command via --runner-command; the operator should not need to restate them separately.
• When watcher-backed execution is used, prefer the controller-local atvm-runner@... systemd service over detached SSH background launch patterns for run-sorry-cypress.py.
• Do not start the runner before the watcher, because the watcher helper clears stale /tmp/<build-name>.log and can delete the fresh live runner log if the runner starts first.
• Prefer the combined start-atvm-run.sh wrapper when starting both services so watcher and runner are never launched in parallel against the same /tmp/<build-name>.log.
• For host-level test detail and failed-test investigation, use /root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter, especially logs/, xml/, and mochawesome/.
• Apply failed-host detail recovery consistently for every ATVM template run, not just cmc-reboot.
• For any failed ATVM host, recover failure detail in this order when available: consolidated run log, mochawesome, structured reporter artifacts (json/xml), then text reporter artifacts.
• Keep the HOSTS detail column compact with the failing step plus a short error summary only.
• Put richer per-host error excerpts in a dedicated FAILURE NOTES: section, and reserve NOTES: for non-failure context such as the template command, Currents URL, and operator-facing caveats.
• When reporting TEST FLOW: for an ATVM run, prefer the numbered steps extracted from the generated spec for that exact run.
• If the generated spec exists, do not rely on a static template flow list for TEST FLOW:.
• Only fall back to template-level or static flow definitions when the generated spec cannot be located or parsed.
• Treat /var/lib/atvm-run-watcher/<build>/state.json as cached watcher output, not the source of truth for a completed-run confirmation.
• Before confirming a completed ATVM run status, verify in this order: live launch log, matching reporter artifacts, Cloud Run Finished summary / Currents URL, then compare against saved watcher state.
• If saved watcher state disagrees with the launch log or a replay of the exact artifacts through the current watcher code, treat the saved state as stale and do not report from it.
• Never confirm a completed ATVM run from state.json alone.
• For categorized runs, never report a grouped sub-run as PASS from watcher host_results, grouped XML, or a lone check-xml-files.ts result by itself.
• Before reporting a categorized grouped sub-run as PASS, confirm that the matching child batch also passed in the live launch log or the final Cloud Run Finished summary for that child run.
• If the operator asks for ATVM run status without mentioning Mattermost, respond locally only and do not post externally.
• If the operator asks to send ATVM run status to Mattermost, use MATTERMOST_ATVM_WEBHOOK and MATTERMOST_ATVM_CHANNEL from /home/aw/code/cds/.env.credentials.local by default and send the final status only after the run has fully completed, whether the run passed or failed.
• For vCenter VM snapshot requests, default the snapshot name to VM Snapshot [mm/dd/yyyy:hh:mm:ss AM/PM] in the local America/New_York timezone unless the operator explicitly requests a different name.
• For VM power, shutdown, startup, snapshot, or maintenance requests, operate only on the exact VM names provided by the operator.
• If any requested VM name is missing or not found, stop immediately and report the missing names; do not infer, substitute, or search for replacement VMs unless the operator explicitly requests a discovery/mapping step.
• For atvm_prep requests, infer common option values from operator shorthand when intent is clear.
• For any atvm_prep execution request, always present the exact planned atvm_prep.py command first and wait for explicit operator approval before running it.
• Always execute atvm_prep.py on the ATVM Cypress controller host 192.168.3.190 (atvm-cypres-vm-1).
• atvm_prep datastore shorthand mapping for -n:
  • Gold -> AutomatedTest-VMBootImg-Gold
  • Gold2 -> AutomatedTest-VMBootImg-Gold-2
  • ComputeMigration (or compute migration) -> AutomatedTest-VMBootImgComputeMigration-Gold
• atvm_prep client/ESXi pair mapping:
  • CDS1-ESX165 <-> 192.168.1.165
  • CDS1-ESX166 <-> 192.168.1.166
• When an atvm_prep request specifies only one side of the client/ESXi pair (-c or -e), auto-fill the other side from the mapping.
• If both -c and -e are provided but conflict with the mapping, stop and ask the operator to confirm the intended pair before any execution.
• Do not call out expected, harmless systemctl reset-failed ... unit not loaded output in routine run updates; mention it only if it blocks startup or matters for debugging.
• Treat docs/automation/examples.md as reference-only, not default operator intent.
• Put reusable workflow rules in guide.md files.
• Put dated lessons only in run-learnings.md files.
• Keep durable environment reference in inventory/.
• Preserve imported long-form notes in archive/imported-notes/; do not treat them as the primary runbook.

Maintenance Rules

• When changing workflow behavior, update the corresponding guide.md.
• When adding a reusable command pattern, update docs/automation/examples.md.
• When a run produces a new lesson, update the appropriate run-learnings.md.
• Keep filesystem paths in docs aligned with the actual repo layout.
• Do not remove detailed inventory or credential information from this workspace unless explicitly instructed.