commit 93b6d7acb8746ad3e209454ed25eef081925885a Author: anthony.wen Date: Wed Mar 11 15:19:25 2026 -0400 Initial commit diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..80dd262 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +log/ diff --git a/atvm/AGENTS.md b/atvm/AGENTS.md new file mode 100644 index 0000000..c7000a5 --- /dev/null +++ b/atvm/AGENTS.md @@ -0,0 +1,162 @@ +# ATVM AGENTS Guide + +This file defines how to operate and maintain the ATVM folder workflows. +It is rebuilt from current files in `/home/aw/code/cds/atvm`. + +## Scope +Two operational tracks exist in this folder: +- Setup/bootstrap track: + - `atvm-setup-script.sh` + - `run-atvm-setup-and-collect-log.sh` + - `atvm-setup-script-guide.md` + - `atvm-setup-script-runs.md` +- Cypress automation track: + - `atvm-automation-guide.md` + - `atvm-automation-examples.md` + - `atvm-automation-runs.md` + +Reference/inventory material: +- `cypress-automation-for-cmc.md` +- `cypress-automation-for-cmc.md:Zone.Identifier` + +## File Roles +- `*-guide.md` files: + - Guide-only procedures, rules, defaults, and checklists. + - No dated or one-off run examples. +- `*-runs.md` files: + - Run-specific learnings only when a run introduces new information. + - No routine/no-change run logs. +- `*-examples.md` files: + - Reusable command examples and commonly used option combinations. + - Keep generic; avoid dated one-off run outcomes. + +## Setup Track: Required Behavior +Use `atvm-setup-script-guide.md` as the procedure source and keep behavior aligned with `atvm-setup-script.sh`. + +### Safety-Critical Rules +1. Never run setup without operator-provided `--expected-ip` and `--expected-hostname`. +2. Never infer expected hostname from target host output. +3. Stop immediately on hostname mismatch or expected-IP-not-assigned. +4. Keep static IP configuration as a final step to avoid mid-run connection loss. + +### Canonical Setup Order +1. Parse args. +2. Validate host identity. +3. Check sudo/privileges. +4. Fix repositories. +5. Configure Ubuntu root SSH/password workflow (Ubuntu only). +6. Install sudo if needed. +7. Configure Oracle default non-UEK kernel (Oracle Linux only). +8. Disable Ubuntu auto-upgrades (Ubuntu only). +9. Run package cleanup/install. +10. Disable SELinux (RHEL-family). +11. Configure static IP. +12. Print summary. +13. Reboot + post-reboot SELinux verifier when applicable. +14. Keep client on until controller log copy + SHA256 verification completes. +15. Power off only after verified success and no real error log lines. + +### Setup Defaults +- ATVM static IP target: `192.168.3.191/22` +- Gateway: `192.168.0.1` +- DNS: `8.8.8.8`, `8.8.4.4` +- Ubuntu root SSH workflow credential in docs/script: `root / cdsi2012` +- Client log file: `atvm_setup_script.log` (typically `/root/atvm_setup_script.log` when run as root) + +### Setup Controller Wrapper Rules +- Wrapper supports: + - run-and-collect (default) + - `--collect-after-complete` +- `run-and-collect` requires env vars: + - `EXPECTED_IP_ARG` + - `EXPECTED_HOSTNAME_ARG` +- Wrapper validates success marker and SHA256 before success. +- Wrapper powers off only when log has no lines matching `^\[ERROR\]`. + +## Cypress Automation Track: Required Behavior +Use `atvm-automation-guide.md` as the execution source. +Use `atvm-automation-examples.md` as the common options/command reference. + +### Controller Client +- Hostname: `atvm-cypres-vm-1` +- IP: `192.168.3.190` +- Credentials: `root / atvmcdsi2012` + +### Mandatory Run Control +1. Before planning a new run, check for active automation processes. +2. Report running/not-running status. +3. If running, ask before termination; terminate only with explicit approval. +4. Always show exact planned command(s) before execution. +5. Execute only after explicit approval. +6. If monitoring is not requested, report immediate command success/failure and any errors. +7. Monitor completion only when explicitly requested by the operator. +8. For monitored runs, allow long runtime windows (15-30+ minutes or longer) and continue until completion unless operator instructs otherwise. +9. Do not terminate monitored runs unless the operator explicitly instructs termination. + +### Status Request Format +When the operator asks for run status, report in this order: +1. Heading/title using the run `build_name`. +2. Completed machines with pass/fail state for each machine. +3. Skipped machines with reason. +4. Remaining machines still to run. +5. Summary counts for finished, passed, failed, and skipped machines. +6. Timing details: + - start time + - end time if complete + - total run time if complete, or elapsed run time if still running + - quickest completed test runtime + - longest completed test runtime + - average completed test runtime +7. Estimated completion time. + +Status details: +- Use the live run log on the automation VM when available. +- Use the run `build_name` as the heading/title when available. +- Show blacklisted machines under skipped machines when they are part of the requested scope. +- Show in-progress machines under remaining machines as `RUNNING`. +- Show not-yet-started machines as `NOT STARTED`. +- Use completed spec results already recorded in the log to determine machine pass/fail state. +- For failed machines, include the failure reason from the run log in the status output. +- Include start time in status output when it can be derived from the log. +- Include end time and total runtime for completed runs, or elapsed runtime for active runs. +- Include quickest completed test runtime, longest completed test runtime, and average completed test runtime under timing details when they can be derived from the log. + +### Automation Blacklist +Always exclude these machines with `--exclude_partial_match` when building ATVM automation commands. + +CMC install blacklist (`BLACKLISTED: CMC INSTALL - CAN'T COMPILE`): +- `atvm6-centos6.0` +- `atvm41-redhat6.0` +- `atvm73-oracle6.0` + +Support-request blacklist (`BLACKLISTED: SUPPORT REQUEST - WAITING`): +- `atvm113-debian9.0.0` +- `atvm115-debian9.1.0` +- `atvm116-debian9.2.0` +- `atvm156-debian9.3.0` + +Re-create blacklist: +- `atvm157-debian13.0.0` + +### Operator Preferences +- Do not include Gold Disk IDs in `--build_name`. +- `--build_name` must not contain spaces; use `-` between words. +- Prefer distro-scoped filtering (for example `--containsVm redhat9`) when possible. + +## Update Policy (Both Tracks) +After each run: +- Update corresponding `*-guide.md` only if workflow/rules/default behavior changed. +- Update corresponding `*-examples.md` when common command patterns/options change. +- Update corresponding `*-runs.md` only if the run produced new learning. + +## Path and Naming Consistency Note +Current repo filenames use hyphen style, but some script text/defaults still show underscore-style paths (for example `atvm_setup_script.sh`, `run_atvm_setup_and_collect_log.sh`, `/home/aw/code/atvm`). + +When operating: +1. Use actual filesystem paths in this repo first (`/home/aw/code/cds/atvm/...`). +2. If script defaults are used, verify they match existing files before execution. +3. If changing path conventions, update scripts and guides in the same change. + +## Non-Goals +- Do not treat `cypress-automation-for-cmc.md` as executable runbook logic. +- Do not record secrets/tokens into new guide or runs entries. diff --git a/atvm/atvm-automation-examples.md b/atvm/atvm-automation-examples.md new file mode 100644 index 0000000..3d76536 --- /dev/null +++ b/atvm/atvm-automation-examples.md @@ -0,0 +1,97 @@ +## Examples + +- `--build_name` values must not include spaces; use `-` between words. +- Add the maintained blacklist to `--exclude_partial_match` for runs that use broad selection or randomization. +- Maintained blacklist: + - `atvm6-centos6.0` + - `atvm41-redhat6.0` + - `atvm73-oracle6.0` + - `atvm113-debian9.0.0` + - `atvm115-debian9.1.0` + - `atvm116-debian9.2.0` + - `atvm156-debian9.3.0` + - `atvm157-debian13.0.0` + +### E2E: Pure iscsi+fc with specific VMs +```bash +python3 cmc-templates.py --template cmc-e2e --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --test_partition --integration_type pure --use_specified_plugin both --specify_vms atvm3-ubuntu18.04 atvm109-w2k12R2; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name nightly-e2e-pure-plugin +``` + +### E2E: Infinibox fc with specific VMs +```bash +python3 cmc-templates.py --template cmc-e2e --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --test_partition --integration_type infinibox --use_specified_plugin fc --specify_vms atvm51-redhat6.10 atvm110-w2k16; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name nightly-e2e-infinibox-plugin +``` + +### E2E: Regular cutover +```bash +python3 cmc-templates.py --template cmc-e2e --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --test_partition --integration_type pure --use_specified_plugin fc --specify_vms atvm93-oracle7.9 atvm111-w2k19 --regular_cutover; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name nightly-e2e-regular-cutover +``` + +### Reboot test +```bash +python3 cmc-templates.py --template cmc-reboot --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --integration_type pure --use_specified_plugin fc --specify_vms atvm37-rocky8.8 atvm112-w2k22 --wait_for_power_on 120; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name nightly-reboot +``` + +### SystemOS test +```bash +python3 cmc-templates.py --template cmc-systemOS --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --specify_vms atvm118-oracle9.3 atvm145-w2k25; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name nightly-systemOS +``` + +### MigrateOPS test +```bash +python3 cmc-templates.py --template cmc-migrateops --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --integration_type pure --use_specified_plugin fc --specify_vms atvm139-redhat9.5 atvm112-w2k22; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name nightly-migrateOPS +``` + +### Compute MigrateOPS: vmware +```bash +python3 cmc-templates.py --template cmc-migrateops-compute-migration --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --vm_platforms vmware --test_partition --specify_vms atvm138-oracle9.4-opt atvm112-w2k22 --set_static_ip_dest; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name nightly-computeMigrateOPS-vmware +``` + +### Compute MigrateOPS: ovirt +```bash +python3 cmc-templates.py --template cmc-migrateops-compute-migration --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --vm_platforms ovirt --test_partition --specify_vms atvm124-redhat8.8 atvm111-w2k19 --set_static_ip_dest; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name nightly-computeMigrateOPS-ovirt +``` + +### Group consistency +```bash +python3 cmc-templates.py --template cmc-group-consistency --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --integration_type pure --use_specified_plugin fc --specify_vms atvm4-ubuntu20.04 atvm112-w2k22 --enable_uuid; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name nightly-consistentyGroup +``` + +### H2H same platform +```bash +python3 cmc-templates.py --template cmc-h2h-same-platf --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --integration_type pure --use_specified_plugin fc --specify_vms atvm38-rocky9.0 atvm112-w2k22; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name nightly-h2hSamePlatform +``` + +### H2H different platform +```bash +python3 cmc-templates.py --template cmc-h2h-diff-platf --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --integration_type pure --use_specified_plugin fc --specify_vms atvm65-redhat8.3 atvm112-w2k22; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name nightly-h2hDifferentPlatform +``` + +### Randomized reboot sanity +```bash +python3 cmc-templates.py --template cmc-reboot --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --test_partition --integration_type pure --use_specified_plugin fc --randomize 1 --exclude_partial_match suse15.0 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0 --wait_for_power_on 120; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name sanity-reboot-iscsi +``` + +### Randomized e2e sanity +```bash +python3 cmc-templates.py --template cmc-e2e --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --test_partition --integration_type pure --use_specified_plugin both --randomize 1 --exclude_partial_match suse15.0 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name sanity-e2e +``` + +### Randomized systemOS sanity +```bash +python3 cmc-templates.py --template cmc-systemOS --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --randomize 1 --exclude_partial_match suse15.0 fedora34 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0; \ +python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name sanity-systemOS +``` diff --git a/atvm/atvm-automation-guide.md b/atvm/atvm-automation-guide.md new file mode 100644 index 0000000..0448b88 --- /dev/null +++ b/atvm/atvm-automation-guide.md @@ -0,0 +1,166 @@ +# Run ATVM Automation Guide + +This file is guide-only documentation for operating ATVM CMC automation. +Do not put specific run examples here. +For reusable command examples and common option combinations, use `atvm-automation-examples.md`. + +## Purpose +Run ATVM CMC automation tests on the designated automation VM without unintended system or file changes. + +## ATVM Cypress Automation Controller Client +- Hostname: `atvm-cypres-vm-1` +- IP: `192.168.3.190` +- Credentials: `root / atvmcdsi2012` + +## Operating Constraints +- Run only scripts/commands explicitly requested. +- Do not make manual system configuration changes on the client. +- Do not edit client files unless explicitly requested. + +## Operator Preferences +- Do not include Gold Disk identifiers in `--build_name`. +- `--build_name` must not contain spaces; use `-` between words. +- For multiple VMs in same distro, use distro-scoped filtering (`--containsVm`) instead of long explicit VM lists. +- Before preparing a new run, always check whether automation is already running. +- Always report whether automation is currently running. +- If running, ask whether to terminate; terminate only with explicit approval. +- After termination approval, terminate first, then present planned command(s), then wait for separate execution approval. +- Before any run, always show exact planned command(s) and wait for explicit approval. +- Execute only after explicit approval (for example `approve`). +- After execution, report immediate success/failure only. +- Do not actively monitor completion unless explicitly requested. +- If monitoring is requested, allow long runtime windows (15-30+ minutes) and continue until completion unless operator instructs otherwise. +- Report command errors immediately. +- `sshpass` may be used where password-based SSH automation is required. + +## Core Scripts +- Template prep: `/root/cdc-e2e-cyp-12.17.4/cmc-templates.py` +- Test execution: `./run-sorry-cypress.py` + +Typical sequence: +1. Run `cmc-templates.py` with requested template/options. +2. Run `run-sorry-cypress.py` with matching config and build name. + +## Config File / Gold Disk Mapping +- `cypress.atvm-config-gold.ts` -> Gold Disk 1 +- `cypress.atvm-config-gold-2.ts` -> Gold Disk 2 +- Additional numbered config variants map to corresponding Gold Disks. + +## Available Templates +- `cmc-e2e` +- `cmc-group-consistency` +- `cmc-h2h-diff-platf` +- `cmc-h2h-same-platf` +- `cmc-migrateops` +- `cmc-migrateops-compute-migration` +- `cmc-reboot` +- `cmc-systemOS` + +## Command Pattern +```bash +python3 cmc-templates.py --template