cds-ai/skills/atvm/references/setup-guide.md

ATVM Setup Script Guide

This file is guide-only documentation for running and maintaining the ATVM setup workflow. Do not put dated run examples here.

Scope

• Client setup script: /home/aw/code/cds/atvm/scripts/atvm-setup-script.sh
• Controller wrapper: /home/aw/code/cds/atvm/scripts/run-atvm-setup-and-collect-log.sh
• Run-learnings log: /home/aw/code/cds/atvm/docs/setup/run-learnings.md

Purpose

The setup flow performs a controlled bootstrap across supported Linux distributions:

1. Validate target host identity using expected IP + expected hostname before any configuration.
2. Fix repositories (especially CD/DVD media repo entries).
3. On Ubuntu, configure root SSH password-login workflow using ATVM_TARGET_PASSWORD for follow-on root operations.
4. On Oracle Linux, set default boot kernel to non-UEK when available.
5. Disable unattended auto-upgrades on Ubuntu.
6. Remove specific storage-related packages and install base tooling.
7. Disable SELinux on Red Hat-family systems.
8. Configure static IP as the final step.
9. Print final summary and write logs to atvm_setup_script.log.
10. On SELinux-capable distros, reboot and verify runtime SELinux status post-reboot.
11. Keep client powered on after successful setup so controller-side log collection + SHA256 verification can complete.
12. Power off from controller only after successful verification and no setup errors.

Execution Model

• Shell safety flags: set -euo pipefail
• Logging: colorized console + plain text log file
• Entry point: main "$@"
• Default operator assumption for setup access: source /home/aw/code/cds/.env.credentials.local and use ATVM_TARGET_USER plus ATVM_TARGET_PASSWORD unless explicitly overridden.
• When the operator refers to 192.168.3.191, treat it as the default ATVM target host.
• For SSH to 192.168.3.191, ignore host key mismatch by default with -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null.
• For SSH to 192.168.3.191, source /home/aw/code/cds/.env.credentials.local and use ATVM_TARGET_USER plus ATVM_TARGET_PASSWORD unless the operator explicitly provides different credentials.

Mandatory Identity Gate

Setup must not start unless operator explicitly provides both values:

• --expected-ip <ip>
• --expected-hostname <hostname>

Rules:

• Connect to the operator-provided target IP directly.
• Do not pre-scan alternate candidate IPs.
• Do not infer hostname from target.
• If hostname is missing from request, stop and ask for it.
• If detected hostname does not exactly match expected hostname, stop immediately.
• If expected IP is not assigned on target, stop immediately.

Finish ATVM VM Hardware Prep

When the operator says finish prepping the ATVM VM hardware for <vm-name>, perform only the standard vCenter hardware prep for that exact VM name.

Safety rules:

• Operate only on the exact VM name supplied by the operator.
• Verify the VM exists and is powered off before changing hardware.
• Do not delete, destroy, remove from inventory, or unregister the ATVM VM.
• Do not create replacement disks when the standard shared ATVM data disks are missing; stop and report the missing datastore paths.
• If the VM already has conflicting data disks, FC passthrough devices, or occupied unit numbers, stop and report instead of adding duplicates.

Standard hardware:

• Attach the two existing data disks from the VM's current ATVM boot datastore under atvm-DISKS/:
  • atvm-DISK_1.vmdk as the first added data disk, normally Hard disk 2 / SCSI unit 1.
  • atvm-DISK_2.vmdk as the second added data disk, normally Hard disk 3 / SCSI unit 2.
• Add the two FC passthrough ports:
  • 0000:85:00.0
  • 0000:85:00.1
• Reserve all guest memory after adding passthrough devices.

Preferred govc flow:

source /home/aw/code/cds/.env.credentials.local
export GOVC_URL="https://${VCENTER_HOST}/sdk"
export GOVC_USERNAME="$VCENTER_USER"
export GOVC_PASSWORD="$VCENTER_PASSWORD"
export GOVC_INSECURE=1

vm="/CDSHQ-Eng/vm/<vm-name>"
ds="<current-atvm-boot-datastore>"

govc vm.info "$vm"
govc device.info -vm "$vm"
govc datastore.ls -ds "$ds" atvm-DISKS

govc vm.disk.attach -vm "$vm" -ds "$ds" -disk atvm-DISKS/atvm-DISK_1.vmdk -link=false
govc vm.disk.attach -vm "$vm" -ds "$ds" -disk atvm-DISKS/atvm-DISK_2.vmdk -link=false

govc device.pci.ls -vm "$vm"
govc device.pci.add -vm "$vm" 0000:85:00.0
govc device.pci.add -vm "$vm" 0000:85:00.1

memory_mb="$(govc object.collect -s "$vm" config.hardware.memoryMB)"
govc vm.change -vm "$vm" -memory-pin=true -mem.reservation "$memory_mb"

Operational notes:

• Determine <current-atvm-boot-datastore> from live vCenter inventory for the VM.
• Use -link=false for the standard ATVM data disks so the existing datastore disks are attached directly, matching the reference ATVM inventory pattern.
• Add FC passthrough ports one at a time. Adding both addresses in one govc device.pci.add call can fail with duplicate device-key errors.
• Verify memory reservation by confirming config.memoryAllocation.reservation equals config.hardware.memoryMB.

Canonical Run Order

1. parse_args
2. validate_target_host_identity
3. check_sudo
4. fix_repositories
5. configure_ubuntu_root_ssh_access (Ubuntu only)
6. install_sudo_if_needed
7. configure_oracle_non_uek_kernel (Oracle Linux only)
8. disable_ubuntu_auto_upgrades (Ubuntu only)
9. run_package_installation
10. disable_selinux (RHEL-family only)
11. configure_static_ip (final configuration step)
12. print_final_summary
13. reboot_and_verify_selinux_if_needed
14. poweroff_client_if_successful (controller-driven after verification)

Core Behavior By Step

Repository Fix

• Debian/Ubuntu: comment cdrom entries in apt lists and run apt-get update.
• RHEL-family/Oracle: disable media/cdrom/dvd repo entries and run yum clean all && yum makecache.
• Fedora: same model via dnf clean all && dnf makecache.
• openSUSE/SLES: disable CD/DVD repos with zypper mr -d and refresh.

Oracle Linux Kernel Handling

• Oracle Linux only.
• Select first non-UEK kernel via grubby --info=ALL and set GRUB default.
• Track whether default changed and whether reboot is required.

Ubuntu Root SSH Workflow

• Ubuntu only.
• Require ATVM_TARGET_PASSWORD in the environment, then set the root password to that value and unlock the root account.
• Write /etc/ssh/sshd_config.d/99-atvm-root-login.conf enabling root + password auth.
• Validate config and restart SSH service.

Ubuntu Auto-Upgrade Disable

• Ubuntu only.
• Update /etc/apt/apt.conf.d/20auto-upgrades to disable periodic update/upgrade actions.

Package Installation

• Package manager detection order: apt-get, dnf, yum, zypper, pacman, apk.
• Pre-cleanup removes multipath/iSCSI packages where applicable.
• Installs kernel headers per distro.
• Base package set includes: curl wget git vim perl gdb scsitools net-tools parted fio ca-certificates python3 elfutils-libelf-devel

SELinux Disable

• RHEL-family only.
• If enforcing/permissive, backup and rewrite /etc/selinux/config to disabled.
• Marks reboot recommendation/requirement in summary.

Static IP Configuration (Final Step)

Hardcoded target values:

• IP: 192.168.3.191
• Prefix: 22
• Gateway: 192.168.0.1
• DNS: 8.8.8.8, 8.8.4.4

Interface detection priority:

1. default-route interface
2. first non-loopback interface with IPv4
3. first non-loopback interface from link list

Network-stack handling includes netplan, NetworkManager/nmcli, wicked, and legacy ifcfg fallback patterns.

SELinux Reboot Verification

• Applies to rhel, centos, rocky, almalinux, fedora, ol when SELinux changed.
• Creates one-time systemd verifier service before reboot.
• Post-reboot service records runtime getenforce and self-removes.
• On success/no real errors, keeps client on for controller log copy/hash verification before controller power-off.
• On errors, leaves client on for manual inspection.

Power-State Rules

• After successful setup, keep client powered on until controller log collection + SHA256 verification completes.
• If verification succeeds and no real error lines exist (^\[ERROR\]), controller powers off client.
• If any real error lines exist, keep client powered on.

Logging and Verification

• Client log filename: atvm_setup_script.log
• Common client log path when run as root: /root/atvm_setup_script.log
• Controller collected log naming: atvm_configuration_<hostname>_<yyyymmdd_hhmmss>.log

Required post-run validation:

1. Copy client log to controller atvm/log/ path.
2. Compare SHA256 between client and copied controller log.
3. Require exact match.

Preferred Execution Commands

Direct client execution:

source /home/aw/code/cds/.env.credentials.local
sudo bash /home/cirrususer/atvm-setup-script.sh \
  --expected-ip <current-client-ip> \
  --expected-hostname <exact-hostname>

Controller run + collect:

source /home/aw/code/cds/.env.credentials.local
EXPECTED_IP_ARG=<current-client-ip> EXPECTED_HOSTNAME_ARG=<exact-hostname> \
/home/aw/code/cds/atvm/scripts/run-atvm-setup-and-collect-log.sh

Controller collect-only after client run:

source /home/aw/code/cds/.env.credentials.local
/home/aw/code/cds/atvm/scripts/run-atvm-setup-and-collect-log.sh --collect-after-complete

Troubleshooting

• If local collected log is missing, do not rerun full setup just for log recovery.
• Use collect-only mode and verify SHA256 after copy.
• If wrapper appears stuck after IP/reboot transition, stop older wrapper sessions and run one fresh collect-only session.
• If sshpass is missing on controller, wrapper can still run but may require repeated interactive password prompts.

Operational Caveats

• Not fully idempotent for all paths; repeated runs may rewrite network configs and create multiple backups.
• Static IP values are hardcoded; adjust before use in other environments.
• Run in maintenance windows because network changes can interrupt active sessions.
• Preserve host identity gating; do not weaken expected IP/hostname checks.

Update Rule

• After each run, update this file only for guide/rule/checklist/default behavior changes.
• Put run-specific outcomes in run-learnings.md only when the run produced a new learning.