From dcd7a998d7448cda517f4d52c0a13e7cf70641cd Mon Sep 17 00:00:00 2001 From: Cirrus Codex Date: Tue, 26 May 2026 22:39:46 -0400 Subject: [PATCH] Tighten kernel upgrade checklist step verification --- tests/cmc-upgrade-kernel-test.md | 154 +++++++++++++++---------------- 1 file changed, 77 insertions(+), 77 deletions(-) diff --git a/tests/cmc-upgrade-kernel-test.md b/tests/cmc-upgrade-kernel-test.md index c9002de..c6c49a2 100644 --- a/tests/cmc-upgrade-kernel-test.md +++ b/tests/cmc-upgrade-kernel-test.md @@ -84,83 +84,83 @@ Validate CMC behavior across staged kernel upgrades on a cloned VM, including CM - Do not begin teardown until every item below is checked complete. - If any checklist item cannot be checked, stop the test and record the blocker. -- [ ] 0. Source `/home/cirrus/cds/.env.credentials.local` and verify required credential variables are present without printing secret values; re-source it in any new shell/session before vCenter, SSH, Red Hat subscription, or CMC actions; start per-step timing and hard stop any step that exceeds 10 minutes. -- [ ] 1. If this is a manual run against an actual Red Hat subscription-managed machine, perform the Red Hat preflight commands `subscription-manager remove --all`, `subscription-manager unregister`, `subscription-manager clean`, and `subscription-manager register --username "$REDHAT_SUBSCRIPTION_USER" --password "$REDHAT_SUBSCRIPTION_PASSWORD"`; otherwise explicitly mark this item skipped. -- [ ] 2. Confirm the requested source host is not a SUSE/SLES machine; if it is SUSE/SLES, hard stop before source power-on or clone creation. -- [ ] 3. Remove offline hosts in `skidamarink` using Cirrus Data MCP tools for offline-host cleanup. -- [ ] 4. From vCenter, confirm source host is powered on for the inspection phase; power it on if it is not already powered on. -- [ ] 5. From vCenter, query guest-tools for the live source host IP address. -- [ ] 6. SSH to the source host IP address found in step 5 using credentials from `/home/cirrus/cds/.env.credentials.local`. -- [ ] 7. On the source host, inspect distro repository files before listing available kernel builds and hard stop if any enabled/source repo points at `192.168.3.199` (`/etc/yum.repos.d/*.repo`, `/etc/apt/sources.list`, `/etc/apt/sources.list.d/*`, `/etc/zypp/repos.d/*.repo`, or equivalent files present on the host). -- [ ] 8. On the source host, record the current OS version and running kernel version before cloning. -- [ ] 9. On the source host, refresh package metadata and build the kernel candidate list from all available versions using the distro command set: RHEL/Oracle/Rocky/Alma: `dnf makecache; dnf list --showduplicates kernel kernel-devel kernel-headers`; older RHEL/CentOS: `yum makecache; yum list --showduplicates kernel kernel-devel kernel-headers`; Ubuntu: `rm -rf /var/lib/apt/lists/* && apt-get clean && apt-get update; apt-cache madison linux-image-generic linux-headers-generic; apt list -a linux-image-generic linux-headers-generic`; Debian: `rm -rf /var/lib/apt/lists/* && apt-get clean && apt-get update; apt-cache madison linux-image-amd64 linux-headers-amd64; apt list -a linux-image-amd64 linux-headers-amd64`. Do not run this test for SUSE/SLES; step 2 must stop those hosts before this point. On Ubuntu, inspect the generic track first. On Debian, inspect the amd64 meta-package track first. Confirm candidate availability with alternate package listing methods if needed before deciding whether the distro default track is usable. If this refreshed list differs from any earlier package view, trust the refreshed live metadata and record that the earlier view was stale. -- [ ] 10. Apply the candidate scope rule: same major OS family only, same minor stream preferred, same OS release preferred where the distro does not use RHEL-style minor streams, and cross-minor/cross-release candidates allowed only when fewer than 2 same-stream upgrade candidates are available. -- [ ] 11. Verify at least 2 upgrade candidates exist in the filtered candidate list. -- [ ] 12. If fewer than 2 candidates exist, hard stop and end the run before clone creation. -- [ ] 13. Confirm steps 7-12 passed; if any stop condition was hit, do not clone. -- [ ] 14. From vCenter, issue the source-host power-off request and wait for `poweredOff`. -- [ ] 15. From vCenter, confirm the source host is still `poweredOff` immediately before cloning. -- [ ] 16. Determine the base clone name `aw999-[source-without-atvmxxx-]`. -- [ ] 17. From vCenter, check whether the base clone name already exists. -- [ ] 18. If needed, choose the next available suffixed clone name using `aw999-[source-without-atvmxxx-]-1`, then `-2`, then `-N` as needed. -- [ ] 19. From vCenter, clone the source VM on `AutomatedTest-UnitTesting` using the resolved clone VM name from steps 16-18, pass only the clone VM name as the destination, and default it to `CDS1-ESX165` / `192.168.1.165` unless overridden. -- [ ] 20. From vCenter, detach the 2 FC PCI adapters from the cloned VM. -- [ ] 21. From vCenter, verify both FC passthrough devices are no longer present on the clone. -- [ ] 22. From vCenter, power on the clone. -- [ ] 23. From vCenter, query guest-tools for the live clone IP. -- [ ] 24. SSH to the live clone IP found in step 23 using credentials from `/home/cirrus/cds/.env.credentials.local`. -- [ ] 25. On the clone, change the OS hostname to the clone name with `.` replaced by `-`. -- [ ] 26. On the clone, convert networking from static IP to DHCP. -- [ ] 27. On the clone, remove/clean static IP configuration references. -- [ ] 28. On the clone, reboot the machine. -- [ ] 29. From vCenter, query guest-tools again for the new live clone IP. -- [ ] 30. SSH to the new live clone IP found in step 29. -- [ ] 31. On the clone, verify DHCP state. -- [ ] 32. If the clone still reports the previous static IP, fix config cleanup and repeat steps 27-31. -- [ ] 33. Continue all remaining steps using the live DHCP IP confirmed in step 31. -- [ ] 34. On the clone, verify `/dev/sdb` is the intended 10 GB source disk, identify the intended 11 GB destination disk, then wipe `/dev/sdb` once and verify no filesystem or partition signatures remain. -- [ ] 35. Using the cirrusdata skill, install CMC on the clone in the `skidamarink` project with `-no-prebuilt-mtdi-nexus`. -- [ ] 36. Using the cirrusdata skill, create the first local migration from the verified 10 GB source disk to the verified 11 GB destination disk in the `skidamarink` project. -- [ ] 37. If migration session creation fails, hard stop as blocker-fail. -- [ ] 38. Using the cirrusdata skill, wait for initial sync completion in the `skidamarink` project. -- [ ] 39. SSH to the live DHCP clone IP confirmed in step 31, refresh package metadata, and check available kernels again using the full distro candidate listing: RHEL/Oracle/Rocky/Alma: `dnf makecache; dnf list --showduplicates kernel kernel-devel kernel-headers`; older RHEL/CentOS: `yum makecache; yum list --showduplicates kernel kernel-devel kernel-headers`; Ubuntu: `rm -rf /var/lib/apt/lists/* && apt-get clean && apt-get update; apt-cache madison linux-image-generic linux-headers-generic; apt list -a linux-image-generic linux-headers-generic`; Debian: `rm -rf /var/lib/apt/lists/* && apt-get clean && apt-get update; apt-cache madison linux-image-amd64 linux-headers-amd64; apt list -a linux-image-amd64 linux-headers-amd64`. If this refreshed list differs from step 9, trust the refreshed live metadata and record that the earlier view was stale. -- [ ] 40. Select the first-upgrade target from the filtered candidate list; it must stay in the same major OS family, must not be the latest candidate, and must follow the candidate scope rule from step 10. If no valid non-latest first-upgrade target exists, hard stop as blocker-fail. -- [ ] 41. On the clone, verify matching dev/header packages are available for the exact first-upgrade target; on Debian/Ubuntu, resolve meta packages to the exact versioned `linux-image-*` and `linux-headers-*` packages before installing. -- [ ] 42. On the clone, install the first-upgrade kernel and matching dev/header packages without rebooting yet. -- [ ] 43. On Red Hat-family systems with `grubby` including Oracle Linux, set the first-upgrade kernel as the grubby default and verify `grubby --default-kernel` returns the selected `/boot/vmlinuz-` path before reboot. -- [ ] 44. On the clone, reboot into the first-upgrade kernel. -- [ ] 45. From vCenter, query guest-tools again for the live clone IP after reboot. -- [ ] 46. SSH to the rebooted clone IP found in step 45. -- [ ] 47. On the clone, verify kernel plus dev/header package versions match the selected first-upgrade version exactly; on Debian/Ubuntu, verify the running kernel maps to the exact installed versioned `linux-image-*` and `linux-headers-*` packages. -- [ ] 48. If versions do not match exactly, stop as blocker-fail. -- [ ] 49. Using the cirrusdata skill, verify the clone is online in the `skidamarink` project. -- [ ] 50. On the clone, verify MTDI and Galaxy Migrate services/driver are up. -- [ ] 51. On the clone, write sample data to the verified 10 GB source disk. -- [ ] 52. Using the cirrusdata skill, trigger sync and confirm tracking status in the `skidamarink` project. -- [ ] 53. Using the cirrusdata skill, uninstall CMC from the clone in the `skidamarink` project. -- [ ] 54. Using Cirrus Data MCP tools, run host cleanup for `skidamarink` and remove the cloned host entry for this test clone only, regardless of online/offline status. -- [ ] 55. Using Cirrus Data MCP tools, verify the cloned host entry and all migration sessions for the cloned host are gone from `skidamarink` before continuing. -- [ ] 56. SSH to the rebooted clone IP found in step 45, refresh package metadata, and check available kernels again using the full distro candidate listing: RHEL/Oracle/Rocky/Alma: `dnf makecache; dnf list --showduplicates kernel kernel-devel kernel-headers`; older RHEL/CentOS: `yum makecache; yum list --showduplicates kernel kernel-devel kernel-headers`; Ubuntu: `rm -rf /var/lib/apt/lists/* && apt-get clean && apt-get update; apt-cache madison linux-image-generic linux-headers-generic; apt list -a linux-image-generic linux-headers-generic`; Debian: `rm -rf /var/lib/apt/lists/* && apt-get clean && apt-get update; apt-cache madison linux-image-amd64 linux-headers-amd64; apt list -a linux-image-amd64 linux-headers-amd64`. If this refreshed list differs from step 39, trust the refreshed live metadata and record that the earlier view was stale. -- [ ] 57. Select the latest-upgrade target kernel from the filtered candidate list; it must stay in the same major OS family, should use the latest available candidate in that scope, and must follow the candidate scope rule from step 10. If no valid latest-upgrade target exists, hard stop as blocker-fail. -- [ ] 58. On the clone, verify matching dev/header packages are available for the exact latest-upgrade target; on Debian/Ubuntu, resolve meta packages to the exact versioned `linux-image-*` and `linux-headers-*` packages before installing. -- [ ] 59. On the clone, install the latest-upgrade kernel and matching dev/header packages without rebooting yet. -- [ ] 60. On Red Hat-family systems with `grubby` including Oracle Linux, set the latest-upgrade kernel as the grubby default and verify `grubby --default-kernel` returns the selected `/boot/vmlinuz-` path before reboot. -- [ ] 61. On the clone, reboot into the latest-upgrade kernel. -- [ ] 62. From vCenter, query guest-tools again for the live clone IP after reboot. -- [ ] 63. SSH to the rebooted clone IP found in step 62. -- [ ] 64. On the clone, verify kernel plus dev/header package versions match the selected latest-upgrade version exactly; on Debian/Ubuntu, verify the running kernel maps to the exact installed versioned `linux-image-*` and `linux-headers-*` packages. -- [ ] 65. If versions do not match exactly, stop as blocker-fail. -- [ ] 66. Using the cirrusdata skill, run the CMC installer again after the prior uninstall and host cleanup, install CMC on the clone in the `skidamarink` project with `-no-prebuilt-mtdi-nexus` on the latest kernel. -- [ ] 67. Using the cirrusdata skill, create the second local migration from the verified 10 GB source disk to the verified 11 GB destination disk in the `skidamarink` project and wait for initial sync completion. -- [ ] 68. If migration session creation fails, hard stop as blocker-fail. -- [ ] 69. Using the cirrusdata skill, confirm the machine is online in the `skidamarink` project. -- [ ] 70. SSH to the live clone IP currently reported by vCenter and verify MTDI and Galaxy Migrate services/driver are up. -- [ ] 71. Only after steps 66-70 all pass, begin success-path cleanup. -- [ ] 72. From vCenter, power off the cloned machine. -- [ ] 73. From vCenter, delete the cloned VM and its disks from inventory. -- [ ] 74. Using Cirrus Data MCP tools, run final host cleanup for `skidamarink`, remove the cloned host entry for this test clone only, and verify the cloned host entry plus all migration sessions for the cloned host are gone. -- [ ] 75. Blocker-fail path after clone creation, as an alternate to steps 71-74: leave the cloned VM powered on and present in inventory for manual inspection, then continue to step 76. -- [ ] 76. Append the current run only to `/home/cirrus/cds/tmp/tests/cmc upgrade test/cmc-upgrade-kernel-test-results.md` with the required host metadata, kernel progression, execution summary, final outcome, and total test duration; do not create or update any other result, run-specific result, or summary files. +- [ ] 0. Source `/home/cirrus/cds/.env.credentials.local` and verify required credential variables are present without printing secret values; re-source it in any new shell/session before vCenter, SSH, Red Hat subscription, or CMC actions; start per-step timing and hard stop any step that exceeds 10 minutes. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 1. If this is a manual run against an actual Red Hat subscription-managed machine, perform the Red Hat preflight commands `subscription-manager remove --all`, `subscription-manager unregister`, `subscription-manager clean`, and `subscription-manager register --username "$REDHAT_SUBSCRIPTION_USER" --password "$REDHAT_SUBSCRIPTION_PASSWORD"`; otherwise explicitly mark this item skipped. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 2. Confirm the requested source host is not a SUSE/SLES machine; if it is SUSE/SLES, hard stop before source power-on or clone creation. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 3. Remove offline hosts in `skidamarink` using Cirrus Data MCP tools for offline-host cleanup. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 4. From vCenter, confirm source host is powered on for the inspection phase; power it on if it is not already powered on. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 5. From vCenter, query guest-tools for the live source host IP address. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 6. SSH to the source host IP address found in step 5 using credentials from `/home/cirrus/cds/.env.credentials.local`. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 7. On the source host, inspect distro repository files before listing available kernel builds and hard stop if any enabled/source repo points at `192.168.3.199` (`/etc/yum.repos.d/*.repo`, `/etc/apt/sources.list`, `/etc/apt/sources.list.d/*`, `/etc/zypp/repos.d/*.repo`, or equivalent files present on the host). Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 8. On the source host, record the current OS version and running kernel version before cloning. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 9. On the source host, refresh package metadata and build the kernel candidate list from all available versions using the distro command set: RHEL/Oracle/Rocky/Alma: `dnf makecache; dnf list --showduplicates kernel kernel-devel kernel-headers`; older RHEL/CentOS: `yum makecache; yum list --showduplicates kernel kernel-devel kernel-headers`; Ubuntu: `rm -rf /var/lib/apt/lists/* && apt-get clean && apt-get update; apt-cache madison linux-image-generic linux-headers-generic; apt list -a linux-image-generic linux-headers-generic`; Debian: `rm -rf /var/lib/apt/lists/* && apt-get clean && apt-get update; apt-cache madison linux-image-amd64 linux-headers-amd64; apt list -a linux-image-amd64 linux-headers-amd64`. Do not run this test for SUSE/SLES; step 2 must stop those hosts before this point. On Ubuntu, inspect the generic track first. On Debian, inspect the amd64 meta-package track first. Confirm candidate availability with alternate package listing methods if needed before deciding whether the distro default track is usable. If this refreshed list differs from any earlier package view, trust the refreshed live metadata and record that the earlier view was stale. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 10. Apply the candidate scope rule: same major OS family only, same minor stream preferred, same OS release preferred where the distro does not use RHEL-style minor streams, and cross-minor/cross-release candidates allowed only when fewer than 2 same-stream upgrade candidates are available. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 11. Verify at least 2 upgrade candidates exist in the filtered candidate list. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 12. If fewer than 2 candidates exist, hard stop and end the run before clone creation. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 13. Confirm steps 7-12 passed; if any stop condition was hit, do not clone. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 14. From vCenter, issue the source-host power-off request and wait for `poweredOff`. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 15. From vCenter, confirm the source host is still `poweredOff` immediately before cloning. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 16. Determine the base clone name `aw999-[source-without-atvmxxx-]`. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 17. From vCenter, check whether the base clone name already exists. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 18. If needed, choose the next available suffixed clone name using `aw999-[source-without-atvmxxx-]-1`, then `-2`, then `-N` as needed. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 19. From vCenter, clone the source VM on `AutomatedTest-UnitTesting` using the resolved clone VM name from steps 16-18, pass only the clone VM name as the destination, and default it to `CDS1-ESX165` / `192.168.1.165` unless overridden. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 20. From vCenter, detach the 2 FC PCI adapters from the cloned VM. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 21. From vCenter, verify both FC passthrough devices are no longer present on the clone. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 22. From vCenter, power on the clone. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 23. From vCenter, query guest-tools for the live clone IP. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 24. SSH to the live clone IP found in step 23 using credentials from `/home/cirrus/cds/.env.credentials.local`. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 25. On the clone, change the OS hostname to the clone name with `.` replaced by `-`. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 26. On the clone, convert networking from static IP to DHCP. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 27. On the clone, remove/clean static IP configuration references. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 28. On the clone, reboot the machine. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 29. From vCenter, query guest-tools again for the new live clone IP. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 30. SSH to the new live clone IP found in step 29. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 31. On the clone, verify DHCP state. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 32. If the clone still reports the previous static IP, fix config cleanup and repeat steps 27-31. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 33. Continue all remaining steps using the live DHCP IP confirmed in step 31. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 34. On the clone, verify `/dev/sdb` is the intended 10 GB source disk, identify the intended 11 GB destination disk, then wipe `/dev/sdb` once and verify no filesystem or partition signatures remain. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 35. Using the cirrusdata skill, install CMC on the clone in the `skidamarink` project with `-no-prebuilt-mtdi-nexus`. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 36. Using the cirrusdata skill, create the first local migration from the verified 10 GB source disk to the verified 11 GB destination disk in the `skidamarink` project. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 37. If migration session creation fails, hard stop as blocker-fail. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 38. Using the cirrusdata skill, wait for initial sync completion in the `skidamarink` project. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 39. SSH to the live DHCP clone IP confirmed in step 31, refresh package metadata, and check available kernels again using the full distro candidate listing: RHEL/Oracle/Rocky/Alma: `dnf makecache; dnf list --showduplicates kernel kernel-devel kernel-headers`; older RHEL/CentOS: `yum makecache; yum list --showduplicates kernel kernel-devel kernel-headers`; Ubuntu: `rm -rf /var/lib/apt/lists/* && apt-get clean && apt-get update; apt-cache madison linux-image-generic linux-headers-generic; apt list -a linux-image-generic linux-headers-generic`; Debian: `rm -rf /var/lib/apt/lists/* && apt-get clean && apt-get update; apt-cache madison linux-image-amd64 linux-headers-amd64; apt list -a linux-image-amd64 linux-headers-amd64`. If this refreshed list differs from step 9, trust the refreshed live metadata and record that the earlier view was stale. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 40. Select the first-upgrade target from the filtered candidate list; it must stay in the same major OS family, must not be the latest candidate, and must follow the candidate scope rule from step 10. If no valid non-latest first-upgrade target exists, hard stop as blocker-fail. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 41. On the clone, verify matching dev/header packages are available for the exact first-upgrade target; on Debian/Ubuntu, resolve meta packages to the exact versioned `linux-image-*` and `linux-headers-*` packages before installing. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 42. On the clone, install the first-upgrade kernel and matching dev/header packages without rebooting yet. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 43. On Red Hat-family systems with `grubby` including Oracle Linux, set the first-upgrade kernel as the grubby default and verify `grubby --default-kernel` returns the selected `/boot/vmlinuz-` path before reboot. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 44. On the clone, reboot into the first-upgrade kernel. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 45. From vCenter, query guest-tools again for the live clone IP after reboot. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 46. SSH to the rebooted clone IP found in step 45. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 47. On the clone, verify kernel plus dev/header package versions match the selected first-upgrade version exactly; on Debian/Ubuntu, verify the running kernel maps to the exact installed versioned `linux-image-*` and `linux-headers-*` packages. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 48. If versions do not match exactly, stop as blocker-fail. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 49. Using the cirrusdata skill, verify the clone is online in the `skidamarink` project. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 50. On the clone, verify MTDI and Galaxy Migrate services/driver are up. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 51. On the clone, write sample data to the verified 10 GB source disk. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 52. Using the cirrusdata skill, trigger sync and confirm tracking status in the `skidamarink` project. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 53. Using the cirrusdata skill, uninstall CMC from the clone in the `skidamarink` project. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 54. Using Cirrus Data MCP tools, run host cleanup for `skidamarink` and remove the cloned host entry for this test clone only, regardless of online/offline status. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 55. Using Cirrus Data MCP tools, verify the cloned host entry and all migration sessions for the cloned host are gone from `skidamarink` before continuing. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 56. SSH to the rebooted clone IP found in step 45, refresh package metadata, and check available kernels again using the full distro candidate listing: RHEL/Oracle/Rocky/Alma: `dnf makecache; dnf list --showduplicates kernel kernel-devel kernel-headers`; older RHEL/CentOS: `yum makecache; yum list --showduplicates kernel kernel-devel kernel-headers`; Ubuntu: `rm -rf /var/lib/apt/lists/* && apt-get clean && apt-get update; apt-cache madison linux-image-generic linux-headers-generic; apt list -a linux-image-generic linux-headers-generic`; Debian: `rm -rf /var/lib/apt/lists/* && apt-get clean && apt-get update; apt-cache madison linux-image-amd64 linux-headers-amd64; apt list -a linux-image-amd64 linux-headers-amd64`. If this refreshed list differs from step 39, trust the refreshed live metadata and record that the earlier view was stale. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 57. Select the latest-upgrade target kernel from the filtered candidate list; it must stay in the same major OS family, should use the latest available candidate in that scope, and must follow the candidate scope rule from step 10. If no valid latest-upgrade target exists, hard stop as blocker-fail. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 58. On the clone, verify matching dev/header packages are available for the exact latest-upgrade target; on Debian/Ubuntu, resolve meta packages to the exact versioned `linux-image-*` and `linux-headers-*` packages before installing. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 59. On the clone, install the latest-upgrade kernel and matching dev/header packages without rebooting yet. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 60. On Red Hat-family systems with `grubby` including Oracle Linux, set the latest-upgrade kernel as the grubby default and verify `grubby --default-kernel` returns the selected `/boot/vmlinuz-` path before reboot. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 61. On the clone, reboot into the latest-upgrade kernel. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 62. From vCenter, query guest-tools again for the live clone IP after reboot. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 63. SSH to the rebooted clone IP found in step 62. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 64. On the clone, verify kernel plus dev/header package versions match the selected latest-upgrade version exactly; on Debian/Ubuntu, verify the running kernel maps to the exact installed versioned `linux-image-*` and `linux-headers-*` packages. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 65. If versions do not match exactly, stop as blocker-fail. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 66. Using the cirrusdata skill, run the CMC installer again after the prior uninstall and host cleanup, install CMC on the clone in the `skidamarink` project with `-no-prebuilt-mtdi-nexus` on the latest kernel. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 67. Using the cirrusdata skill, create the second local migration from the verified 10 GB source disk to the verified 11 GB destination disk in the `skidamarink` project and wait for initial sync completion. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 68. If migration session creation fails, hard stop as blocker-fail. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 69. Using the cirrusdata skill, confirm the machine is online in the `skidamarink` project. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 70. SSH to the live clone IP currently reported by vCenter and verify MTDI and Galaxy Migrate services/driver are up. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 71. Only after steps 66-70 all pass, begin success-path cleanup. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 72. From vCenter, power off the cloned machine. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 73. From vCenter, delete the cloned VM and its disks from inventory. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 74. Using Cirrus Data MCP tools, run final host cleanup for `skidamarink`, remove the cloned host entry for this test clone only, and verify the cloned host entry plus all migration sessions for the cloned host are gone. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 75. Blocker-fail path after clone creation, as an alternate to steps 71-74: leave the cloned VM powered on and present in inventory for manual inspection, then continue to step 76. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. +- [ ] 76. Append the current run only to `/home/cirrus/cds/tmp/tests/cmc upgrade test/cmc-upgrade-kernel-test-results.md` with the required host metadata, kernel progression, execution summary, final outcome, and total test duration; do not create or update any other result, run-specific result, or summary files. Verify this step completed successfully, then re-read `tests/cmc-upgrade-kernel-test.md` to confirm the next step before performing it. ## Stop Conditions Stop immediately and record a blocker if any of these occur: