From e1a551cdb26f7c9f46cce7aa52ecde57a8bc4a25 Mon Sep 17 00:00:00 2001 From: "anthony.wen" Date: Thu, 12 Mar 2026 11:03:08 -0400 Subject: [PATCH] Update ATVM automation blacklist and status formatting guidance Add atvm144-suse15.0 to the ATVM automation blacklist because it crashes when creating a migration session. Update maintained exclude examples to include the new blacklist entry. Tighten status reporting guidance to require one machine per line. Add a Notes section for failure reasons and operator-facing context. Record the new run learnings in atvm-automation-runs.md. --- atvm/AGENTS.md | 18 ++++++++++++------ atvm/atvm-automation-examples.md | 6 +++--- atvm/atvm-automation-guide.md | 20 +++++++++++++------- atvm/atvm-automation-runs.md | 17 +++++++++++++++++ 4 files changed, 45 insertions(+), 16 deletions(-) diff --git a/atvm/AGENTS.md b/atvm/AGENTS.md index 6033c83..f0f84a0 100644 --- a/atvm/AGENTS.md +++ b/atvm/AGENTS.md @@ -97,27 +97,30 @@ Use `atvm-automation-examples.md` as the common options/command reference. When the operator asks for run status, report in this order: 1. Heading/title using the run `build_name`. 2. Completed machines with machine name first and status second for each machine. -3. Skipped machines with reason. -4. Remaining machines still to run. -5. Summary counts for finished, passed, failed, and skipped machines. -6. Timing details: +3. Notes. +4. Skipped machines with reason. +5. Remaining machines still to run. +6. Summary counts for finished, passed, failed, and skipped machines. +7. Timing details: - start time - end time if complete - total run time if complete, or elapsed run time if still running - quickest completed test runtime - longest completed test runtime - average completed test runtime -7. Estimated completion time. +8. Estimated completion time. Status details: - Use the live run log on the automation VM when available. - Use the run `build_name` as the heading/title when available. - Format every machine entry as `machine-name - STATUS`. +- Put each machine on its own line; do not combine multiple machine statuses on one line. +- Put failure reasons, oddities, and operator-facing context in a dedicated `Notes` section. - Show blacklisted machines under skipped machines when they are part of the requested scope. - Show in-progress machines under remaining machines as `RUNNING`. - Show not-yet-started machines as `NOT STARTED`. - Use completed spec results already recorded in the log to determine machine pass/fail state. -- For failed machines, include the failure reason from the run log in the status output. +- For failed machines, mark the machine `FAIL` in the completed list and include the failure reason in `Notes`. - Include start time in status output when it can be derived from the log. - Include end time and total runtime for completed runs, or elapsed runtime for active runs. - Include quickest completed test runtime, longest completed test runtime, and average completed test runtime under timing details when they can be derived from the log. @@ -131,6 +134,9 @@ CMC install blacklist (`BLACKLISTED: CMC INSTALL - CAN'T COMPILE`): - `atvm41-redhat6.0` - `atvm73-oracle6.0` +Crash blacklist (`BLACKLISTED: CRASHES WHEN CREATING MIGRATION SESSION - BUG`): +- `atvm144-suse15.0` + Support-request blacklist (`BLACKLISTED: SUPPORT REQUEST - WAITING`): - `atvm113-debian9.0.0` - `atvm115-debian9.1.0` diff --git a/atvm/atvm-automation-examples.md b/atvm/atvm-automation-examples.md index 3d76536..aaaf462 100644 --- a/atvm/atvm-automation-examples.md +++ b/atvm/atvm-automation-examples.md @@ -80,18 +80,18 @@ python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name ### Randomized reboot sanity ```bash -python3 cmc-templates.py --template cmc-reboot --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --test_partition --integration_type pure --use_specified_plugin fc --randomize 1 --exclude_partial_match suse15.0 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0 --wait_for_power_on 120; \ +python3 cmc-templates.py --template cmc-reboot --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --test_partition --integration_type pure --use_specified_plugin fc --randomize 1 --exclude_partial_match suse15.0 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0 --wait_for_power_on 120; \ python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name sanity-reboot-iscsi ``` ### Randomized e2e sanity ```bash -python3 cmc-templates.py --template cmc-e2e --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --test_partition --integration_type pure --use_specified_plugin both --randomize 1 --exclude_partial_match suse15.0 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0; \ +python3 cmc-templates.py --template cmc-e2e --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --test_partition --integration_type pure --use_specified_plugin both --randomize 1 --exclude_partial_match suse15.0 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0; \ python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name sanity-e2e ``` ### Randomized systemOS sanity ```bash -python3 cmc-templates.py --template cmc-systemOS --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --randomize 1 --exclude_partial_match suse15.0 fedora34 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0; \ +python3 cmc-templates.py --template cmc-systemOS --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --randomize 1 --exclude_partial_match suse15.0 fedora34 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0; \ python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name sanity-systemOS ``` diff --git a/atvm/atvm-automation-guide.md b/atvm/atvm-automation-guide.md index 28f0ddf..06668ac 100644 --- a/atvm/atvm-automation-guide.md +++ b/atvm/atvm-automation-guide.md @@ -83,6 +83,9 @@ Permanently blacklisted because CMC cannot compile: - `atvm41-redhat6.0` - `atvm73-oracle6.0` +Temporarily blacklisted because the run crashes when creating a migration session: +- `atvm144-suse15.0` + Temporarily blacklisted while support requests are waiting: - `atvm113-debian9.0.0` - `atvm115-debian9.1.0` @@ -93,7 +96,7 @@ Temporarily blacklisted until re-created: - `atvm157-debian13.0.0` Preferred exclude list: -- `--exclude_partial_match atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0` +- `--exclude_partial_match atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0` ## Running-Automation Check (Mandatory) Before any new automation request: @@ -134,22 +137,25 @@ When asked for one VM or a VM set: When the operator asks for the status of an ATVM automation run, report in this order: 1. Heading/title using the run `build_name`. 2. Completed machines with machine name first and status second for each machine. -3. Skipped machines with reason. -4. Remaining machines still to run. -5. Summary counts for finished, passed, failed, and skipped machines. -6. Timing details: +3. Notes. +4. Skipped machines with reason. +5. Remaining machines still to run. +6. Summary counts for finished, passed, failed, and skipped machines. +7. Timing details: - start time - end time if complete - total run time if complete, or elapsed run time if still running - quickest completed test runtime - longest completed test runtime - average completed test runtime -7. Estimated completion time. +8. Estimated completion time. Status-report expectations: - Use the live automation VM state when available. - Derive the heading/title from the run `build_name` when available. - Format every machine entry as `machine-name - STATUS`. +- Put each machine on its own line; never combine multiple machines into one paragraph or comma-separated line. +- Use a separate `Notes` section for failure reasons, anomalies, or operator-relevant context rather than cramming those details into the completed-machine list. - For categorized runs, reconstruct the whole run across all category batches; do not treat the current live category batch as the full run scope. - Derive completed-machine status from completed spec results already written during the same run. - Parse all same-run `test-result-*.xml` files, not only machine-named `test-result-atvm*.xml` files. @@ -168,6 +174,6 @@ Status-report expectations: - If a machine is currently in progress, show it under remaining machines as `RUNNING`. - If a machine has not started yet, show it under remaining machines as `NOT STARTED`. - If no failures are present in completed spec results, report those completed machines as `PASS`. -- If a completed spec result shows a failure, report that machine as `FAIL` and include the failure reason from the run log. +- If a completed spec result shows a failure, report that machine as `FAIL` in the completed list and put the failure reason in `Notes`. - Base the completion estimate on the full remaining machine count and recent per-machine runtime visible in the run log. - Make the estimate explicitly refer to completion of the entire remaining run, not only the current machine/spec. diff --git a/atvm/atvm-automation-runs.md b/atvm/atvm-automation-runs.md index 0c4e52c..a1717dc 100644 --- a/atvm/atvm-automation-runs.md +++ b/atvm/atvm-automation-runs.md @@ -74,3 +74,20 @@ This file stores run-specific examples only when a run produced a new learning r - Extract the machine name from XML contents such as `testsuite file=`, `testsuite name=`, or `testcase name=` when the filename does not include the machine name. - Treat `check-xml-files.ts` XML outputs as bookkeeping steps, not machine results. - Prefer the most recently written same-run XML per machine when multiple XML files exist for that machine. + +## Run Learning: 2026-03-12 (Status output must be one machine per line with notes separated) +- Observed requirement: + - Listing multiple completed machines on one line makes run status harder to scan and does not meet the expected reporting format. + - Failure reasons and extra context should be separated from the machine status list so the list stays clean. +- Action for future runs: + - Under completed, skipped, and remaining sections, put exactly one machine status on each line. + - Add a `Notes` section after completed machines for failure reasons, anomalies, and other operator-relevant context. + - Keep completed machine lines in the form `machine-name - STATUS` and avoid appending long explanations inline. + +## Run Learning: 2026-03-12 (Add suse15.0 machine to blacklist) +- Observed requirement: + - `atvm144-suse15.0` must be excluded from automation runs because it crashes while creating the migration session. +- Action for future runs: + - Add `atvm144-suse15.0` to the maintained blacklist. + - Record the reason as `CRASHES WHEN CREATING MIGRATION SESSION - BUG`. + - Include it in reusable `--exclude_partial_match` command examples.