anthony.wen/cds-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update ATVM automation blacklist and status formatting guidance

Browse Source 
Add atvm144-suse15.0 to the ATVM automation blacklist because it crashes when creating a migration session.

Update maintained exclude examples to include the new blacklist entry.

Tighten status reporting guidance to require one machine per line.

Add a Notes section for failure reasons and operator-facing context.

Record the new run learnings in atvm-automation-runs.md.
This commit is contained in:
anthony.wen
2026-03-12 11:03:08 -04:00
parent 48f189792b
commit e1a551cdb2
4 changed files with 45 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
atvm/AGENTS.md
View File

@@ -97,27 +97,30 @@ Use `atvm-automation-examples.md` as the common options/command reference.
When the operator asks for run status, report in this order: When the operator asks for run status, report in this order:
1. Heading/title using the run `build_name`. 1. Heading/title using the run `build_name`.
2. Completed machines with machine name first and status second for each machine. 2. Completed machines with machine name first and status second for each machine.
3. Skipped machines with reason. 3. Notes.
4. Remaining machines still to run. 4. Skipped machines with reason.
5. Summary counts for finished, passed, failed, and skipped machines. 5. Remaining machines still to run.
6. Timing details: 6. Summary counts for finished, passed, failed, and skipped machines.
7. Timing details:
- start time - start time
- end time if complete - end time if complete
- total run time if complete, or elapsed run time if still running - total run time if complete, or elapsed run time if still running
- quickest completed test runtime - quickest completed test runtime
- longest completed test runtime - longest completed test runtime
- average completed test runtime - average completed test runtime
7. Estimated completion time. 8. Estimated completion time.
Status details: Status details:
- Use the live run log on the automation VM when available. - Use the live run log on the automation VM when available.
- Use the run `build_name` as the heading/title when available. - Use the run `build_name` as the heading/title when available.
- Format every machine entry as `machine-name - STATUS`. - Format every machine entry as `machine-name - STATUS`.
- Put each machine on its own line; do not combine multiple machine statuses on one line.
- Put failure reasons, oddities, and operator-facing context in a dedicated `Notes` section.
- Show blacklisted machines under skipped machines when they are part of the requested scope. - Show blacklisted machines under skipped machines when they are part of the requested scope.
- Show in-progress machines under remaining machines as `RUNNING`. - Show in-progress machines under remaining machines as `RUNNING`.
- Show not-yet-started machines as `NOT STARTED`. - Show not-yet-started machines as `NOT STARTED`.
- Use completed spec results already recorded in the log to determine machine pass/fail state. - Use completed spec results already recorded in the log to determine machine pass/fail state.
- For failed machines, include the failure reason from the run log in the status output. - For failed machines, mark the machine `FAIL` in the completed list and include the failure reason in `Notes`.
- Include start time in status output when it can be derived from the log. - Include start time in status output when it can be derived from the log.
- Include end time and total runtime for completed runs, or elapsed runtime for active runs. - Include end time and total runtime for completed runs, or elapsed runtime for active runs.
- Include quickest completed test runtime, longest completed test runtime, and average completed test runtime under timing details when they can be derived from the log. - Include quickest completed test runtime, longest completed test runtime, and average completed test runtime under timing details when they can be derived from the log.
@@ -131,6 +134,9 @@ CMC install blacklist (`BLACKLISTED: CMC INSTALL - CAN'T COMPILE`):
- `atvm41-redhat6.0` - `atvm41-redhat6.0`
- `atvm73-oracle6.0` - `atvm73-oracle6.0`
Crash blacklist (`BLACKLISTED: CRASHES WHEN CREATING MIGRATION SESSION - BUG`):
- `atvm144-suse15.0`
Support-request blacklist (`BLACKLISTED: SUPPORT REQUEST - WAITING`): Support-request blacklist (`BLACKLISTED: SUPPORT REQUEST - WAITING`):
- `atvm113-debian9.0.0` - `atvm113-debian9.0.0`
- `atvm115-debian9.1.0` - `atvm115-debian9.1.0`

6
atvm/atvm-automation-examples.md
View File

@@ -80,18 +80,18 @@ python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name
### Randomized reboot sanity ### Randomized reboot sanity
```bash ```bash
python3 cmc-templates.py --template cmc-reboot --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --test_partition --integration_type pure --use_specified_plugin fc --randomize 1 --exclude_partial_match suse15.0 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0 --wait_for_power_on 120; \ python3 cmc-templates.py --template cmc-reboot --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --test_partition --integration_type pure --use_specified_plugin fc --randomize 1 --exclude_partial_match suse15.0 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0 --wait_for_power_on 120; \
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name sanity-reboot-iscsi python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name sanity-reboot-iscsi
``` ```
### Randomized e2e sanity ### Randomized e2e sanity
```bash ```bash
python3 cmc-templates.py --template cmc-e2e --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --test_partition --integration_type pure --use_specified_plugin both --randomize 1 --exclude_partial_match suse15.0 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0; \ python3 cmc-templates.py --template cmc-e2e --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --test_partition --integration_type pure --use_specified_plugin both --randomize 1 --exclude_partial_match suse15.0 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0; \
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name sanity-e2e python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name sanity-e2e
``` ```
### Randomized systemOS sanity ### Randomized systemOS sanity
```bash ```bash
python3 cmc-templates.py --template cmc-systemOS --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --randomize 1 --exclude_partial_match suse15.0 fedora34 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0; \ python3 cmc-templates.py --template cmc-systemOS --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --randomize 1 --exclude_partial_match suse15.0 fedora34 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0; \
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name sanity-systemOS python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name sanity-systemOS
``` ```

20
atvm/atvm-automation-guide.md
View File

@@ -83,6 +83,9 @@ Permanently blacklisted because CMC cannot compile:
- `atvm41-redhat6.0` - `atvm41-redhat6.0`
- `atvm73-oracle6.0` - `atvm73-oracle6.0`
Temporarily blacklisted because the run crashes when creating a migration session:
- `atvm144-suse15.0`
Temporarily blacklisted while support requests are waiting: Temporarily blacklisted while support requests are waiting:
- `atvm113-debian9.0.0` - `atvm113-debian9.0.0`
- `atvm115-debian9.1.0` - `atvm115-debian9.1.0`
@@ -93,7 +96,7 @@ Temporarily blacklisted until re-created:
- `atvm157-debian13.0.0` - `atvm157-debian13.0.0`
Preferred exclude list: Preferred exclude list:
- `--exclude_partial_match atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0` - `--exclude_partial_match atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0`
## Running-Automation Check (Mandatory) ## Running-Automation Check (Mandatory)
Before any new automation request: Before any new automation request:
@@ -134,22 +137,25 @@ When asked for one VM or a VM set:
When the operator asks for the status of an ATVM automation run, report in this order: When the operator asks for the status of an ATVM automation run, report in this order:
1. Heading/title using the run `build_name`. 1. Heading/title using the run `build_name`.
2. Completed machines with machine name first and status second for each machine. 2. Completed machines with machine name first and status second for each machine.
3. Skipped machines with reason. 3. Notes.
4. Remaining machines still to run. 4. Skipped machines with reason.
5. Summary counts for finished, passed, failed, and skipped machines. 5. Remaining machines still to run.
6. Timing details: 6. Summary counts for finished, passed, failed, and skipped machines.
7. Timing details:
- start time - start time
- end time if complete - end time if complete
- total run time if complete, or elapsed run time if still running - total run time if complete, or elapsed run time if still running
- quickest completed test runtime - quickest completed test runtime
- longest completed test runtime - longest completed test runtime
- average completed test runtime - average completed test runtime
7. Estimated completion time. 8. Estimated completion time.
Status-report expectations: Status-report expectations:
- Use the live automation VM state when available. - Use the live automation VM state when available.
- Derive the heading/title from the run `build_name` when available. - Derive the heading/title from the run `build_name` when available.
- Format every machine entry as `machine-name - STATUS`. - Format every machine entry as `machine-name - STATUS`.
- Put each machine on its own line; never combine multiple machines into one paragraph or comma-separated line.
- Use a separate `Notes` section for failure reasons, anomalies, or operator-relevant context rather than cramming those details into the completed-machine list.
- For categorized runs, reconstruct the whole run across all category batches; do not treat the current live category batch as the full run scope. - For categorized runs, reconstruct the whole run across all category batches; do not treat the current live category batch as the full run scope.
- Derive completed-machine status from completed spec results already written during the same run. - Derive completed-machine status from completed spec results already written during the same run.
- Parse all same-run `test-result-*.xml` files, not only machine-named `test-result-atvm*.xml` files. - Parse all same-run `test-result-*.xml` files, not only machine-named `test-result-atvm*.xml` files.
@@ -168,6 +174,6 @@ Status-report expectations:
- If a machine is currently in progress, show it under remaining machines as `RUNNING`. - If a machine is currently in progress, show it under remaining machines as `RUNNING`.
- If a machine has not started yet, show it under remaining machines as `NOT STARTED`. - If a machine has not started yet, show it under remaining machines as `NOT STARTED`.
- If no failures are present in completed spec results, report those completed machines as `PASS`. - If no failures are present in completed spec results, report those completed machines as `PASS`.
- If a completed spec result shows a failure, report that machine as `FAIL` and include the failure reason from the run log. - If a completed spec result shows a failure, report that machine as `FAIL` in the completed list and put the failure reason in `Notes`.
- Base the completion estimate on the full remaining machine count and recent per-machine runtime visible in the run log. - Base the completion estimate on the full remaining machine count and recent per-machine runtime visible in the run log.
- Make the estimate explicitly refer to completion of the entire remaining run, not only the current machine/spec. - Make the estimate explicitly refer to completion of the entire remaining run, not only the current machine/spec.

17
atvm/atvm-automation-runs.md
View File

@@ -74,3 +74,20 @@ This file stores run-specific examples only when a run produced a new learning r
- Extract the machine name from XML contents such as `testsuite file=`, `testsuite name=`, or `testcase name=` when the filename does not include the machine name. - Extract the machine name from XML contents such as `testsuite file=`, `testsuite name=`, or `testcase name=` when the filename does not include the machine name.
- Treat `check-xml-files.ts` XML outputs as bookkeeping steps, not machine results. - Treat `check-xml-files.ts` XML outputs as bookkeeping steps, not machine results.
- Prefer the most recently written same-run XML per machine when multiple XML files exist for that machine. - Prefer the most recently written same-run XML per machine when multiple XML files exist for that machine.
## Run Learning: 2026-03-12 (Status output must be one machine per line with notes separated)
- Observed requirement:
- Listing multiple completed machines on one line makes run status harder to scan and does not meet the expected reporting format.
- Failure reasons and extra context should be separated from the machine status list so the list stays clean.
- Action for future runs:
- Under completed, skipped, and remaining sections, put exactly one machine status on each line.
- Add a `Notes` section after completed machines for failure reasons, anomalies, and other operator-relevant context.
- Keep completed machine lines in the form `machine-name - STATUS` and avoid appending long explanations inline.
## Run Learning: 2026-03-12 (Add suse15.0 machine to blacklist)
- Observed requirement:
- `atvm144-suse15.0` must be excluded from automation runs because it crashes while creating the migration session.
- Action for future runs:
- Add `atvm144-suse15.0` to the maintained blacklist.
- Record the reason as `CRASHES WHEN CREATING MIGRATION SESSION - BUG`.
- Include it in reusable `--exclude_partial_match` command examples.