Update ATVM automation blacklist and status formatting guidance
Add atvm144-suse15.0 to the ATVM automation blacklist because it crashes when creating a migration session. Update maintained exclude examples to include the new blacklist entry. Tighten status reporting guidance to require one machine per line. Add a Notes section for failure reasons and operator-facing context. Record the new run learnings in atvm-automation-runs.md.
This commit is contained in:
@@ -97,27 +97,30 @@ Use `atvm-automation-examples.md` as the common options/command reference.
|
||||
When the operator asks for run status, report in this order:
|
||||
1. Heading/title using the run `build_name`.
|
||||
2. Completed machines with machine name first and status second for each machine.
|
||||
3. Skipped machines with reason.
|
||||
4. Remaining machines still to run.
|
||||
5. Summary counts for finished, passed, failed, and skipped machines.
|
||||
6. Timing details:
|
||||
3. Notes.
|
||||
4. Skipped machines with reason.
|
||||
5. Remaining machines still to run.
|
||||
6. Summary counts for finished, passed, failed, and skipped machines.
|
||||
7. Timing details:
|
||||
- start time
|
||||
- end time if complete
|
||||
- total run time if complete, or elapsed run time if still running
|
||||
- quickest completed test runtime
|
||||
- longest completed test runtime
|
||||
- average completed test runtime
|
||||
7. Estimated completion time.
|
||||
8. Estimated completion time.
|
||||
|
||||
Status details:
|
||||
- Use the live run log on the automation VM when available.
|
||||
- Use the run `build_name` as the heading/title when available.
|
||||
- Format every machine entry as `machine-name - STATUS`.
|
||||
- Put each machine on its own line; do not combine multiple machine statuses on one line.
|
||||
- Put failure reasons, oddities, and operator-facing context in a dedicated `Notes` section.
|
||||
- Show blacklisted machines under skipped machines when they are part of the requested scope.
|
||||
- Show in-progress machines under remaining machines as `RUNNING`.
|
||||
- Show not-yet-started machines as `NOT STARTED`.
|
||||
- Use completed spec results already recorded in the log to determine machine pass/fail state.
|
||||
- For failed machines, include the failure reason from the run log in the status output.
|
||||
- For failed machines, mark the machine `FAIL` in the completed list and include the failure reason in `Notes`.
|
||||
- Include start time in status output when it can be derived from the log.
|
||||
- Include end time and total runtime for completed runs, or elapsed runtime for active runs.
|
||||
- Include quickest completed test runtime, longest completed test runtime, and average completed test runtime under timing details when they can be derived from the log.
|
||||
@@ -131,6 +134,9 @@ CMC install blacklist (`BLACKLISTED: CMC INSTALL - CAN'T COMPILE`):
|
||||
- `atvm41-redhat6.0`
|
||||
- `atvm73-oracle6.0`
|
||||
|
||||
Crash blacklist (`BLACKLISTED: CRASHES WHEN CREATING MIGRATION SESSION - BUG`):
|
||||
- `atvm144-suse15.0`
|
||||
|
||||
Support-request blacklist (`BLACKLISTED: SUPPORT REQUEST - WAITING`):
|
||||
- `atvm113-debian9.0.0`
|
||||
- `atvm115-debian9.1.0`
|
||||
|
||||
@@ -80,18 +80,18 @@ python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name
|
||||
|
||||
### Randomized reboot sanity
|
||||
```bash
|
||||
python3 cmc-templates.py --template cmc-reboot --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --test_partition --integration_type pure --use_specified_plugin fc --randomize 1 --exclude_partial_match suse15.0 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0 --wait_for_power_on 120; \
|
||||
python3 cmc-templates.py --template cmc-reboot --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --test_partition --integration_type pure --use_specified_plugin fc --randomize 1 --exclude_partial_match suse15.0 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0 --wait_for_power_on 120; \
|
||||
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name sanity-reboot-iscsi
|
||||
```
|
||||
|
||||
### Randomized e2e sanity
|
||||
```bash
|
||||
python3 cmc-templates.py --template cmc-e2e --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --test_partition --integration_type pure --use_specified_plugin both --randomize 1 --exclude_partial_match suse15.0 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0; \
|
||||
python3 cmc-templates.py --template cmc-e2e --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --test_partition --integration_type pure --use_specified_plugin both --randomize 1 --exclude_partial_match suse15.0 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0; \
|
||||
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name sanity-e2e
|
||||
```
|
||||
|
||||
### Randomized systemOS sanity
|
||||
```bash
|
||||
python3 cmc-templates.py --template cmc-systemOS --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --randomize 1 --exclude_partial_match suse15.0 fedora34 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0; \
|
||||
python3 cmc-templates.py --template cmc-systemOS --ignore_force_shutdown --config_file_path ./cypress.atvm-config.ts --randomize 1 --exclude_partial_match suse15.0 fedora34 atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0; \
|
||||
python3 ./run-sorry-cypress.py --config_file cypress.atvm-config.ts --build_name sanity-systemOS
|
||||
```
|
||||
|
||||
@@ -83,6 +83,9 @@ Permanently blacklisted because CMC cannot compile:
|
||||
- `atvm41-redhat6.0`
|
||||
- `atvm73-oracle6.0`
|
||||
|
||||
Temporarily blacklisted because the run crashes when creating a migration session:
|
||||
- `atvm144-suse15.0`
|
||||
|
||||
Temporarily blacklisted while support requests are waiting:
|
||||
- `atvm113-debian9.0.0`
|
||||
- `atvm115-debian9.1.0`
|
||||
@@ -93,7 +96,7 @@ Temporarily blacklisted until re-created:
|
||||
- `atvm157-debian13.0.0`
|
||||
|
||||
Preferred exclude list:
|
||||
- `--exclude_partial_match atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0`
|
||||
- `--exclude_partial_match atvm6-centos6.0 atvm41-redhat6.0 atvm73-oracle6.0 atvm144-suse15.0 atvm113-debian9.0.0 atvm115-debian9.1.0 atvm116-debian9.2.0 atvm156-debian9.3.0 atvm157-debian13.0.0`
|
||||
|
||||
## Running-Automation Check (Mandatory)
|
||||
Before any new automation request:
|
||||
@@ -134,22 +137,25 @@ When asked for one VM or a VM set:
|
||||
When the operator asks for the status of an ATVM automation run, report in this order:
|
||||
1. Heading/title using the run `build_name`.
|
||||
2. Completed machines with machine name first and status second for each machine.
|
||||
3. Skipped machines with reason.
|
||||
4. Remaining machines still to run.
|
||||
5. Summary counts for finished, passed, failed, and skipped machines.
|
||||
6. Timing details:
|
||||
3. Notes.
|
||||
4. Skipped machines with reason.
|
||||
5. Remaining machines still to run.
|
||||
6. Summary counts for finished, passed, failed, and skipped machines.
|
||||
7. Timing details:
|
||||
- start time
|
||||
- end time if complete
|
||||
- total run time if complete, or elapsed run time if still running
|
||||
- quickest completed test runtime
|
||||
- longest completed test runtime
|
||||
- average completed test runtime
|
||||
7. Estimated completion time.
|
||||
8. Estimated completion time.
|
||||
|
||||
Status-report expectations:
|
||||
- Use the live automation VM state when available.
|
||||
- Derive the heading/title from the run `build_name` when available.
|
||||
- Format every machine entry as `machine-name - STATUS`.
|
||||
- Put each machine on its own line; never combine multiple machines into one paragraph or comma-separated line.
|
||||
- Use a separate `Notes` section for failure reasons, anomalies, or operator-relevant context rather than cramming those details into the completed-machine list.
|
||||
- For categorized runs, reconstruct the whole run across all category batches; do not treat the current live category batch as the full run scope.
|
||||
- Derive completed-machine status from completed spec results already written during the same run.
|
||||
- Parse all same-run `test-result-*.xml` files, not only machine-named `test-result-atvm*.xml` files.
|
||||
@@ -168,6 +174,6 @@ Status-report expectations:
|
||||
- If a machine is currently in progress, show it under remaining machines as `RUNNING`.
|
||||
- If a machine has not started yet, show it under remaining machines as `NOT STARTED`.
|
||||
- If no failures are present in completed spec results, report those completed machines as `PASS`.
|
||||
- If a completed spec result shows a failure, report that machine as `FAIL` and include the failure reason from the run log.
|
||||
- If a completed spec result shows a failure, report that machine as `FAIL` in the completed list and put the failure reason in `Notes`.
|
||||
- Base the completion estimate on the full remaining machine count and recent per-machine runtime visible in the run log.
|
||||
- Make the estimate explicitly refer to completion of the entire remaining run, not only the current machine/spec.
|
||||
|
||||
@@ -74,3 +74,20 @@ This file stores run-specific examples only when a run produced a new learning r
|
||||
- Extract the machine name from XML contents such as `testsuite file=`, `testsuite name=`, or `testcase name=` when the filename does not include the machine name.
|
||||
- Treat `check-xml-files.ts` XML outputs as bookkeeping steps, not machine results.
|
||||
- Prefer the most recently written same-run XML per machine when multiple XML files exist for that machine.
|
||||
|
||||
## Run Learning: 2026-03-12 (Status output must be one machine per line with notes separated)
|
||||
- Observed requirement:
|
||||
- Listing multiple completed machines on one line makes run status harder to scan and does not meet the expected reporting format.
|
||||
- Failure reasons and extra context should be separated from the machine status list so the list stays clean.
|
||||
- Action for future runs:
|
||||
- Under completed, skipped, and remaining sections, put exactly one machine status on each line.
|
||||
- Add a `Notes` section after completed machines for failure reasons, anomalies, and other operator-relevant context.
|
||||
- Keep completed machine lines in the form `machine-name - STATUS` and avoid appending long explanations inline.
|
||||
|
||||
## Run Learning: 2026-03-12 (Add suse15.0 machine to blacklist)
|
||||
- Observed requirement:
|
||||
- `atvm144-suse15.0` must be excluded from automation runs because it crashes while creating the migration session.
|
||||
- Action for future runs:
|
||||
- Add `atvm144-suse15.0` to the maintained blacklist.
|
||||
- Record the reason as `CRASHES WHEN CREATING MIGRATION SESSION - BUG`.
|
||||
- Include it in reusable `--exclude_partial_match` command examples.
|
||||
|
||||
Reference in New Issue
Block a user