anthony.wen/cds-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Improve ATVM watcher status metadata and run workflow

Browse Source
This commit is contained in:
anthony.wen
2026-04-14 07:56:53 -04:00
parent 6403fa3d42
commit a22ff8edf1
7 changed files with 91 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

33
atvm/docs/automation/guide.md
View File

@@ -104,8 +104,9 @@ Typical sequence:
5. Wait for `cmc-templates.py` to fully finish and confirm success.
6. Verify the generated `.ts` files and the config `specPattern` include every requested VM before starting the runner.
7. If the watcher is approved, make sure the controller's deployed watcher code is the intended version before relying on its posts.
8. If the watcher is approved, start the watcher before launching `run-sorry-cypress.py`.
9. Run `run-sorry-cypress.py` with the matching approved config and build name.
8. If the watcher is approved, build the watcher-start command so it automatically includes the exact approved `cmc-templates.py` command via `--template-command` and the exact approved `run-sorry-cypress.py` command via `--runner-command`.
9. If the watcher is approved, start the watcher before launching `run-sorry-cypress.py`.
10. Run `run-sorry-cypress.py` with the matching approved config and build name.
Completed-run verification sequence:
1. Read the launch log for the build.
@@ -200,19 +201,20 @@ Before any new automation request:
4. When the watcher is available, present the watcher-start command separately from the core run commands.
5. Treat `approve` as approval to execute the ATVM run and start the watcher for that build.
6. Treat `approve without watcher` as approval to execute the ATVM run without starting the watcher.
7. If the run uses `--categorize` and the watcher is requested, include `--categorize` on the watcher start command too so the watcher tracks sequential categorized sub-runs correctly.
8. Run only approved command(s), no extra options and no silent substitutions.
9. When both template generation and the Cypress runner are requested, run them sequentially, not in parallel.
10. Do not launch `run-sorry-cypress.py` until `cmc-templates.py` has exited successfully and finished updating the intended config/spec files.
11. After `cmc-templates.py`, always verify that the generated spec files on disk and the config `specPattern` both contain the full requested VM set before launching `run-sorry-cypress.py`.
12. If any requested VM is missing from the generated files or `specPattern`, stop and report the mismatch instead of launching the runner.
13. Treat displayed commands as a review gate: do not execute either command until the operator has had a chance to review them and explicitly approve.
14. If the operator asks to change plugin, config, filters, build name, Gold Disk, or scope after commands are shown, discard the old plan, show the revised commands, and wait for new approval.
15. If the planned command is `cmc-reboot` with `--use_specified_plugin both`, add the FC+iSCSI timing warning to the review message and require explicit confirmation that `both` is intended before execution.
16. If monitoring was not requested, report immediate success/failure for each command.
17. If monitoring was requested, keep monitoring until completion and report final outcome.
18. When the watcher is requested, launch the watcher before `run-sorry-cypress.py`.
19. Do not start the runner before the watcher, because the watcher helper clears stale `/tmp/<build-name>.log` and can delete the fresh live runner log if the runner starts first.
7. When the watcher is requested, the displayed watcher-start command must automatically include the exact approved `cmc-templates.py` command via `--template-command` and the exact approved `run-sorry-cypress.py` command via `--runner-command`.
8. If the run uses `--categorize` and the watcher is requested, include `--categorize` on the watcher start command too so the watcher tracks sequential categorized sub-runs correctly.
9. Run only approved command(s), no extra options and no silent substitutions.
10. When both template generation and the Cypress runner are requested, run them sequentially, not in parallel.
11. Do not launch `run-sorry-cypress.py` until `cmc-templates.py` has exited successfully and finished updating the intended config/spec files.
12. After `cmc-templates.py`, always verify that the generated spec files on disk and the config `specPattern` both contain the full requested VM set before launching `run-sorry-cypress.py`.
13. If any requested VM is missing from the generated files or `specPattern`, stop and report the mismatch instead of launching the runner.
14. Treat displayed commands as a review gate: do not execute either command until the operator has had a chance to review them and explicitly approve.
15. If the operator asks to change plugin, config, filters, build name, Gold Disk, or scope after commands are shown, discard the old plan, show the revised commands, and wait for new approval.
16. If the planned command is `cmc-reboot` with `--use_specified_plugin both`, add the FC+iSCSI timing warning to the review message and require explicit confirmation that `both` is intended before execution.
17. If monitoring was not requested, report immediate success/failure for each command.
18. If monitoring was requested, keep monitoring until completion and report final outcome.
19. When the watcher is requested, launch the watcher before `run-sorry-cypress.py`.
20. Do not start the runner before the watcher, because the watcher helper clears stale `/tmp/<build-name>.log` and can delete the fresh live runner log if the runner starts first.
## Requested Test Style
When asked for one VM or a VM set:
@@ -276,6 +278,7 @@ Status-report expectations:
- Order the status sections as `SUMMARY:`, `HOSTS:`, `TIMING:`, `COVERAGE:`, `TEST FLOW:`, `FAILURE NOTES:`, then `NOTES:`.
- Keep `NOTES:` focused on non-failure operator-facing value such as the Currents run URL, real anomalies unrelated to the direct failure text, or material fallback behavior.
- Include the exact `cmc-templates.py` command used to trigger the ATVM automation run in `NOTES:`, without the outer `sshpass`/`ssh` wrapper and without trimming it.
- Include the exact `run-sorry-cypress.py` command used to launch the ATVM automation run in `NOTES:`, without the outer `sshpass`/`ssh` wrapper and without trimming it.
- Do not include generic watcher bookkeeping messages in `NOTES:` such as artifact-detection confirmations.
- Do not include internal watcher fallback notes in `NOTES:` such as `check-xml-files.ts` validation confirmations or reporter-artifact recovery details.
- The `HOSTS:` table includes `Host`, `Kernel`, `Status`, and `Detail` columns in that order.

1
atvm/docs/automation/status-template.md
View File

@@ -79,6 +79,7 @@ Use this as the default ATVM automation run-status template for:
- Put broader non-failure context under `NOTES:`.
- When available, put the persistent Currents run URL in `NOTES:` so operators can open the exact recorded run directly.
- Include the exact `cmc-templates.py` command used to trigger the run in `NOTES:`, without the outer `sshpass`/`ssh` wrapper.
- Include the exact `run-sorry-cypress.py` command used to launch the run in `NOTES:`, without the outer `sshpass`/`ssh` wrapper.
- Keep `FAILURE NOTES:` limited to detailed per-host error excerpts.
- Keep `NOTES:` limited to meaningful non-failure operator-facing items such as the Currents link, real anomalies, or important fallback behavior.
- Do not include generic watcher bookkeeping lines in `NOTES:` such as "run artifacts were detected" or "final reporting artifacts were detected."