Require watcher startup before ATVM runner launch

- update the ATVM automation guide to make watcher-first launch order explicit whenever the watcher is approved
- update the ATVM AGENTS rules so the runner is never started before the watcher for watcher-backed runs
- add a 2026-03-27 run learning documenting that the watcher helper can delete the live runner log if the runner starts first

atvm/AGENTS.md

@@ -62,6 +62,8 @@ This file defines how to operate and maintain the ATVM workspace in `/home/aw/co
 - For ATVM run approvals, treat `approve` as run-without-watcher and `approve with watcher` as run-with-watcher.
 - After `cmc-templates.py`, always verify that the generated spec files and the config `specPattern` still include every requested VM before starting `run-sorry-cypress.py`.
 - If any requested VM is missing after template generation, stop and report the mismatch instead of launching the runner.
+- When the watcher is requested, start the watcher before `run-sorry-cypress.py`.
+- Do not start the runner before the watcher, because the watcher helper clears stale `/tmp/<build-name>.log` and can delete the fresh live runner log if the runner starts first.
 - For host-level test detail and failed-test investigation, use `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter`, especially `logs/`, `xml/`, and `mochawesome/`.
 - If the operator asks for ATVM run status without mentioning Mattermost, respond locally only and do not post externally.
 - If the operator asks to send ATVM run status to Mattermost, use `MATTERMOST_ATVM_WEBHOOK` and `MATTERMOST_ATVM_CHANNEL` from `/home/aw/code/cds/.env.credentials.local` by default and send the final status only after the run has fully completed, whether the run passed or failed.

atvm/docs/automation/guide.md

@@ -81,7 +81,8 @@ Typical sequence:
 4. Run `cmc-templates.py` with the approved options.
 5. Wait for `cmc-templates.py` to fully finish and confirm success.
 6. Verify the generated `.ts` files and the config `specPattern` include every requested VM before starting the runner.
-7. Run `run-sorry-cypress.py` with the matching approved config and build name.
+7. If the watcher is approved, start the watcher before launching `run-sorry-cypress.py`.
+8. Run `run-sorry-cypress.py` with the matching approved config and build name.
 
 ## Config File / Gold Disk Mapping
 - `cypress.atvm-config-gold.ts` -> Gold Disk 1
@@ -173,6 +174,8 @@ Before any new automation request:
 14. If the operator asks to change plugin, config, filters, build name, Gold Disk, or scope after commands are shown, discard the old plan, show the revised commands, and wait for new approval.
 15. If monitoring was not requested, report immediate success/failure for each command.
 16. If monitoring was requested, keep monitoring until completion and report final outcome.
+17. When the watcher is requested, launch the watcher before `run-sorry-cypress.py`.
+18. Do not start the runner before the watcher, because the watcher helper clears stale `/tmp/<build-name>.log` and can delete the fresh live runner log if the runner starts first.
 
 ## Requested Test Style
 When asked for one VM or a VM set:

atvm/docs/automation/run-learnings.md

@@ -211,3 +211,13 @@ This file stores run-specific examples only when a run produced a new learning r
     - `9. Trigger cmotion and do I/O test before actual cutover`
     - `...`
     - `22. Power off`
+
+## Run Learning: 2026-03-27 (Start watcher before runner when watcher is requested)
+- Observed failure mode:
+  - Starting `run-sorry-cypress.py` before the watcher can race with the watcher helper's stale-log cleanup.
+  - The watcher helper clears stale `/tmp/<build-name>.log` before startup.
+  - If the runner has already opened the new log, the helper can delete that live log path, leaving the watcher unable to read the run by filename.
+- Action for future runs:
+  - When the watcher is approved, start the watcher before `run-sorry-cypress.py`.
+  - Keep the order as: template generation, verification, watcher start, runner start.
+  - Do not launch the runner first when the watcher is part of the approved command set.