Add ATVM systemd runner service

atvm/watcher-service/README.md

@@ -19,10 +19,18 @@ The watcher does not run indefinitely. It is designed for one run per service in
 
 ## Files
 
+- `atvm-runner@.service`
+  - `systemd` template unit for one runner instance per build name
 - `atvm_run_watcher.py`
   - main watcher implementation
 - `atvm-run-watcher@.service`
   - `systemd` template unit for one watcher instance per build name
+- `run-atvm-runner.sh`
+  - runner wrapper used by the `systemd` runner unit
+- `start-atvm-runner.sh`
+  - helper to write per-run runner environment data and start a runner instance
+- `cancel-atvm-runner.sh`
+  - helper to stop a runner instance
 - `start-atvm-run-watcher.sh`
   - helper to write per-run environment data and start a watcher instance
 - `cancel-atvm-run-watcher.sh`
@@ -33,6 +41,7 @@ The watcher does not run indefinitely. It is designed for one run per service in
 These are the default install targets assumed by the included unit file:
 
 - service package root: `/opt/atvm-watcher-service`
+- runner unit: `/etc/systemd/system/atvm-runner@.service`
 - watcher state root: `/var/lib/atvm-run-watcher`
 - controller ATVM automation root: `/root/cdc-e2e-cyp-12.17.4`
 - watcher environment file: `/etc/atvm-run-watcher.env`
@@ -46,9 +55,9 @@ Each watcher instance is tied to one requested build name.
 
 Typical workflow:
 
-1. Launch the ATVM run.
-2. Start the watcher for that run.
-3. The watcher polls the run log, process state, and `cmcReporter` artifacts.
+1. Start the watcher for that run.
+2. Start the runner service for that run.
+3. The watcher polls the runner log, process state, and `cmcReporter` artifacts.
    - before starting, the helper resets any prior watcher state for the same requested build name so stale cancellation or posted markers do not leak into a new run
 4. For non-categorized runs, when the run reaches a terminal state:
    - `COMPLETED` or `FAILED`
@@ -88,9 +97,18 @@ Optional metadata for better status formatting:
 - `ATVM_WATCHER_SCOPE_DESCRIPTION`
 - `ATVM_WATCHER_CATEGORIZED`
 
+Runner environment required per run:
+
+- `ATVM_RUNNER_COMMAND`
+
+Runner environment optional per run:
+
+- `ATVM_RUNNER_WORKDIR`
+- `ATVM_RUNNER_LOG`
+
 ## Start Example
 
-This helper writes a per-run environment file and starts the matching instance:
+These helpers write per-run environment files and start the matching instances:
 
 ```bash
 ./start-atvm-run-watcher.sh \
@@ -103,6 +121,10 @@ This helper writes a per-run environment file and starts the matching instance:
   --integration-plugin "pure with fc" \
   --categorize \
   --scope-description "mixed Linux and Windows FC E2E validation on the gold datastore set"
+
+./start-atvm-runner.sh \
+  --build-name e2e-redhat9.6-ubuntu24.04-w2k25-fc \
+  --runner-command "python3 ./run-sorry-cypress.py --config_file cypress.atvm-config-gold.ts --build_name e2e-redhat9.6-ubuntu24.04-w2k25-fc --categorize"
 ```
 
 That results in:
@@ -111,6 +133,7 @@ That results in:
   - `/var/lib/atvm-run-watcher/e2e-redhat9.6-ubuntu24.04-w2k25-fc`
 - service instance:
   - `atvm-run-watcher@e2e-redhat9.6-ubuntu24.04-w2k25-fc.service`
+  - `atvm-runner@e2e-redhat9.6-ubuntu24.04-w2k25-fc.service`
 
 The helper also:
 
@@ -126,9 +149,16 @@ The helper also:
 
 This writes a cancellation marker, updates `state.json` to `CANCELLED`, and stops the watcher instance. The watcher will not send Mattermost results for that run.
 
+Runner cancel example:
+
+```bash
+./cancel-atvm-runner.sh --build-name e2e-redhat9.6-ubuntu24.04-w2k25-fc
+```
+
 ## Notes
 
 - The watcher uses the same ATVM status layout documented in `atvm/docs/automation/status-template.md`.
+- Prefer the controller-local `atvm-runner@...` service over ad hoc `nohup` or detached SSH launch patterns for `run-sorry-cypress.py`.
 - Kernel values are resolved from `atvm/inventory/vm-inventory.md`.
 - Categorized execution is treated as sequential grouped ATVM sub-runs, not as one parent run with internal phases.
 - In categorized mode, the watcher writes per-subrun state under `subruns/` and posts each completed grouped run separately.