cds-ai/atvm/watcher-service

ATVM Watcher Service

This folder contains a per-run ATVM watcher service package that is intended to be reviewed locally first and installed on the ATVM Cypress controller later only when explicitly requested.

Purpose

Watch an ATVM automation request until it reaches a terminal state, then:

• for non-categorized runs:
  • post one final status to Mattermost if the run state is COMPLETED or FAILED
• for categorized runs:
  • detect each sequential categorized sub-run
  • post one final status per completed categorized sub-run if that grouped run state is COMPLETED or FAILED
• verify each Mattermost post succeeded
• write durable watcher state
• exit cleanly so the service stops

The watcher does not run indefinitely. It is designed for one run per service instance.

Files

• atvm_run_watcher.py
  • main watcher implementation
• atvm-run-watcher@.service
  • systemd template unit for one watcher instance per build name
• start-atvm-run-watcher.sh
  • helper to write per-run environment data and start a watcher instance
• cancel-atvm-run-watcher.sh
  • helper to mark a run cancelled and stop the watcher instance

Intended Controller Paths

These are the default install targets assumed by the included unit file:

• service package root: /opt/atvm-watcher-service
• watcher state root: /var/lib/atvm-run-watcher
• controller ATVM automation root: /root/cdc-e2e-cyp-12.17.4
• watcher environment file: /etc/atvm-run-watcher.env

Use /opt/atvm-watcher-service as the controller install root for future installs and reinstalls. Do not treat /root/atvm-watcher-service as the preferred long-term install location.

Per-Run Behavior

Each watcher instance is tied to one requested build name.

Typical workflow:

1. Launch the ATVM run.
2. Start the watcher for that run.
3. The watcher polls the run log, process state, and cmcReporter artifacts.
   • before starting, the helper resets any prior watcher state for the same requested build name so stale cancellation or posted markers do not leak into a new run
4. For non-categorized runs, when the run reaches a terminal state:
   • COMPLETED or FAILED
     • build the final ATVM status
     • send the status to Mattermost
     • verify Mattermost returned ok
     • mark the run as posted
     • exit
   • CANCELLED, TERMINATED, HUNG, or UNKNOWN
     • do not post
     • mark the final state
     • exit
5. For categorized runs:
   • detect each grouped sub-run in sequence from the parent run log
   • wait for that grouped sub-run to finish
   • send one Mattermost post for that grouped sub-run if it reached COMPLETED or FAILED
   • keep the watcher alive while the parent categorized runner or related child Cypress process is still active
   • do not treat one completed grouped sub-run as proof that the whole parent request is finished
   • continue to the next grouped sub-run
   • exit after the parent request reaches a terminal state

Required Environment

The service expects the local credentials file values to be made available on the controller through the service environment:

• MATTERMOST_ATVM_WEBHOOK
• MATTERMOST_ATVM_CHANNEL

Optional metadata for better status formatting:

• ATVM_WATCHER_TEMPLATE
• ATVM_WATCHER_CONFIG_FAMILY
• ATVM_WATCHER_MIGRATION_STYLE
• ATVM_WATCHER_INTEGRATION_PLUGIN
• ATVM_WATCHER_SCOPE_DESCRIPTION
• ATVM_WATCHER_CATEGORIZED

Start Example

This helper writes a per-run environment file and starts the matching instance:

./start-atvm-run-watcher.sh \
  --build-name e2e-redhat9.6-ubuntu24.04-w2k25-fc \
  --template cmc-e2e \
  --config-family gold \
  --migration-style "ATVM end-to-end migration validation" \
  --integration-plugin "pure with fc" \
  --categorize \
  --scope-description "mixed Linux and Windows FC E2E validation on the gold datastore set"

That results in:

• state dir:
  • /var/lib/atvm-run-watcher/e2e-redhat9.6-ubuntu24.04-w2k25-fc
• service instance:
  • atvm-run-watcher@e2e-redhat9.6-ubuntu24.04-w2k25-fc.service

The helper also:

• stops any stale watcher instance for that same requested build name
• removes the old watcher state directory for that requested build name
• starts the new watcher with a clean state root for the new run

Cancel Example

./cancel-atvm-run-watcher.sh --build-name e2e-redhat9.6-ubuntu24.04-w2k25-fc

This writes a cancellation marker, updates state.json to CANCELLED, and stops the watcher instance. The watcher will not send Mattermost results for that run.

Notes

• The watcher uses the same ATVM status layout documented in atvm/docs/automation/status-template.md.
• Kernel values are resolved from atvm/inventory/vm-inventory.md.
• Categorized execution is treated as sequential grouped ATVM sub-runs, not as one parent run with internal phases.
• In categorized mode, the watcher writes per-subrun state under subruns/ and posts each completed grouped run separately.
• In categorized mode, if the child build id label does not match the host/spec actually being executed, the watcher reports the grouped run using the inferred host-based group name instead of trusting the raw child build id label.
• Best-practice controller install path: /opt/atvm-watcher-service.
• This package is local-only right now. Nothing here is installed on the controller yet.