Switch ATVM watcher status posts to MS Teams

This commit is contained in:
Anthony Wen
2026-07-27 19:07:50 -04:00
parent 67886174b3
commit 063d13d01a
8 changed files with 107 additions and 90 deletions
+15 -15
View File
@@ -10,9 +10,9 @@ Install the local watcher/runner package so the controller can:
- start one requested ATVM Cypress runner per service instance
- watch one requested ATVM run per watcher instance
- for non-categorized runs, send one final Mattermost status only for `COMPLETED` or `FAILED`
- for categorized runs, send one final Mattermost status per completed categorized sub-run/group
- suppress Mattermost posts for `CANCELLED`, `TERMINATED`, `HUNG`, and `UNKNOWN`
- for non-categorized runs, send one final MS Teams status only for `COMPLETED` or `FAILED`
- for categorized runs, send one final MS Teams status per completed categorized sub-run/group
- suppress MS Teams posts for `CANCELLED`, `TERMINATED`, `HUNG`, and `UNKNOWN`
- stop automatically after the watched run reaches a terminal state
## Controller Target Layout
@@ -64,7 +64,7 @@ The controller must have:
- `python3`
- `systemd`
- outbound network access to the Mattermost webhook
- outbound network access to the MS Teams webhook
- read access to:
- `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter`
- `/tmp/<build-name>.log`
@@ -73,8 +73,8 @@ The controller must have:
The controller needs a watcher environment file with:
- `MATTERMOST_ATVM_WEBHOOK`
- `MATTERMOST_ATVM_CHANNEL`
- `MS_TEAMS_ATVM_WEBHOOK`
- `MS_TEAMS_ATVM_CHANNEL`
Recommended file:
@@ -110,7 +110,7 @@ Recommended permissions:
- `cancel-atvm-run-watcher.sh`
4. Create `/etc/atvm-run-watcher.env`.
- add Mattermost webhook/channel
- add MS Teams webhook/channel
- keep permissions restricted
5. Install the `systemd` unit file.
@@ -134,7 +134,7 @@ Recommended permissions:
- launch a real run
- start the watcher for that build name
- if the run uses `--categorize`, also pass `--categorize` to the watcher start helper
- confirm final Mattermost delivery for a completed run
- confirm final MS Teams delivery for a completed run
- confirm categorized execution sends one post per completed grouped sub-run
- confirm the watcher stays alive between categorized grouped runs while the parent request is still active
- confirm reused parent build names do not inherit stale `cancelled.marker`, `posted.marker`, or `subruns/` state from older runs
@@ -242,7 +242,7 @@ The cancel helper should:
- write `cancelled.marker`
- update `state.json` so the final watcher state is `CANCELLED`
- stop the watcher instance
- avoid any Mattermost post for that run
- avoid any MS Teams post for that run
## Operational Notes
@@ -255,7 +255,7 @@ The cancel helper should:
- In categorized execution, the watcher must remain alive until the parent request has actually gone inactive past the grace window, even if one grouped sub-run already completed.
- The watcher exits after the run reaches a terminal state.
- The watcher writes state under `/var/lib/atvm-run-watcher/<build-name>`.
- The watcher prevents duplicate Mattermost posts by writing posted markers.
- The watcher prevents duplicate MS Teams posts by writing posted markers.
- Categorized sub-run state is written under `/var/lib/atvm-run-watcher/<build-name>/subruns/<subrun-key>/`.
## Failure Handling
@@ -263,16 +263,16 @@ The cancel helper should:
Expected terminal behavior:
- `COMPLETED`
- post to Mattermost
- verify `ok`
- post to MS Teams
- verify the webhook returned HTTP 2xx, typically `202 Accepted`
- exit
- `FAILED`
- post to Mattermost
- verify `ok`
- post to MS Teams
- verify the webhook returned HTTP 2xx, typically `202 Accepted`
- exit
- categorized `COMPLETED` / `FAILED`
- post once for that grouped sub-run
- verify `ok`
- verify the webhook returned HTTP 2xx, typically `202 Accepted`
- continue until the parent request finishes
- `CANCELLED`
- write final `CANCELLED` state to `state.json`
+10 -10
View File
@@ -7,11 +7,11 @@ This folder contains a per-run ATVM watcher service package that is intended to
Watch an ATVM automation request until it reaches a terminal state, then:
- for non-categorized runs:
- post one final status to Mattermost if the run state is `COMPLETED` or `FAILED`
- post one final status to MS Teams if the run state is `COMPLETED` or `FAILED`
- for categorized runs:
- detect each sequential categorized sub-run
- post one final status per completed categorized sub-run if that grouped run state is `COMPLETED` or `FAILED`
- verify each Mattermost post succeeded
- verify each MS Teams post succeeded
- write durable watcher state
- exit cleanly so the service stops
@@ -65,8 +65,8 @@ Typical workflow:
5. For non-categorized runs, when the run reaches a terminal state:
- `COMPLETED` or `FAILED`
- build the final ATVM status
- send the status to Mattermost
- verify Mattermost returned `ok`
- send the status to MS Teams
- verify the MS Teams workflow webhook returned HTTP 2xx, typically `202 Accepted`
- mark the run as posted
- exit
- `CANCELLED`, `TERMINATED`, `HUNG`, or `UNKNOWN`
@@ -74,9 +74,9 @@ Typical workflow:
- mark the final state
- exit
6. For categorized runs:
- detect each grouped sub-run in sequence from the parent run log
- wait for that grouped sub-run to finish
- send one Mattermost post for that grouped sub-run if it reached `COMPLETED` or `FAILED`
- detect each grouped sub-run in sequence from the parent run log
- wait for that grouped sub-run to finish
- send one MS Teams post for that grouped sub-run if it reached `COMPLETED` or `FAILED`
- keep the watcher alive while the parent categorized runner or related child Cypress process is still active
- do not treat one completed grouped sub-run as proof that the whole parent request is finished
- continue to the next grouped sub-run
@@ -86,8 +86,8 @@ Typical workflow:
The service expects the local credentials file values to be made available on the controller through the service environment:
- `MATTERMOST_ATVM_WEBHOOK`
- `MATTERMOST_ATVM_CHANNEL`
- `MS_TEAMS_ATVM_WEBHOOK`
- `MS_TEAMS_ATVM_CHANNEL`
Optional metadata for better status formatting:
@@ -168,7 +168,7 @@ The helper also:
./cancel-atvm-run-watcher.sh --build-name e2e-redhat9.6-ubuntu24.04-w2k25-fc
```
This writes a cancellation marker, updates `state.json` to `CANCELLED`, and stops the watcher instance. The watcher will not send Mattermost results for that run.
This writes a cancellation marker, updates `state.json` to `CANCELLED`, and stops the watcher instance. The watcher will not send MS Teams results for that run.
Runner cancel example:
+36 -19
View File
@@ -1627,16 +1627,37 @@ def build_status_markdown(
return "\n".join(lines)
def post_to_mattermost(text: str) -> str:
webhook = os.environ["MATTERMOST_ATVM_WEBHOOK"]
payload = {"text": text}
channel = os.environ.get("MATTERMOST_ATVM_CHANNEL")
def post_to_teams(text: str) -> str:
webhook = os.environ["MS_TEAMS_ATVM_WEBHOOK"]
payload = {
"type": "message",
"attachments": [
{
"contentType": "application/vnd.microsoft.card.adaptive",
"contentUrl": None,
"content": {
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
"type": "AdaptiveCard",
"version": "1.4",
"body": [
{
"type": "TextBlock",
"text": text,
"wrap": True,
}
],
},
}
],
}
channel = os.environ.get("MS_TEAMS_ATVM_CHANNEL")
if channel:
payload["channel"] = channel
data = json.dumps(payload).encode()
request = urllib.request.Request(webhook, data=data, headers={"Content-Type": "application/json"})
with urllib.request.urlopen(request) as response:
return response.read().decode().strip()
body = response.read().decode().strip()
return f"HTTP {response.status}" + (f": {body}" if body else "")
def sanitize_key(raw: str) -> str:
@@ -2334,13 +2355,11 @@ if __name__ == "__main__":
log_text=current_log_text,
)
print(status_text)
response = post_to_mattermost(status_text)
if response != "ok":
raise SystemExit(f"Mattermost webhook did not return ok for {subrun['display_name']}: {response!r}")
subrun_posted_marker.write_text("ok\n", encoding="utf-8")
subrun_state["mattermost_posted"] = True
subrun_state["mattermost_response"] = response
print(f"[watcher] Mattermost post confirmed for {subrun['display_name']}.")
response = post_to_teams(status_text)
subrun_posted_marker.write_text(f"{response}\n", encoding="utf-8")
subrun_state["teams_posted"] = True
subrun_state["teams_response"] = response
print(f"[watcher] Teams post confirmed for {subrun['display_name']}: {response}.")
write_state(subrun_state_file, subrun_state)
if run_state == "RUNNING":
@@ -2363,14 +2382,12 @@ if __name__ == "__main__":
print(status_text)
if not metadata.get("categorized") and run_state in {"COMPLETED", "FAILED"} and not posted_marker.exists():
response = post_to_mattermost(status_text)
if response != "ok":
raise SystemExit(f"Mattermost webhook did not return ok: {response!r}")
posted_marker.write_text("ok\n", encoding="utf-8")
state["mattermost_posted"] = True
state["mattermost_response"] = response
response = post_to_teams(status_text)
posted_marker.write_text(f"{response}\n", encoding="utf-8")
state["teams_posted"] = True
state["teams_response"] = response
write_state(state_file, state)
print(f"[watcher] Mattermost post confirmed for {build_name}.")
print(f"[watcher] Teams post confirmed for {build_name}: {response}.")
state["closed_at"] = now_utc().isoformat()
write_state(state_file, state)