Switch ATVM watcher status posts to MS Teams
This commit is contained in:
@@ -10,9 +10,9 @@ Install the local watcher/runner package so the controller can:
|
||||
|
||||
- start one requested ATVM Cypress runner per service instance
|
||||
- watch one requested ATVM run per watcher instance
|
||||
- for non-categorized runs, send one final Mattermost status only for `COMPLETED` or `FAILED`
|
||||
- for categorized runs, send one final Mattermost status per completed categorized sub-run/group
|
||||
- suppress Mattermost posts for `CANCELLED`, `TERMINATED`, `HUNG`, and `UNKNOWN`
|
||||
- for non-categorized runs, send one final MS Teams status only for `COMPLETED` or `FAILED`
|
||||
- for categorized runs, send one final MS Teams status per completed categorized sub-run/group
|
||||
- suppress MS Teams posts for `CANCELLED`, `TERMINATED`, `HUNG`, and `UNKNOWN`
|
||||
- stop automatically after the watched run reaches a terminal state
|
||||
|
||||
## Controller Target Layout
|
||||
@@ -64,7 +64,7 @@ The controller must have:
|
||||
|
||||
- `python3`
|
||||
- `systemd`
|
||||
- outbound network access to the Mattermost webhook
|
||||
- outbound network access to the MS Teams webhook
|
||||
- read access to:
|
||||
- `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter`
|
||||
- `/tmp/<build-name>.log`
|
||||
@@ -73,8 +73,8 @@ The controller must have:
|
||||
|
||||
The controller needs a watcher environment file with:
|
||||
|
||||
- `MATTERMOST_ATVM_WEBHOOK`
|
||||
- `MATTERMOST_ATVM_CHANNEL`
|
||||
- `MS_TEAMS_ATVM_WEBHOOK`
|
||||
- `MS_TEAMS_ATVM_CHANNEL`
|
||||
|
||||
Recommended file:
|
||||
|
||||
@@ -110,7 +110,7 @@ Recommended permissions:
|
||||
- `cancel-atvm-run-watcher.sh`
|
||||
|
||||
4. Create `/etc/atvm-run-watcher.env`.
|
||||
- add Mattermost webhook/channel
|
||||
- add MS Teams webhook/channel
|
||||
- keep permissions restricted
|
||||
|
||||
5. Install the `systemd` unit file.
|
||||
@@ -134,7 +134,7 @@ Recommended permissions:
|
||||
- launch a real run
|
||||
- start the watcher for that build name
|
||||
- if the run uses `--categorize`, also pass `--categorize` to the watcher start helper
|
||||
- confirm final Mattermost delivery for a completed run
|
||||
- confirm final MS Teams delivery for a completed run
|
||||
- confirm categorized execution sends one post per completed grouped sub-run
|
||||
- confirm the watcher stays alive between categorized grouped runs while the parent request is still active
|
||||
- confirm reused parent build names do not inherit stale `cancelled.marker`, `posted.marker`, or `subruns/` state from older runs
|
||||
@@ -242,7 +242,7 @@ The cancel helper should:
|
||||
- write `cancelled.marker`
|
||||
- update `state.json` so the final watcher state is `CANCELLED`
|
||||
- stop the watcher instance
|
||||
- avoid any Mattermost post for that run
|
||||
- avoid any MS Teams post for that run
|
||||
|
||||
## Operational Notes
|
||||
|
||||
@@ -255,7 +255,7 @@ The cancel helper should:
|
||||
- In categorized execution, the watcher must remain alive until the parent request has actually gone inactive past the grace window, even if one grouped sub-run already completed.
|
||||
- The watcher exits after the run reaches a terminal state.
|
||||
- The watcher writes state under `/var/lib/atvm-run-watcher/<build-name>`.
|
||||
- The watcher prevents duplicate Mattermost posts by writing posted markers.
|
||||
- The watcher prevents duplicate MS Teams posts by writing posted markers.
|
||||
- Categorized sub-run state is written under `/var/lib/atvm-run-watcher/<build-name>/subruns/<subrun-key>/`.
|
||||
|
||||
## Failure Handling
|
||||
@@ -263,16 +263,16 @@ The cancel helper should:
|
||||
Expected terminal behavior:
|
||||
|
||||
- `COMPLETED`
|
||||
- post to Mattermost
|
||||
- verify `ok`
|
||||
- post to MS Teams
|
||||
- verify the webhook returned HTTP 2xx, typically `202 Accepted`
|
||||
- exit
|
||||
- `FAILED`
|
||||
- post to Mattermost
|
||||
- verify `ok`
|
||||
- post to MS Teams
|
||||
- verify the webhook returned HTTP 2xx, typically `202 Accepted`
|
||||
- exit
|
||||
- categorized `COMPLETED` / `FAILED`
|
||||
- post once for that grouped sub-run
|
||||
- verify `ok`
|
||||
- verify the webhook returned HTTP 2xx, typically `202 Accepted`
|
||||
- continue until the parent request finishes
|
||||
- `CANCELLED`
|
||||
- write final `CANCELLED` state to `state.json`
|
||||
|
||||
@@ -7,11 +7,11 @@ This folder contains a per-run ATVM watcher service package that is intended to
|
||||
Watch an ATVM automation request until it reaches a terminal state, then:
|
||||
|
||||
- for non-categorized runs:
|
||||
- post one final status to Mattermost if the run state is `COMPLETED` or `FAILED`
|
||||
- post one final status to MS Teams if the run state is `COMPLETED` or `FAILED`
|
||||
- for categorized runs:
|
||||
- detect each sequential categorized sub-run
|
||||
- post one final status per completed categorized sub-run if that grouped run state is `COMPLETED` or `FAILED`
|
||||
- verify each Mattermost post succeeded
|
||||
- verify each MS Teams post succeeded
|
||||
- write durable watcher state
|
||||
- exit cleanly so the service stops
|
||||
|
||||
@@ -65,8 +65,8 @@ Typical workflow:
|
||||
5. For non-categorized runs, when the run reaches a terminal state:
|
||||
- `COMPLETED` or `FAILED`
|
||||
- build the final ATVM status
|
||||
- send the status to Mattermost
|
||||
- verify Mattermost returned `ok`
|
||||
- send the status to MS Teams
|
||||
- verify the MS Teams workflow webhook returned HTTP 2xx, typically `202 Accepted`
|
||||
- mark the run as posted
|
||||
- exit
|
||||
- `CANCELLED`, `TERMINATED`, `HUNG`, or `UNKNOWN`
|
||||
@@ -74,9 +74,9 @@ Typical workflow:
|
||||
- mark the final state
|
||||
- exit
|
||||
6. For categorized runs:
|
||||
- detect each grouped sub-run in sequence from the parent run log
|
||||
- wait for that grouped sub-run to finish
|
||||
- send one Mattermost post for that grouped sub-run if it reached `COMPLETED` or `FAILED`
|
||||
- detect each grouped sub-run in sequence from the parent run log
|
||||
- wait for that grouped sub-run to finish
|
||||
- send one MS Teams post for that grouped sub-run if it reached `COMPLETED` or `FAILED`
|
||||
- keep the watcher alive while the parent categorized runner or related child Cypress process is still active
|
||||
- do not treat one completed grouped sub-run as proof that the whole parent request is finished
|
||||
- continue to the next grouped sub-run
|
||||
@@ -86,8 +86,8 @@ Typical workflow:
|
||||
|
||||
The service expects the local credentials file values to be made available on the controller through the service environment:
|
||||
|
||||
- `MATTERMOST_ATVM_WEBHOOK`
|
||||
- `MATTERMOST_ATVM_CHANNEL`
|
||||
- `MS_TEAMS_ATVM_WEBHOOK`
|
||||
- `MS_TEAMS_ATVM_CHANNEL`
|
||||
|
||||
Optional metadata for better status formatting:
|
||||
|
||||
@@ -168,7 +168,7 @@ The helper also:
|
||||
./cancel-atvm-run-watcher.sh --build-name e2e-redhat9.6-ubuntu24.04-w2k25-fc
|
||||
```
|
||||
|
||||
This writes a cancellation marker, updates `state.json` to `CANCELLED`, and stops the watcher instance. The watcher will not send Mattermost results for that run.
|
||||
This writes a cancellation marker, updates `state.json` to `CANCELLED`, and stops the watcher instance. The watcher will not send MS Teams results for that run.
|
||||
|
||||
Runner cancel example:
|
||||
|
||||
|
||||
@@ -1627,16 +1627,37 @@ def build_status_markdown(
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def post_to_mattermost(text: str) -> str:
|
||||
webhook = os.environ["MATTERMOST_ATVM_WEBHOOK"]
|
||||
payload = {"text": text}
|
||||
channel = os.environ.get("MATTERMOST_ATVM_CHANNEL")
|
||||
def post_to_teams(text: str) -> str:
|
||||
webhook = os.environ["MS_TEAMS_ATVM_WEBHOOK"]
|
||||
payload = {
|
||||
"type": "message",
|
||||
"attachments": [
|
||||
{
|
||||
"contentType": "application/vnd.microsoft.card.adaptive",
|
||||
"contentUrl": None,
|
||||
"content": {
|
||||
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
|
||||
"type": "AdaptiveCard",
|
||||
"version": "1.4",
|
||||
"body": [
|
||||
{
|
||||
"type": "TextBlock",
|
||||
"text": text,
|
||||
"wrap": True,
|
||||
}
|
||||
],
|
||||
},
|
||||
}
|
||||
],
|
||||
}
|
||||
channel = os.environ.get("MS_TEAMS_ATVM_CHANNEL")
|
||||
if channel:
|
||||
payload["channel"] = channel
|
||||
data = json.dumps(payload).encode()
|
||||
request = urllib.request.Request(webhook, data=data, headers={"Content-Type": "application/json"})
|
||||
with urllib.request.urlopen(request) as response:
|
||||
return response.read().decode().strip()
|
||||
body = response.read().decode().strip()
|
||||
return f"HTTP {response.status}" + (f": {body}" if body else "")
|
||||
|
||||
|
||||
def sanitize_key(raw: str) -> str:
|
||||
@@ -2334,13 +2355,11 @@ if __name__ == "__main__":
|
||||
log_text=current_log_text,
|
||||
)
|
||||
print(status_text)
|
||||
response = post_to_mattermost(status_text)
|
||||
if response != "ok":
|
||||
raise SystemExit(f"Mattermost webhook did not return ok for {subrun['display_name']}: {response!r}")
|
||||
subrun_posted_marker.write_text("ok\n", encoding="utf-8")
|
||||
subrun_state["mattermost_posted"] = True
|
||||
subrun_state["mattermost_response"] = response
|
||||
print(f"[watcher] Mattermost post confirmed for {subrun['display_name']}.")
|
||||
response = post_to_teams(status_text)
|
||||
subrun_posted_marker.write_text(f"{response}\n", encoding="utf-8")
|
||||
subrun_state["teams_posted"] = True
|
||||
subrun_state["teams_response"] = response
|
||||
print(f"[watcher] Teams post confirmed for {subrun['display_name']}: {response}.")
|
||||
write_state(subrun_state_file, subrun_state)
|
||||
|
||||
if run_state == "RUNNING":
|
||||
@@ -2363,14 +2382,12 @@ if __name__ == "__main__":
|
||||
print(status_text)
|
||||
|
||||
if not metadata.get("categorized") and run_state in {"COMPLETED", "FAILED"} and not posted_marker.exists():
|
||||
response = post_to_mattermost(status_text)
|
||||
if response != "ok":
|
||||
raise SystemExit(f"Mattermost webhook did not return ok: {response!r}")
|
||||
posted_marker.write_text("ok\n", encoding="utf-8")
|
||||
state["mattermost_posted"] = True
|
||||
state["mattermost_response"] = response
|
||||
response = post_to_teams(status_text)
|
||||
posted_marker.write_text(f"{response}\n", encoding="utf-8")
|
||||
state["teams_posted"] = True
|
||||
state["teams_response"] = response
|
||||
write_state(state_file, state)
|
||||
print(f"[watcher] Mattermost post confirmed for {build_name}.")
|
||||
print(f"[watcher] Teams post confirmed for {build_name}: {response}.")
|
||||
|
||||
state["closed_at"] = now_utc().isoformat()
|
||||
write_state(state_file, state)
|
||||
|
||||
Reference in New Issue
Block a user