Switch ATVM watcher status posts to MS Teams
This commit is contained in:
Anthony Wen
@@ -50,7 +50,7 @@ Run ATVM CMC automation tests on the designated automation VM without unintended
|
||||
- If the operator changes any part of the request before execution, rebuild commands and execute the revised command set.
|
||||
- Default to watcher-backed execution for every run unless the operator explicitly asks to run without watcher.
|
||||
- When `--categorize` is used with watcher enabled, treat the watcher as a sequential grouped-run watcher:
|
||||
- it must post one final Mattermost status per completed categorized group/sub-run
|
||||
- it must post one final MS Teams status per completed categorized group/sub-run
|
||||
- it must stay active between grouped sub-runs while the parent categorized request is still running
|
||||
- it must not stop after the first grouped run simply because one grouped run completed
|
||||
- if the child build id label does not match the actual host/spec being executed, report the grouped run using the inferred host-based group instead of the raw child build id label
|
||||
@@ -273,19 +273,19 @@ When asked for one VM or a VM set:
|
||||
- If monitoring was not requested, run commands and report execution success/failure and any errors.
|
||||
- If monitoring was requested, do not terminate processes automatically; only terminate if the operator explicitly instructs termination.
|
||||
|
||||
## Mattermost Status Posting
|
||||
## MS Teams Status Posting
|
||||
- Treat a normal ATVM status request as local-only output by default.
|
||||
- When the operator asks to send ATVM automation run status to Mattermost, use the local defaults from `/home/aw/code/cds/.env.credentials.local`.
|
||||
- Default Mattermost variables:
|
||||
- `MATTERMOST_ATVM_WEBHOOK`
|
||||
- `MATTERMOST_ATVM_CHANNEL`
|
||||
- When the operator asks to send ATVM automation run status to MS Teams, use the local defaults from `/home/aw/code/cds/.env.credentials.local`.
|
||||
- Default MS Teams variables:
|
||||
- `MS_TEAMS_ATVM_WEBHOOK`
|
||||
- `MS_TEAMS_ATVM_CHANNEL`
|
||||
- Treat these as the default destination for ATVM automation run-status posts unless the operator explicitly overrides them.
|
||||
- Send the final ATVM run status only after the run has fully completed, regardless of whether the run passed or failed.
|
||||
- Do not send interim or in-progress ATVM run status updates to Mattermost unless the operator explicitly asks for that.
|
||||
- Use the same ATVM status layout that would be shown to the operator locally when posting to Mattermost.
|
||||
- Do not send interim or in-progress ATVM run status updates to MS Teams unless the operator explicitly asks for that.
|
||||
- Use the same ATVM status layout that would be shown to the operator locally when posting to MS Teams.
|
||||
- Default status template: `/home/aw/code/cds/atvm/docs/automation/status-template.md`
|
||||
- Do not post to Mattermost unless the operator explicitly asks for the run status to be sent there.
|
||||
- For categorized execution with watcher enabled, send one Mattermost status per completed categorized sub-run/group after that grouped run fully finishes.
|
||||
- Do not post to MS Teams unless the operator explicitly asks for the run status to be sent there.
|
||||
- For categorized execution with watcher enabled, send one MS Teams status per completed categorized sub-run/group after that grouped run fully finishes.
|
||||
|
||||
## Status Reporting Format
|
||||
When the operator asks for the status of an ATVM automation run, report in this order:
|
||||
@@ -310,7 +310,7 @@ When the operator asks for the status of an ATVM automation run, report in this
|
||||
|
||||
Status-report expectations:
|
||||
- Use the same display layout for every ATVM automation status response regardless of test type (`e2e`, `systemOS`, `reboot`, `migrateops`, and others).
|
||||
- Use `/home/aw/code/cds/atvm/docs/automation/status-template.md` as the default template for both local status output and Mattermost status posts.
|
||||
- Use `/home/aw/code/cds/atvm/docs/automation/status-template.md` as the default template for both local status output and MS Teams status posts.
|
||||
- The default ATVM status template uses flat bullet-list sections for `COVERAGE:`, `TEST FLOW:`, `FAILURE NOTES:`, and `NOTES:`, and Markdown tables for `SUMMARY:`, `HOSTS:`, and `TIMING:`.
|
||||
- Order the status sections as `SUMMARY:`, `HOSTS:`, `TIMING:`, `COVERAGE:`, `TEST FLOW:`, `FAILURE NOTES:`, then `NOTES:`.
|
||||
- Keep `NOTES:` focused on non-failure operator-facing value such as the Currents run URL, real anomalies unrelated to the direct failure text, or material fallback behavior.
|
||||
@@ -322,7 +322,7 @@ Status-report expectations:
|
||||
- Do not show total test/failure counts in the `Detail` column. Keep those counts in watcher state and summary logic only.
|
||||
- For passed hosts, render `Detail` as `completed`.
|
||||
- For any failed host, keep the `Detail` column compact by showing the failing step plus a short error summary, not the full raw stack trace.
|
||||
- If richer failure text is available, put the longer trimmed excerpt in `FAILURE NOTES:` so the result stays readable in Mattermost and local status output.
|
||||
- If richer failure text is available, put the longer trimmed excerpt in `FAILURE NOTES:` so the result stays readable in MS Teams and local status output.
|
||||
- In `COVERAGE:`, describe the important `cmc-templates.py` command inputs such as template, categorize mode, datastore/config family, config filename, migration style, any real plugin/integration path, and other operator-relevant run options, but do not list target hosts there or include verbose prose scope descriptions.
|
||||
- Only include coverage fields that the template command actually used. Do not show empty or irrelevant fields such as an integration/plugin path for templates that did not use one.
|
||||
- If `categorize mode: enabled` is already shown in `COVERAGE:`, do not also repeat `--categorize` under `run options`.
|
||||
@@ -341,7 +341,7 @@ Status-report expectations:
|
||||
- If the hostname is not present in `vm-inventory.md`, report the kernel value as `unknown`.
|
||||
- Treat references to the "ATVM automation run" or "automation run" as referring to this ATVM folder workflow and the automation VM at `192.168.3.190`, not to Cirrus project operations such as the `atvm - cypress` project.
|
||||
- Treat a status request as a request for live status by default.
|
||||
- Unless the operator explicitly asks to send the status to Mattermost, print the status only in the local terminal response.
|
||||
- Unless the operator explicitly asks to send the status to MS Teams, print the status only in the local terminal response.
|
||||
- Use the live automation VM state when available.
|
||||
- If no automation is currently running, fall back to the most recent historical run artifacts and logs.
|
||||
- Prefer local automation evidence in this order: active runner processes, live automation-VM files, shell history for the last launch command, then historical reporter artifacts.
|
||||
@@ -375,4 +375,4 @@ Status-report expectations:
|
||||
- Use `Notes` for extra context beyond the machine-specific same-line failure description.
|
||||
- Base the completion estimate on the full remaining machine count and recent per-machine runtime visible in the run log.
|
||||
- Make the estimate explicitly refer to completion of the entire remaining run, not only the current machine/spec.
|
||||
- When the operator also asks to send the status to Mattermost, send this same final status output to the configured Mattermost destination only after the run has fully completed.
|
||||
- When the operator also asks to send the status to MS Teams, send this same final status output to the configured MS Teams destination only after the run has fully completed.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Use this as the default ATVM automation run-status template for:
|
||||
- local status responses in the terminal
|
||||
- Mattermost status posts after a completed run
|
||||
- MS Teams status posts after a completed run
|
||||
|
||||
## Layout
|
||||
|
||||
@@ -173,4 +173,4 @@ Use this as the default ATVM automation run-status template for:
|
||||
- See `/home/aw/code/cds/atvm/docs/automation/examples.md` for `cmc-e2e` examples.
|
||||
- Resolve kernel values by cross-referencing hostnames against `/home/aw/code/cds/atvm/inventory/vm-inventory.md`.
|
||||
- If no kernel value can be verified from `vm-inventory.md`, use `unknown`.
|
||||
- Use the same template for Mattermost and local operator-visible status output.
|
||||
- Use the same template for MS Teams and local operator-visible status output.
|
||||
|
||||
+25
-25
@@ -1,7 +1,7 @@
|
||||
# ATVM Mattermost Watcher Design
|
||||
# ATVM MS Teams Watcher Design
|
||||
|
||||
## Purpose
|
||||
Design a controller-local watcher on the ATVM Cypress machine (`192.168.3.190`) that monitors an ATVM automation run and posts final run status to Mattermost only after the watched scope has fully completed.
|
||||
Design a controller-local watcher on the ATVM Cypress machine (`192.168.3.190`) that monitors an ATVM automation run and posts final run status to MS Teams only after the watched scope has fully completed.
|
||||
|
||||
This watcher must continue working even if the local operator machine is offline.
|
||||
|
||||
@@ -18,16 +18,16 @@ Preferred deployment target:
|
||||
- controller host: `192.168.3.190`
|
||||
- ATVM automation root: `/root/cdc-e2e-cyp-12.17.4`
|
||||
|
||||
## Mattermost Destination
|
||||
## MS Teams Destination
|
||||
Use the local credential file in this workspace as the source of defaults:
|
||||
- `/home/aw/code/cds/.env.credentials.local`
|
||||
|
||||
Expected variables:
|
||||
- `MATTERMOST_ATVM_WEBHOOK`
|
||||
- `MATTERMOST_ATVM_CHANNEL`
|
||||
- `MS_TEAMS_ATVM_WEBHOOK`
|
||||
- `MS_TEAMS_ATVM_CHANNEL`
|
||||
|
||||
## Run Completion Rule
|
||||
The watcher must send Mattermost results only after the watched scope has fully completed.
|
||||
The watcher must send MS Teams results only after the watched scope has fully completed.
|
||||
|
||||
A non-categorized run is considered fully completed only when:
|
||||
- there are no active runner processes for the run
|
||||
@@ -40,7 +40,7 @@ A categorized run must be treated differently:
|
||||
- each categorized group is its own run/job
|
||||
- the watcher must detect each grouped sub-run in order
|
||||
- the watcher must wait for that grouped sub-run to complete
|
||||
- then send that grouped sub-run's final Mattermost status
|
||||
- then send that grouped sub-run's final MS Teams status
|
||||
- then continue watching for the next grouped sub-run
|
||||
- the watcher must remain alive while the parent categorized request or related child Cypress process is still active
|
||||
- one completed grouped sub-run must not be treated as proof that the parent categorized request is finished
|
||||
@@ -82,14 +82,14 @@ Definitions:
|
||||
- `RUNNING`
|
||||
- the run is still active and not yet complete
|
||||
|
||||
## Mattermost Posting Rule
|
||||
Post to Mattermost only when the watched scope has fully completed.
|
||||
## MS Teams Posting Rule
|
||||
Post to MS Teams only when the watched scope has fully completed.
|
||||
|
||||
Send Mattermost status for:
|
||||
Send MS Teams status for:
|
||||
- `COMPLETED`
|
||||
- `FAILED`
|
||||
|
||||
Do not send Mattermost status for:
|
||||
Do not send MS Teams status for:
|
||||
- `CANCELLED`
|
||||
- `TERMINATED`
|
||||
- `HUNG`
|
||||
@@ -100,14 +100,14 @@ Important clarification:
|
||||
- a completed run with failed hosts should still be posted
|
||||
- a cancelled, terminated, hung, or unknown run should not be posted
|
||||
- for categorized execution, this rule applies per categorized sub-run
|
||||
- one categorized group completion should produce one Mattermost post
|
||||
- one categorized group completion should produce one MS Teams post
|
||||
- do not send one parent-level aggregate post in place of the per-group posts
|
||||
|
||||
## Required Cancellation / Termination Handling
|
||||
If a run is cancelled or terminated, the watcher must:
|
||||
- detect that the run was cancelled or manually killed
|
||||
- stop waiting for normal completion
|
||||
- mark the run as closed without posting final Mattermost status
|
||||
- mark the run as closed without posting final MS Teams status
|
||||
- prevent any later success/failure post for that same run
|
||||
|
||||
## State Tracking Requirements
|
||||
@@ -118,7 +118,7 @@ For each run, keep durable state such as:
|
||||
- controller-side watcher state
|
||||
- completion marker
|
||||
- cancellation / termination marker
|
||||
- Mattermost posted marker
|
||||
- MS Teams posted marker
|
||||
- last observed machine summary
|
||||
- timestamps for first seen, last seen, closed
|
||||
|
||||
@@ -128,7 +128,7 @@ For categorized runs, keep durable state for:
|
||||
- whether each categorized sub-run has already been posted
|
||||
|
||||
## Duplicate-Post Prevention
|
||||
The watcher must prevent duplicate Mattermost posts.
|
||||
The watcher must prevent duplicate MS Teams posts.
|
||||
|
||||
Required behavior:
|
||||
- for non-categorized execution, only one final post per run
|
||||
@@ -159,12 +159,12 @@ Normal completion workflow:
|
||||
4. For non-categorized execution:
|
||||
- wait for the run to fully complete
|
||||
- build one final status summary
|
||||
- post one final Mattermost status
|
||||
- post one final MS Teams status
|
||||
5. For categorized execution:
|
||||
- detect each grouped sub-run in order
|
||||
- wait for that grouped sub-run to fully complete
|
||||
- build that grouped sub-run's final status summary
|
||||
- post that grouped sub-run's final Mattermost status
|
||||
- post that grouped sub-run's final MS Teams status
|
||||
- continue to the next grouped sub-run
|
||||
6. Watcher marks the completed watched scope as posted and closed.
|
||||
|
||||
@@ -172,24 +172,24 @@ Cancellation / termination workflow:
|
||||
1. Operator stops the ATVM run.
|
||||
2. Watcher detects cancellation / termination, or an explicit cancellation marker is written.
|
||||
3. Watcher marks the run `CANCELLED` or `TERMINATED`.
|
||||
4. Watcher exits cleanly without posting to Mattermost.
|
||||
4. Watcher exits cleanly without posting to MS Teams.
|
||||
5. Watcher prevents later duplicate or misleading final-post behavior.
|
||||
|
||||
## Failure Semantics
|
||||
Host-level failures do not suppress Mattermost posting.
|
||||
Host-level failures do not suppress MS Teams posting.
|
||||
|
||||
If:
|
||||
- the run has fully completed
|
||||
- and one or more hosts failed
|
||||
|
||||
Then:
|
||||
- final Mattermost status should still be sent
|
||||
- final MS Teams status should still be sent
|
||||
- final run-level state should be treated as completed-with-failures
|
||||
|
||||
## Hang / Unknown Semantics
|
||||
If the run cannot be safely classified as completed, failed, cancelled, or terminated:
|
||||
- classify it as `HUNG` or `UNKNOWN`
|
||||
- do not post to Mattermost
|
||||
- do not post to MS Teams
|
||||
- require operator review
|
||||
|
||||
## Logging Requirements
|
||||
@@ -197,7 +197,7 @@ The watcher should log:
|
||||
- the run id / build name being monitored
|
||||
- each state transition
|
||||
- posting decisions
|
||||
- reasons for suppressing a Mattermost post
|
||||
- reasons for suppressing a MS Teams post
|
||||
- duplicate-post prevention decisions
|
||||
- final closed state
|
||||
|
||||
@@ -207,9 +207,9 @@ This watcher design must satisfy all of the following:
|
||||
- survive local operator machine downtime
|
||||
- use `systemd`
|
||||
- distinguish run states clearly
|
||||
- send Mattermost only after full completion of the watched scope
|
||||
- send MS Teams only after full completion of the watched scope
|
||||
- send completion results whether hosts passed or failed
|
||||
- never send Mattermost for cancelled, terminated, hung, or unknown runs
|
||||
- never send MS Teams for cancelled, terminated, hung, or unknown runs
|
||||
- prevent duplicate or misleading posts
|
||||
- treat `--categorize` as sequential ATVM sub-runs, not as one parent run with internal phases
|
||||
- send one Mattermost post per completed categorized sub-run
|
||||
- send one MS Teams post per completed categorized sub-run
|
||||
@@ -22,7 +22,7 @@ The intended cleanup is:
|
||||
Examples of what tracked docs should say instead of storing raw values:
|
||||
- `Use ATVM_CONTROLLER_PASSWORD from /home/aw/code/cds/.env.credentials.local`
|
||||
- `Use VCENTER_USER and VCENTER_PASSWORD from /home/aw/code/cds/.env.credentials.local`
|
||||
- `Use MATTERMOST_ATVM_WEBHOOK from /home/aw/code/cds/.env.credentials.local`
|
||||
- `Use MS_TEAMS_ATVM_WEBHOOK from /home/aw/code/cds/.env.credentials.local`
|
||||
|
||||
Recommended scope of cleanup:
|
||||
- `atvm/inventory/accounts-and-credentials.md`
|
||||
@@ -83,7 +83,7 @@ Do these in this order:
|
||||
This is the most important step.
|
||||
|
||||
Examples:
|
||||
- regenerate Mattermost webhook URLs
|
||||
- regenerate MS Teams webhook URLs
|
||||
- replace API tokens
|
||||
- rotate passwords
|
||||
- regenerate TOTP/shared secrets if applicable
|
||||
|
||||
Reference in New Issue
Block a user