Switch ATVM watcher status posts to MS Teams
This commit is contained in:
+3
-3
@@ -51,7 +51,7 @@ This file defines how to operate and maintain the ATVM workspace in `/home/aw/co
|
|||||||
- Controller credential source: `/home/aw/code/cds/.env.credentials.local` via `ATVM_CONTROLLER_USER` and `ATVM_CONTROLLER_PASSWORD`
|
- Controller credential source: `/home/aw/code/cds/.env.credentials.local` via `ATVM_CONTROLLER_USER` and `ATVM_CONTROLLER_PASSWORD`
|
||||||
- Detailed test artifact root on controller: `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter`
|
- Detailed test artifact root on controller: `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter`
|
||||||
- For vCenter inspection, prefer `govc` and raw vCenter REST calls before inventing or depending on higher-level wrappers.
|
- For vCenter inspection, prefer `govc` and raw vCenter REST calls before inventing or depending on higher-level wrappers.
|
||||||
- Default Mattermost status destination config: `/home/aw/code/cds/.env.credentials.local`
|
- Default MS Teams status destination config: `/home/aw/code/cds/.env.credentials.local`
|
||||||
- Default plugin-bearing template plugin: `--use_specified_plugin iscsi`
|
- Default plugin-bearing template plugin: `--use_specified_plugin iscsi`
|
||||||
- Always include `--ignore_force_shutdown` unless explicitly told not to.
|
- Always include `--ignore_force_shutdown` unless explicitly told not to.
|
||||||
- Always include `--test_partition` unless explicitly told not to.
|
- Always include `--test_partition` unless explicitly told not to.
|
||||||
@@ -101,8 +101,8 @@ This file defines how to operate and maintain the ATVM workspace in `/home/aw/co
|
|||||||
- Never confirm a completed ATVM run from `state.json` alone.
|
- Never confirm a completed ATVM run from `state.json` alone.
|
||||||
- For categorized runs, never report a grouped sub-run as `PASS` from watcher `host_results`, grouped XML, or a lone `check-xml-files.ts` result by itself.
|
- For categorized runs, never report a grouped sub-run as `PASS` from watcher `host_results`, grouped XML, or a lone `check-xml-files.ts` result by itself.
|
||||||
- Before reporting a categorized grouped sub-run as `PASS`, confirm that the matching child batch also passed in the live launch log or the final `Cloud Run Finished` summary for that child run.
|
- Before reporting a categorized grouped sub-run as `PASS`, confirm that the matching child batch also passed in the live launch log or the final `Cloud Run Finished` summary for that child run.
|
||||||
- If the operator asks for ATVM run status without mentioning Mattermost, respond locally only and do not post externally.
|
- If the operator asks for ATVM run status without mentioning MS Teams, respond locally only and do not post externally.
|
||||||
- If the operator asks to send ATVM run status to Mattermost, use `MATTERMOST_ATVM_WEBHOOK` and `MATTERMOST_ATVM_CHANNEL` from `/home/aw/code/cds/.env.credentials.local` by default and send the final status only after the run has fully completed, whether the run passed or failed.
|
- If the operator asks to send ATVM run status to MS Teams, use `MS_TEAMS_ATVM_WEBHOOK` and `MS_TEAMS_ATVM_CHANNEL` from `/home/aw/code/cds/.env.credentials.local` by default and send the final status only after the run has fully completed, whether the run passed or failed.
|
||||||
- For vCenter VM snapshot requests, default the snapshot name to `VM Snapshot [mm/dd/yyyy:hh:mm:ss AM/PM]` in the local `America/New_York` timezone unless the operator explicitly requests a different name.
|
- For vCenter VM snapshot requests, default the snapshot name to `VM Snapshot [mm/dd/yyyy:hh:mm:ss AM/PM]` in the local `America/New_York` timezone unless the operator explicitly requests a different name.
|
||||||
- For VM power, shutdown, startup, snapshot, or maintenance requests, operate only on the exact VM names provided by the operator.
|
- For VM power, shutdown, startup, snapshot, or maintenance requests, operate only on the exact VM names provided by the operator.
|
||||||
- If any requested VM name is missing or not found, stop immediately and report the missing names; do not infer, substitute, or search for replacement VMs unless the operator explicitly requests a discovery/mapping step.
|
- If any requested VM name is missing or not found, stop immediately and report the missing names; do not infer, substitute, or search for replacement VMs unless the operator explicitly requests a discovery/mapping step.
|
||||||
|
|||||||
@@ -50,7 +50,7 @@ Run ATVM CMC automation tests on the designated automation VM without unintended
|
|||||||
- If the operator changes any part of the request before execution, rebuild commands and execute the revised command set.
|
- If the operator changes any part of the request before execution, rebuild commands and execute the revised command set.
|
||||||
- Default to watcher-backed execution for every run unless the operator explicitly asks to run without watcher.
|
- Default to watcher-backed execution for every run unless the operator explicitly asks to run without watcher.
|
||||||
- When `--categorize` is used with watcher enabled, treat the watcher as a sequential grouped-run watcher:
|
- When `--categorize` is used with watcher enabled, treat the watcher as a sequential grouped-run watcher:
|
||||||
- it must post one final Mattermost status per completed categorized group/sub-run
|
- it must post one final MS Teams status per completed categorized group/sub-run
|
||||||
- it must stay active between grouped sub-runs while the parent categorized request is still running
|
- it must stay active between grouped sub-runs while the parent categorized request is still running
|
||||||
- it must not stop after the first grouped run simply because one grouped run completed
|
- it must not stop after the first grouped run simply because one grouped run completed
|
||||||
- if the child build id label does not match the actual host/spec being executed, report the grouped run using the inferred host-based group instead of the raw child build id label
|
- if the child build id label does not match the actual host/spec being executed, report the grouped run using the inferred host-based group instead of the raw child build id label
|
||||||
@@ -273,19 +273,19 @@ When asked for one VM or a VM set:
|
|||||||
- If monitoring was not requested, run commands and report execution success/failure and any errors.
|
- If monitoring was not requested, run commands and report execution success/failure and any errors.
|
||||||
- If monitoring was requested, do not terminate processes automatically; only terminate if the operator explicitly instructs termination.
|
- If monitoring was requested, do not terminate processes automatically; only terminate if the operator explicitly instructs termination.
|
||||||
|
|
||||||
## Mattermost Status Posting
|
## MS Teams Status Posting
|
||||||
- Treat a normal ATVM status request as local-only output by default.
|
- Treat a normal ATVM status request as local-only output by default.
|
||||||
- When the operator asks to send ATVM automation run status to Mattermost, use the local defaults from `/home/aw/code/cds/.env.credentials.local`.
|
- When the operator asks to send ATVM automation run status to MS Teams, use the local defaults from `/home/aw/code/cds/.env.credentials.local`.
|
||||||
- Default Mattermost variables:
|
- Default MS Teams variables:
|
||||||
- `MATTERMOST_ATVM_WEBHOOK`
|
- `MS_TEAMS_ATVM_WEBHOOK`
|
||||||
- `MATTERMOST_ATVM_CHANNEL`
|
- `MS_TEAMS_ATVM_CHANNEL`
|
||||||
- Treat these as the default destination for ATVM automation run-status posts unless the operator explicitly overrides them.
|
- Treat these as the default destination for ATVM automation run-status posts unless the operator explicitly overrides them.
|
||||||
- Send the final ATVM run status only after the run has fully completed, regardless of whether the run passed or failed.
|
- Send the final ATVM run status only after the run has fully completed, regardless of whether the run passed or failed.
|
||||||
- Do not send interim or in-progress ATVM run status updates to Mattermost unless the operator explicitly asks for that.
|
- Do not send interim or in-progress ATVM run status updates to MS Teams unless the operator explicitly asks for that.
|
||||||
- Use the same ATVM status layout that would be shown to the operator locally when posting to Mattermost.
|
- Use the same ATVM status layout that would be shown to the operator locally when posting to MS Teams.
|
||||||
- Default status template: `/home/aw/code/cds/atvm/docs/automation/status-template.md`
|
- Default status template: `/home/aw/code/cds/atvm/docs/automation/status-template.md`
|
||||||
- Do not post to Mattermost unless the operator explicitly asks for the run status to be sent there.
|
- Do not post to MS Teams unless the operator explicitly asks for the run status to be sent there.
|
||||||
- For categorized execution with watcher enabled, send one Mattermost status per completed categorized sub-run/group after that grouped run fully finishes.
|
- For categorized execution with watcher enabled, send one MS Teams status per completed categorized sub-run/group after that grouped run fully finishes.
|
||||||
|
|
||||||
## Status Reporting Format
|
## Status Reporting Format
|
||||||
When the operator asks for the status of an ATVM automation run, report in this order:
|
When the operator asks for the status of an ATVM automation run, report in this order:
|
||||||
@@ -310,7 +310,7 @@ When the operator asks for the status of an ATVM automation run, report in this
|
|||||||
|
|
||||||
Status-report expectations:
|
Status-report expectations:
|
||||||
- Use the same display layout for every ATVM automation status response regardless of test type (`e2e`, `systemOS`, `reboot`, `migrateops`, and others).
|
- Use the same display layout for every ATVM automation status response regardless of test type (`e2e`, `systemOS`, `reboot`, `migrateops`, and others).
|
||||||
- Use `/home/aw/code/cds/atvm/docs/automation/status-template.md` as the default template for both local status output and Mattermost status posts.
|
- Use `/home/aw/code/cds/atvm/docs/automation/status-template.md` as the default template for both local status output and MS Teams status posts.
|
||||||
- The default ATVM status template uses flat bullet-list sections for `COVERAGE:`, `TEST FLOW:`, `FAILURE NOTES:`, and `NOTES:`, and Markdown tables for `SUMMARY:`, `HOSTS:`, and `TIMING:`.
|
- The default ATVM status template uses flat bullet-list sections for `COVERAGE:`, `TEST FLOW:`, `FAILURE NOTES:`, and `NOTES:`, and Markdown tables for `SUMMARY:`, `HOSTS:`, and `TIMING:`.
|
||||||
- Order the status sections as `SUMMARY:`, `HOSTS:`, `TIMING:`, `COVERAGE:`, `TEST FLOW:`, `FAILURE NOTES:`, then `NOTES:`.
|
- Order the status sections as `SUMMARY:`, `HOSTS:`, `TIMING:`, `COVERAGE:`, `TEST FLOW:`, `FAILURE NOTES:`, then `NOTES:`.
|
||||||
- Keep `NOTES:` focused on non-failure operator-facing value such as the Currents run URL, real anomalies unrelated to the direct failure text, or material fallback behavior.
|
- Keep `NOTES:` focused on non-failure operator-facing value such as the Currents run URL, real anomalies unrelated to the direct failure text, or material fallback behavior.
|
||||||
@@ -322,7 +322,7 @@ Status-report expectations:
|
|||||||
- Do not show total test/failure counts in the `Detail` column. Keep those counts in watcher state and summary logic only.
|
- Do not show total test/failure counts in the `Detail` column. Keep those counts in watcher state and summary logic only.
|
||||||
- For passed hosts, render `Detail` as `completed`.
|
- For passed hosts, render `Detail` as `completed`.
|
||||||
- For any failed host, keep the `Detail` column compact by showing the failing step plus a short error summary, not the full raw stack trace.
|
- For any failed host, keep the `Detail` column compact by showing the failing step plus a short error summary, not the full raw stack trace.
|
||||||
- If richer failure text is available, put the longer trimmed excerpt in `FAILURE NOTES:` so the result stays readable in Mattermost and local status output.
|
- If richer failure text is available, put the longer trimmed excerpt in `FAILURE NOTES:` so the result stays readable in MS Teams and local status output.
|
||||||
- In `COVERAGE:`, describe the important `cmc-templates.py` command inputs such as template, categorize mode, datastore/config family, config filename, migration style, any real plugin/integration path, and other operator-relevant run options, but do not list target hosts there or include verbose prose scope descriptions.
|
- In `COVERAGE:`, describe the important `cmc-templates.py` command inputs such as template, categorize mode, datastore/config family, config filename, migration style, any real plugin/integration path, and other operator-relevant run options, but do not list target hosts there or include verbose prose scope descriptions.
|
||||||
- Only include coverage fields that the template command actually used. Do not show empty or irrelevant fields such as an integration/plugin path for templates that did not use one.
|
- Only include coverage fields that the template command actually used. Do not show empty or irrelevant fields such as an integration/plugin path for templates that did not use one.
|
||||||
- If `categorize mode: enabled` is already shown in `COVERAGE:`, do not also repeat `--categorize` under `run options`.
|
- If `categorize mode: enabled` is already shown in `COVERAGE:`, do not also repeat `--categorize` under `run options`.
|
||||||
@@ -341,7 +341,7 @@ Status-report expectations:
|
|||||||
- If the hostname is not present in `vm-inventory.md`, report the kernel value as `unknown`.
|
- If the hostname is not present in `vm-inventory.md`, report the kernel value as `unknown`.
|
||||||
- Treat references to the "ATVM automation run" or "automation run" as referring to this ATVM folder workflow and the automation VM at `192.168.3.190`, not to Cirrus project operations such as the `atvm - cypress` project.
|
- Treat references to the "ATVM automation run" or "automation run" as referring to this ATVM folder workflow and the automation VM at `192.168.3.190`, not to Cirrus project operations such as the `atvm - cypress` project.
|
||||||
- Treat a status request as a request for live status by default.
|
- Treat a status request as a request for live status by default.
|
||||||
- Unless the operator explicitly asks to send the status to Mattermost, print the status only in the local terminal response.
|
- Unless the operator explicitly asks to send the status to MS Teams, print the status only in the local terminal response.
|
||||||
- Use the live automation VM state when available.
|
- Use the live automation VM state when available.
|
||||||
- If no automation is currently running, fall back to the most recent historical run artifacts and logs.
|
- If no automation is currently running, fall back to the most recent historical run artifacts and logs.
|
||||||
- Prefer local automation evidence in this order: active runner processes, live automation-VM files, shell history for the last launch command, then historical reporter artifacts.
|
- Prefer local automation evidence in this order: active runner processes, live automation-VM files, shell history for the last launch command, then historical reporter artifacts.
|
||||||
@@ -375,4 +375,4 @@ Status-report expectations:
|
|||||||
- Use `Notes` for extra context beyond the machine-specific same-line failure description.
|
- Use `Notes` for extra context beyond the machine-specific same-line failure description.
|
||||||
- Base the completion estimate on the full remaining machine count and recent per-machine runtime visible in the run log.
|
- Base the completion estimate on the full remaining machine count and recent per-machine runtime visible in the run log.
|
||||||
- Make the estimate explicitly refer to completion of the entire remaining run, not only the current machine/spec.
|
- Make the estimate explicitly refer to completion of the entire remaining run, not only the current machine/spec.
|
||||||
- When the operator also asks to send the status to Mattermost, send this same final status output to the configured Mattermost destination only after the run has fully completed.
|
- When the operator also asks to send the status to MS Teams, send this same final status output to the configured MS Teams destination only after the run has fully completed.
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
Use this as the default ATVM automation run-status template for:
|
Use this as the default ATVM automation run-status template for:
|
||||||
- local status responses in the terminal
|
- local status responses in the terminal
|
||||||
- Mattermost status posts after a completed run
|
- MS Teams status posts after a completed run
|
||||||
|
|
||||||
## Layout
|
## Layout
|
||||||
|
|
||||||
@@ -173,4 +173,4 @@ Use this as the default ATVM automation run-status template for:
|
|||||||
- See `/home/aw/code/cds/atvm/docs/automation/examples.md` for `cmc-e2e` examples.
|
- See `/home/aw/code/cds/atvm/docs/automation/examples.md` for `cmc-e2e` examples.
|
||||||
- Resolve kernel values by cross-referencing hostnames against `/home/aw/code/cds/atvm/inventory/vm-inventory.md`.
|
- Resolve kernel values by cross-referencing hostnames against `/home/aw/code/cds/atvm/inventory/vm-inventory.md`.
|
||||||
- If no kernel value can be verified from `vm-inventory.md`, use `unknown`.
|
- If no kernel value can be verified from `vm-inventory.md`, use `unknown`.
|
||||||
- Use the same template for Mattermost and local operator-visible status output.
|
- Use the same template for MS Teams and local operator-visible status output.
|
||||||
|
|||||||
+25
-25
@@ -1,7 +1,7 @@
|
|||||||
# ATVM Mattermost Watcher Design
|
# ATVM MS Teams Watcher Design
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Design a controller-local watcher on the ATVM Cypress machine (`192.168.3.190`) that monitors an ATVM automation run and posts final run status to Mattermost only after the watched scope has fully completed.
|
Design a controller-local watcher on the ATVM Cypress machine (`192.168.3.190`) that monitors an ATVM automation run and posts final run status to MS Teams only after the watched scope has fully completed.
|
||||||
|
|
||||||
This watcher must continue working even if the local operator machine is offline.
|
This watcher must continue working even if the local operator machine is offline.
|
||||||
|
|
||||||
@@ -18,16 +18,16 @@ Preferred deployment target:
|
|||||||
- controller host: `192.168.3.190`
|
- controller host: `192.168.3.190`
|
||||||
- ATVM automation root: `/root/cdc-e2e-cyp-12.17.4`
|
- ATVM automation root: `/root/cdc-e2e-cyp-12.17.4`
|
||||||
|
|
||||||
## Mattermost Destination
|
## MS Teams Destination
|
||||||
Use the local credential file in this workspace as the source of defaults:
|
Use the local credential file in this workspace as the source of defaults:
|
||||||
- `/home/aw/code/cds/.env.credentials.local`
|
- `/home/aw/code/cds/.env.credentials.local`
|
||||||
|
|
||||||
Expected variables:
|
Expected variables:
|
||||||
- `MATTERMOST_ATVM_WEBHOOK`
|
- `MS_TEAMS_ATVM_WEBHOOK`
|
||||||
- `MATTERMOST_ATVM_CHANNEL`
|
- `MS_TEAMS_ATVM_CHANNEL`
|
||||||
|
|
||||||
## Run Completion Rule
|
## Run Completion Rule
|
||||||
The watcher must send Mattermost results only after the watched scope has fully completed.
|
The watcher must send MS Teams results only after the watched scope has fully completed.
|
||||||
|
|
||||||
A non-categorized run is considered fully completed only when:
|
A non-categorized run is considered fully completed only when:
|
||||||
- there are no active runner processes for the run
|
- there are no active runner processes for the run
|
||||||
@@ -40,7 +40,7 @@ A categorized run must be treated differently:
|
|||||||
- each categorized group is its own run/job
|
- each categorized group is its own run/job
|
||||||
- the watcher must detect each grouped sub-run in order
|
- the watcher must detect each grouped sub-run in order
|
||||||
- the watcher must wait for that grouped sub-run to complete
|
- the watcher must wait for that grouped sub-run to complete
|
||||||
- then send that grouped sub-run's final Mattermost status
|
- then send that grouped sub-run's final MS Teams status
|
||||||
- then continue watching for the next grouped sub-run
|
- then continue watching for the next grouped sub-run
|
||||||
- the watcher must remain alive while the parent categorized request or related child Cypress process is still active
|
- the watcher must remain alive while the parent categorized request or related child Cypress process is still active
|
||||||
- one completed grouped sub-run must not be treated as proof that the parent categorized request is finished
|
- one completed grouped sub-run must not be treated as proof that the parent categorized request is finished
|
||||||
@@ -82,14 +82,14 @@ Definitions:
|
|||||||
- `RUNNING`
|
- `RUNNING`
|
||||||
- the run is still active and not yet complete
|
- the run is still active and not yet complete
|
||||||
|
|
||||||
## Mattermost Posting Rule
|
## MS Teams Posting Rule
|
||||||
Post to Mattermost only when the watched scope has fully completed.
|
Post to MS Teams only when the watched scope has fully completed.
|
||||||
|
|
||||||
Send Mattermost status for:
|
Send MS Teams status for:
|
||||||
- `COMPLETED`
|
- `COMPLETED`
|
||||||
- `FAILED`
|
- `FAILED`
|
||||||
|
|
||||||
Do not send Mattermost status for:
|
Do not send MS Teams status for:
|
||||||
- `CANCELLED`
|
- `CANCELLED`
|
||||||
- `TERMINATED`
|
- `TERMINATED`
|
||||||
- `HUNG`
|
- `HUNG`
|
||||||
@@ -100,14 +100,14 @@ Important clarification:
|
|||||||
- a completed run with failed hosts should still be posted
|
- a completed run with failed hosts should still be posted
|
||||||
- a cancelled, terminated, hung, or unknown run should not be posted
|
- a cancelled, terminated, hung, or unknown run should not be posted
|
||||||
- for categorized execution, this rule applies per categorized sub-run
|
- for categorized execution, this rule applies per categorized sub-run
|
||||||
- one categorized group completion should produce one Mattermost post
|
- one categorized group completion should produce one MS Teams post
|
||||||
- do not send one parent-level aggregate post in place of the per-group posts
|
- do not send one parent-level aggregate post in place of the per-group posts
|
||||||
|
|
||||||
## Required Cancellation / Termination Handling
|
## Required Cancellation / Termination Handling
|
||||||
If a run is cancelled or terminated, the watcher must:
|
If a run is cancelled or terminated, the watcher must:
|
||||||
- detect that the run was cancelled or manually killed
|
- detect that the run was cancelled or manually killed
|
||||||
- stop waiting for normal completion
|
- stop waiting for normal completion
|
||||||
- mark the run as closed without posting final Mattermost status
|
- mark the run as closed without posting final MS Teams status
|
||||||
- prevent any later success/failure post for that same run
|
- prevent any later success/failure post for that same run
|
||||||
|
|
||||||
## State Tracking Requirements
|
## State Tracking Requirements
|
||||||
@@ -118,7 +118,7 @@ For each run, keep durable state such as:
|
|||||||
- controller-side watcher state
|
- controller-side watcher state
|
||||||
- completion marker
|
- completion marker
|
||||||
- cancellation / termination marker
|
- cancellation / termination marker
|
||||||
- Mattermost posted marker
|
- MS Teams posted marker
|
||||||
- last observed machine summary
|
- last observed machine summary
|
||||||
- timestamps for first seen, last seen, closed
|
- timestamps for first seen, last seen, closed
|
||||||
|
|
||||||
@@ -128,7 +128,7 @@ For categorized runs, keep durable state for:
|
|||||||
- whether each categorized sub-run has already been posted
|
- whether each categorized sub-run has already been posted
|
||||||
|
|
||||||
## Duplicate-Post Prevention
|
## Duplicate-Post Prevention
|
||||||
The watcher must prevent duplicate Mattermost posts.
|
The watcher must prevent duplicate MS Teams posts.
|
||||||
|
|
||||||
Required behavior:
|
Required behavior:
|
||||||
- for non-categorized execution, only one final post per run
|
- for non-categorized execution, only one final post per run
|
||||||
@@ -159,12 +159,12 @@ Normal completion workflow:
|
|||||||
4. For non-categorized execution:
|
4. For non-categorized execution:
|
||||||
- wait for the run to fully complete
|
- wait for the run to fully complete
|
||||||
- build one final status summary
|
- build one final status summary
|
||||||
- post one final Mattermost status
|
- post one final MS Teams status
|
||||||
5. For categorized execution:
|
5. For categorized execution:
|
||||||
- detect each grouped sub-run in order
|
- detect each grouped sub-run in order
|
||||||
- wait for that grouped sub-run to fully complete
|
- wait for that grouped sub-run to fully complete
|
||||||
- build that grouped sub-run's final status summary
|
- build that grouped sub-run's final status summary
|
||||||
- post that grouped sub-run's final Mattermost status
|
- post that grouped sub-run's final MS Teams status
|
||||||
- continue to the next grouped sub-run
|
- continue to the next grouped sub-run
|
||||||
6. Watcher marks the completed watched scope as posted and closed.
|
6. Watcher marks the completed watched scope as posted and closed.
|
||||||
|
|
||||||
@@ -172,24 +172,24 @@ Cancellation / termination workflow:
|
|||||||
1. Operator stops the ATVM run.
|
1. Operator stops the ATVM run.
|
||||||
2. Watcher detects cancellation / termination, or an explicit cancellation marker is written.
|
2. Watcher detects cancellation / termination, or an explicit cancellation marker is written.
|
||||||
3. Watcher marks the run `CANCELLED` or `TERMINATED`.
|
3. Watcher marks the run `CANCELLED` or `TERMINATED`.
|
||||||
4. Watcher exits cleanly without posting to Mattermost.
|
4. Watcher exits cleanly without posting to MS Teams.
|
||||||
5. Watcher prevents later duplicate or misleading final-post behavior.
|
5. Watcher prevents later duplicate or misleading final-post behavior.
|
||||||
|
|
||||||
## Failure Semantics
|
## Failure Semantics
|
||||||
Host-level failures do not suppress Mattermost posting.
|
Host-level failures do not suppress MS Teams posting.
|
||||||
|
|
||||||
If:
|
If:
|
||||||
- the run has fully completed
|
- the run has fully completed
|
||||||
- and one or more hosts failed
|
- and one or more hosts failed
|
||||||
|
|
||||||
Then:
|
Then:
|
||||||
- final Mattermost status should still be sent
|
- final MS Teams status should still be sent
|
||||||
- final run-level state should be treated as completed-with-failures
|
- final run-level state should be treated as completed-with-failures
|
||||||
|
|
||||||
## Hang / Unknown Semantics
|
## Hang / Unknown Semantics
|
||||||
If the run cannot be safely classified as completed, failed, cancelled, or terminated:
|
If the run cannot be safely classified as completed, failed, cancelled, or terminated:
|
||||||
- classify it as `HUNG` or `UNKNOWN`
|
- classify it as `HUNG` or `UNKNOWN`
|
||||||
- do not post to Mattermost
|
- do not post to MS Teams
|
||||||
- require operator review
|
- require operator review
|
||||||
|
|
||||||
## Logging Requirements
|
## Logging Requirements
|
||||||
@@ -197,7 +197,7 @@ The watcher should log:
|
|||||||
- the run id / build name being monitored
|
- the run id / build name being monitored
|
||||||
- each state transition
|
- each state transition
|
||||||
- posting decisions
|
- posting decisions
|
||||||
- reasons for suppressing a Mattermost post
|
- reasons for suppressing a MS Teams post
|
||||||
- duplicate-post prevention decisions
|
- duplicate-post prevention decisions
|
||||||
- final closed state
|
- final closed state
|
||||||
|
|
||||||
@@ -207,9 +207,9 @@ This watcher design must satisfy all of the following:
|
|||||||
- survive local operator machine downtime
|
- survive local operator machine downtime
|
||||||
- use `systemd`
|
- use `systemd`
|
||||||
- distinguish run states clearly
|
- distinguish run states clearly
|
||||||
- send Mattermost only after full completion of the watched scope
|
- send MS Teams only after full completion of the watched scope
|
||||||
- send completion results whether hosts passed or failed
|
- send completion results whether hosts passed or failed
|
||||||
- never send Mattermost for cancelled, terminated, hung, or unknown runs
|
- never send MS Teams for cancelled, terminated, hung, or unknown runs
|
||||||
- prevent duplicate or misleading posts
|
- prevent duplicate or misleading posts
|
||||||
- treat `--categorize` as sequential ATVM sub-runs, not as one parent run with internal phases
|
- treat `--categorize` as sequential ATVM sub-runs, not as one parent run with internal phases
|
||||||
- send one Mattermost post per completed categorized sub-run
|
- send one MS Teams post per completed categorized sub-run
|
||||||
@@ -22,7 +22,7 @@ The intended cleanup is:
|
|||||||
Examples of what tracked docs should say instead of storing raw values:
|
Examples of what tracked docs should say instead of storing raw values:
|
||||||
- `Use ATVM_CONTROLLER_PASSWORD from /home/aw/code/cds/.env.credentials.local`
|
- `Use ATVM_CONTROLLER_PASSWORD from /home/aw/code/cds/.env.credentials.local`
|
||||||
- `Use VCENTER_USER and VCENTER_PASSWORD from /home/aw/code/cds/.env.credentials.local`
|
- `Use VCENTER_USER and VCENTER_PASSWORD from /home/aw/code/cds/.env.credentials.local`
|
||||||
- `Use MATTERMOST_ATVM_WEBHOOK from /home/aw/code/cds/.env.credentials.local`
|
- `Use MS_TEAMS_ATVM_WEBHOOK from /home/aw/code/cds/.env.credentials.local`
|
||||||
|
|
||||||
Recommended scope of cleanup:
|
Recommended scope of cleanup:
|
||||||
- `atvm/inventory/accounts-and-credentials.md`
|
- `atvm/inventory/accounts-and-credentials.md`
|
||||||
@@ -83,7 +83,7 @@ Do these in this order:
|
|||||||
This is the most important step.
|
This is the most important step.
|
||||||
|
|
||||||
Examples:
|
Examples:
|
||||||
- regenerate Mattermost webhook URLs
|
- regenerate MS Teams webhook URLs
|
||||||
- replace API tokens
|
- replace API tokens
|
||||||
- rotate passwords
|
- rotate passwords
|
||||||
- regenerate TOTP/shared secrets if applicable
|
- regenerate TOTP/shared secrets if applicable
|
||||||
|
|||||||
@@ -10,9 +10,9 @@ Install the local watcher/runner package so the controller can:
|
|||||||
|
|
||||||
- start one requested ATVM Cypress runner per service instance
|
- start one requested ATVM Cypress runner per service instance
|
||||||
- watch one requested ATVM run per watcher instance
|
- watch one requested ATVM run per watcher instance
|
||||||
- for non-categorized runs, send one final Mattermost status only for `COMPLETED` or `FAILED`
|
- for non-categorized runs, send one final MS Teams status only for `COMPLETED` or `FAILED`
|
||||||
- for categorized runs, send one final Mattermost status per completed categorized sub-run/group
|
- for categorized runs, send one final MS Teams status per completed categorized sub-run/group
|
||||||
- suppress Mattermost posts for `CANCELLED`, `TERMINATED`, `HUNG`, and `UNKNOWN`
|
- suppress MS Teams posts for `CANCELLED`, `TERMINATED`, `HUNG`, and `UNKNOWN`
|
||||||
- stop automatically after the watched run reaches a terminal state
|
- stop automatically after the watched run reaches a terminal state
|
||||||
|
|
||||||
## Controller Target Layout
|
## Controller Target Layout
|
||||||
@@ -64,7 +64,7 @@ The controller must have:
|
|||||||
|
|
||||||
- `python3`
|
- `python3`
|
||||||
- `systemd`
|
- `systemd`
|
||||||
- outbound network access to the Mattermost webhook
|
- outbound network access to the MS Teams webhook
|
||||||
- read access to:
|
- read access to:
|
||||||
- `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter`
|
- `/root/cdc-e2e-cyp-12.17.4/cypress/cmcReporter`
|
||||||
- `/tmp/<build-name>.log`
|
- `/tmp/<build-name>.log`
|
||||||
@@ -73,8 +73,8 @@ The controller must have:
|
|||||||
|
|
||||||
The controller needs a watcher environment file with:
|
The controller needs a watcher environment file with:
|
||||||
|
|
||||||
- `MATTERMOST_ATVM_WEBHOOK`
|
- `MS_TEAMS_ATVM_WEBHOOK`
|
||||||
- `MATTERMOST_ATVM_CHANNEL`
|
- `MS_TEAMS_ATVM_CHANNEL`
|
||||||
|
|
||||||
Recommended file:
|
Recommended file:
|
||||||
|
|
||||||
@@ -110,7 +110,7 @@ Recommended permissions:
|
|||||||
- `cancel-atvm-run-watcher.sh`
|
- `cancel-atvm-run-watcher.sh`
|
||||||
|
|
||||||
4. Create `/etc/atvm-run-watcher.env`.
|
4. Create `/etc/atvm-run-watcher.env`.
|
||||||
- add Mattermost webhook/channel
|
- add MS Teams webhook/channel
|
||||||
- keep permissions restricted
|
- keep permissions restricted
|
||||||
|
|
||||||
5. Install the `systemd` unit file.
|
5. Install the `systemd` unit file.
|
||||||
@@ -134,7 +134,7 @@ Recommended permissions:
|
|||||||
- launch a real run
|
- launch a real run
|
||||||
- start the watcher for that build name
|
- start the watcher for that build name
|
||||||
- if the run uses `--categorize`, also pass `--categorize` to the watcher start helper
|
- if the run uses `--categorize`, also pass `--categorize` to the watcher start helper
|
||||||
- confirm final Mattermost delivery for a completed run
|
- confirm final MS Teams delivery for a completed run
|
||||||
- confirm categorized execution sends one post per completed grouped sub-run
|
- confirm categorized execution sends one post per completed grouped sub-run
|
||||||
- confirm the watcher stays alive between categorized grouped runs while the parent request is still active
|
- confirm the watcher stays alive between categorized grouped runs while the parent request is still active
|
||||||
- confirm reused parent build names do not inherit stale `cancelled.marker`, `posted.marker`, or `subruns/` state from older runs
|
- confirm reused parent build names do not inherit stale `cancelled.marker`, `posted.marker`, or `subruns/` state from older runs
|
||||||
@@ -242,7 +242,7 @@ The cancel helper should:
|
|||||||
- write `cancelled.marker`
|
- write `cancelled.marker`
|
||||||
- update `state.json` so the final watcher state is `CANCELLED`
|
- update `state.json` so the final watcher state is `CANCELLED`
|
||||||
- stop the watcher instance
|
- stop the watcher instance
|
||||||
- avoid any Mattermost post for that run
|
- avoid any MS Teams post for that run
|
||||||
|
|
||||||
## Operational Notes
|
## Operational Notes
|
||||||
|
|
||||||
@@ -255,7 +255,7 @@ The cancel helper should:
|
|||||||
- In categorized execution, the watcher must remain alive until the parent request has actually gone inactive past the grace window, even if one grouped sub-run already completed.
|
- In categorized execution, the watcher must remain alive until the parent request has actually gone inactive past the grace window, even if one grouped sub-run already completed.
|
||||||
- The watcher exits after the run reaches a terminal state.
|
- The watcher exits after the run reaches a terminal state.
|
||||||
- The watcher writes state under `/var/lib/atvm-run-watcher/<build-name>`.
|
- The watcher writes state under `/var/lib/atvm-run-watcher/<build-name>`.
|
||||||
- The watcher prevents duplicate Mattermost posts by writing posted markers.
|
- The watcher prevents duplicate MS Teams posts by writing posted markers.
|
||||||
- Categorized sub-run state is written under `/var/lib/atvm-run-watcher/<build-name>/subruns/<subrun-key>/`.
|
- Categorized sub-run state is written under `/var/lib/atvm-run-watcher/<build-name>/subruns/<subrun-key>/`.
|
||||||
|
|
||||||
## Failure Handling
|
## Failure Handling
|
||||||
@@ -263,16 +263,16 @@ The cancel helper should:
|
|||||||
Expected terminal behavior:
|
Expected terminal behavior:
|
||||||
|
|
||||||
- `COMPLETED`
|
- `COMPLETED`
|
||||||
- post to Mattermost
|
- post to MS Teams
|
||||||
- verify `ok`
|
- verify the webhook returned HTTP 2xx, typically `202 Accepted`
|
||||||
- exit
|
- exit
|
||||||
- `FAILED`
|
- `FAILED`
|
||||||
- post to Mattermost
|
- post to MS Teams
|
||||||
- verify `ok`
|
- verify the webhook returned HTTP 2xx, typically `202 Accepted`
|
||||||
- exit
|
- exit
|
||||||
- categorized `COMPLETED` / `FAILED`
|
- categorized `COMPLETED` / `FAILED`
|
||||||
- post once for that grouped sub-run
|
- post once for that grouped sub-run
|
||||||
- verify `ok`
|
- verify the webhook returned HTTP 2xx, typically `202 Accepted`
|
||||||
- continue until the parent request finishes
|
- continue until the parent request finishes
|
||||||
- `CANCELLED`
|
- `CANCELLED`
|
||||||
- write final `CANCELLED` state to `state.json`
|
- write final `CANCELLED` state to `state.json`
|
||||||
|
|||||||
@@ -7,11 +7,11 @@ This folder contains a per-run ATVM watcher service package that is intended to
|
|||||||
Watch an ATVM automation request until it reaches a terminal state, then:
|
Watch an ATVM automation request until it reaches a terminal state, then:
|
||||||
|
|
||||||
- for non-categorized runs:
|
- for non-categorized runs:
|
||||||
- post one final status to Mattermost if the run state is `COMPLETED` or `FAILED`
|
- post one final status to MS Teams if the run state is `COMPLETED` or `FAILED`
|
||||||
- for categorized runs:
|
- for categorized runs:
|
||||||
- detect each sequential categorized sub-run
|
- detect each sequential categorized sub-run
|
||||||
- post one final status per completed categorized sub-run if that grouped run state is `COMPLETED` or `FAILED`
|
- post one final status per completed categorized sub-run if that grouped run state is `COMPLETED` or `FAILED`
|
||||||
- verify each Mattermost post succeeded
|
- verify each MS Teams post succeeded
|
||||||
- write durable watcher state
|
- write durable watcher state
|
||||||
- exit cleanly so the service stops
|
- exit cleanly so the service stops
|
||||||
|
|
||||||
@@ -65,8 +65,8 @@ Typical workflow:
|
|||||||
5. For non-categorized runs, when the run reaches a terminal state:
|
5. For non-categorized runs, when the run reaches a terminal state:
|
||||||
- `COMPLETED` or `FAILED`
|
- `COMPLETED` or `FAILED`
|
||||||
- build the final ATVM status
|
- build the final ATVM status
|
||||||
- send the status to Mattermost
|
- send the status to MS Teams
|
||||||
- verify Mattermost returned `ok`
|
- verify the MS Teams workflow webhook returned HTTP 2xx, typically `202 Accepted`
|
||||||
- mark the run as posted
|
- mark the run as posted
|
||||||
- exit
|
- exit
|
||||||
- `CANCELLED`, `TERMINATED`, `HUNG`, or `UNKNOWN`
|
- `CANCELLED`, `TERMINATED`, `HUNG`, or `UNKNOWN`
|
||||||
@@ -76,7 +76,7 @@ Typical workflow:
|
|||||||
6. For categorized runs:
|
6. For categorized runs:
|
||||||
- detect each grouped sub-run in sequence from the parent run log
|
- detect each grouped sub-run in sequence from the parent run log
|
||||||
- wait for that grouped sub-run to finish
|
- wait for that grouped sub-run to finish
|
||||||
- send one Mattermost post for that grouped sub-run if it reached `COMPLETED` or `FAILED`
|
- send one MS Teams post for that grouped sub-run if it reached `COMPLETED` or `FAILED`
|
||||||
- keep the watcher alive while the parent categorized runner or related child Cypress process is still active
|
- keep the watcher alive while the parent categorized runner or related child Cypress process is still active
|
||||||
- do not treat one completed grouped sub-run as proof that the whole parent request is finished
|
- do not treat one completed grouped sub-run as proof that the whole parent request is finished
|
||||||
- continue to the next grouped sub-run
|
- continue to the next grouped sub-run
|
||||||
@@ -86,8 +86,8 @@ Typical workflow:
|
|||||||
|
|
||||||
The service expects the local credentials file values to be made available on the controller through the service environment:
|
The service expects the local credentials file values to be made available on the controller through the service environment:
|
||||||
|
|
||||||
- `MATTERMOST_ATVM_WEBHOOK`
|
- `MS_TEAMS_ATVM_WEBHOOK`
|
||||||
- `MATTERMOST_ATVM_CHANNEL`
|
- `MS_TEAMS_ATVM_CHANNEL`
|
||||||
|
|
||||||
Optional metadata for better status formatting:
|
Optional metadata for better status formatting:
|
||||||
|
|
||||||
@@ -168,7 +168,7 @@ The helper also:
|
|||||||
./cancel-atvm-run-watcher.sh --build-name e2e-redhat9.6-ubuntu24.04-w2k25-fc
|
./cancel-atvm-run-watcher.sh --build-name e2e-redhat9.6-ubuntu24.04-w2k25-fc
|
||||||
```
|
```
|
||||||
|
|
||||||
This writes a cancellation marker, updates `state.json` to `CANCELLED`, and stops the watcher instance. The watcher will not send Mattermost results for that run.
|
This writes a cancellation marker, updates `state.json` to `CANCELLED`, and stops the watcher instance. The watcher will not send MS Teams results for that run.
|
||||||
|
|
||||||
Runner cancel example:
|
Runner cancel example:
|
||||||
|
|
||||||
|
|||||||
@@ -1627,16 +1627,37 @@ def build_status_markdown(
|
|||||||
return "\n".join(lines)
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
def post_to_mattermost(text: str) -> str:
|
def post_to_teams(text: str) -> str:
|
||||||
webhook = os.environ["MATTERMOST_ATVM_WEBHOOK"]
|
webhook = os.environ["MS_TEAMS_ATVM_WEBHOOK"]
|
||||||
payload = {"text": text}
|
payload = {
|
||||||
channel = os.environ.get("MATTERMOST_ATVM_CHANNEL")
|
"type": "message",
|
||||||
|
"attachments": [
|
||||||
|
{
|
||||||
|
"contentType": "application/vnd.microsoft.card.adaptive",
|
||||||
|
"contentUrl": None,
|
||||||
|
"content": {
|
||||||
|
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
|
||||||
|
"type": "AdaptiveCard",
|
||||||
|
"version": "1.4",
|
||||||
|
"body": [
|
||||||
|
{
|
||||||
|
"type": "TextBlock",
|
||||||
|
"text": text,
|
||||||
|
"wrap": True,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
},
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
channel = os.environ.get("MS_TEAMS_ATVM_CHANNEL")
|
||||||
if channel:
|
if channel:
|
||||||
payload["channel"] = channel
|
payload["channel"] = channel
|
||||||
data = json.dumps(payload).encode()
|
data = json.dumps(payload).encode()
|
||||||
request = urllib.request.Request(webhook, data=data, headers={"Content-Type": "application/json"})
|
request = urllib.request.Request(webhook, data=data, headers={"Content-Type": "application/json"})
|
||||||
with urllib.request.urlopen(request) as response:
|
with urllib.request.urlopen(request) as response:
|
||||||
return response.read().decode().strip()
|
body = response.read().decode().strip()
|
||||||
|
return f"HTTP {response.status}" + (f": {body}" if body else "")
|
||||||
|
|
||||||
|
|
||||||
def sanitize_key(raw: str) -> str:
|
def sanitize_key(raw: str) -> str:
|
||||||
@@ -2334,13 +2355,11 @@ if __name__ == "__main__":
|
|||||||
log_text=current_log_text,
|
log_text=current_log_text,
|
||||||
)
|
)
|
||||||
print(status_text)
|
print(status_text)
|
||||||
response = post_to_mattermost(status_text)
|
response = post_to_teams(status_text)
|
||||||
if response != "ok":
|
subrun_posted_marker.write_text(f"{response}\n", encoding="utf-8")
|
||||||
raise SystemExit(f"Mattermost webhook did not return ok for {subrun['display_name']}: {response!r}")
|
subrun_state["teams_posted"] = True
|
||||||
subrun_posted_marker.write_text("ok\n", encoding="utf-8")
|
subrun_state["teams_response"] = response
|
||||||
subrun_state["mattermost_posted"] = True
|
print(f"[watcher] Teams post confirmed for {subrun['display_name']}: {response}.")
|
||||||
subrun_state["mattermost_response"] = response
|
|
||||||
print(f"[watcher] Mattermost post confirmed for {subrun['display_name']}.")
|
|
||||||
write_state(subrun_state_file, subrun_state)
|
write_state(subrun_state_file, subrun_state)
|
||||||
|
|
||||||
if run_state == "RUNNING":
|
if run_state == "RUNNING":
|
||||||
@@ -2363,14 +2382,12 @@ if __name__ == "__main__":
|
|||||||
print(status_text)
|
print(status_text)
|
||||||
|
|
||||||
if not metadata.get("categorized") and run_state in {"COMPLETED", "FAILED"} and not posted_marker.exists():
|
if not metadata.get("categorized") and run_state in {"COMPLETED", "FAILED"} and not posted_marker.exists():
|
||||||
response = post_to_mattermost(status_text)
|
response = post_to_teams(status_text)
|
||||||
if response != "ok":
|
posted_marker.write_text(f"{response}\n", encoding="utf-8")
|
||||||
raise SystemExit(f"Mattermost webhook did not return ok: {response!r}")
|
state["teams_posted"] = True
|
||||||
posted_marker.write_text("ok\n", encoding="utf-8")
|
state["teams_response"] = response
|
||||||
state["mattermost_posted"] = True
|
|
||||||
state["mattermost_response"] = response
|
|
||||||
write_state(state_file, state)
|
write_state(state_file, state)
|
||||||
print(f"[watcher] Mattermost post confirmed for {build_name}.")
|
print(f"[watcher] Teams post confirmed for {build_name}: {response}.")
|
||||||
|
|
||||||
state["closed_at"] = now_utc().isoformat()
|
state["closed_at"] = now_utc().isoformat()
|
||||||
write_state(state_file, state)
|
write_state(state_file, state)
|
||||||
|
|||||||
Reference in New Issue
Block a user