Add ATVM coverage and functional sections to status template
- update the ATVM status template to include COVERAGE: and FUNCTIONALLY: sections ahead of the existing summary tables - document that COVERAGE: should describe intended run scope without listing target hosts - document that FUNCTIONALLY: should summarize the intended workflow steps at a high level - align the ATVM automation guide so local status output and Mattermost posts use the expanded default format
This commit is contained in:
@@ -190,24 +190,28 @@ When asked for one VM or a VM set:
|
|||||||
## Status Reporting Format
|
## Status Reporting Format
|
||||||
When the operator asks for the status of an ATVM automation run, report in this order:
|
When the operator asks for the status of an ATVM automation run, report in this order:
|
||||||
1. Heading/title using the run `build_name`.
|
1. Heading/title using the run `build_name`.
|
||||||
2. Completed machines with machine name first and status second for each machine.
|
2. `COVERAGE:` section describing what the run was intended to cover, excluding the target-host list.
|
||||||
3. Notes.
|
3. `FUNCTIONALLY:` section describing the intended workflow steps at a high level.
|
||||||
4. Skipped machines with reason.
|
4. Completed machines with machine name first and status second for each machine.
|
||||||
5. Remaining machines still to run.
|
5. Notes.
|
||||||
6. Summary counts for finished, passed, failed, and skipped machines.
|
6. Skipped machines with reason.
|
||||||
7. Timing details:
|
7. Remaining machines still to run.
|
||||||
|
8. Summary counts for finished, passed, failed, and skipped machines.
|
||||||
|
9. Timing details:
|
||||||
- start time
|
- start time
|
||||||
- end time if complete
|
- end time if complete
|
||||||
- total run time if complete, or elapsed run time if still running
|
- total run time if complete, or elapsed run time if still running
|
||||||
- quickest completed test runtime
|
- quickest completed test runtime
|
||||||
- longest completed test runtime
|
- longest completed test runtime
|
||||||
- average completed test runtime
|
- average completed test runtime
|
||||||
8. Estimated completion time.
|
10. Estimated completion time.
|
||||||
|
|
||||||
Status-report expectations:
|
Status-report expectations:
|
||||||
- Use the same display layout for every ATVM automation status response regardless of test type (`e2e`, `systemOS`, `reboot`, `migrateops`, and others).
|
- Use the same display layout for every ATVM automation status response regardless of test type (`e2e`, `systemOS`, `reboot`, `migrateops`, and others).
|
||||||
- Use `/home/aw/code/cds/atvm/docs/automation/status-template.md` as the default template for both local status output and Mattermost status posts.
|
- Use `/home/aw/code/cds/atvm/docs/automation/status-template.md` as the default template for both local status output and Mattermost status posts.
|
||||||
- The default ATVM status template uses Markdown tables for `SUMMARY:`, `HOSTS:`, and `TIMING:` and uses `NOTES:` for flat operator-facing notes.
|
- The default ATVM status template uses flat bullet-list sections for `COVERAGE:` and `FUNCTIONALLY:`, Markdown tables for `SUMMARY:`, `HOSTS:`, and `TIMING:`, and uses `NOTES:` for flat operator-facing notes.
|
||||||
|
- In `COVERAGE:`, describe the template, datastore/config family, migration style, and plugin/integration path, but do not list target hosts there.
|
||||||
|
- In `FUNCTIONALLY:`, summarize what the run is intended to do at a high level, even when the run failed before reaching those steps.
|
||||||
- Treat references to the "ATVM automation run" or "automation run" as referring to this ATVM folder workflow and the automation VM at `192.168.3.190`, not to Cirrus project operations such as the `atvm - cypress` project.
|
- Treat references to the "ATVM automation run" or "automation run" as referring to this ATVM folder workflow and the automation VM at `192.168.3.190`, not to Cirrus project operations such as the `atvm - cypress` project.
|
||||||
- Treat a status request as a request for live status by default.
|
- Treat a status request as a request for live status by default.
|
||||||
- Unless the operator explicitly asks to send the status to Mattermost, print the status only in the local terminal response.
|
- Unless the operator explicitly asks to send the status to Mattermost, print the status only in the local terminal response.
|
||||||
|
|||||||
@@ -10,6 +10,18 @@ Use this as the default ATVM automation run-status template for:
|
|||||||
## ATVM Run Status
|
## ATVM Run Status
|
||||||
### <build_name>
|
### <build_name>
|
||||||
|
|
||||||
|
**COVERAGE:**
|
||||||
|
- template: `<template-name>`
|
||||||
|
- datastore/config family: `<config family>`
|
||||||
|
- migration style: `<high-level test style>`
|
||||||
|
- integration/plugin path: `<integration/plugin>`
|
||||||
|
- scope of this run: `<batch or run scope>`
|
||||||
|
|
||||||
|
**FUNCTIONALLY:**
|
||||||
|
- <intended functional step>
|
||||||
|
- <intended functional step>
|
||||||
|
- <intended functional step>
|
||||||
|
|
||||||
**SUMMARY:**
|
**SUMMARY:**
|
||||||
|
|
||||||
| Metric | Value |
|
| Metric | Value |
|
||||||
@@ -45,10 +57,11 @@ Use this as the default ATVM automation run-status template for:
|
|||||||
```
|
```
|
||||||
|
|
||||||
## Rules
|
## Rules
|
||||||
- Keep `SUMMARY:`, `HOSTS:`, `TIMING:`, and `NOTES:` in that order.
|
- Keep `COVERAGE:`, `FUNCTIONALLY:`, `SUMMARY:`, `HOSTS:`, `TIMING:`, and `NOTES:` in that order.
|
||||||
- Use the title format:
|
- Use the title format:
|
||||||
- `## ATVM Run Status`
|
- `## ATVM Run Status`
|
||||||
- `### <build_name>`
|
- `### <build_name>`
|
||||||
|
- Use flat bullet lists for `COVERAGE:` and `FUNCTIONALLY:`.
|
||||||
- Use Markdown tables for `SUMMARY:`, `HOSTS:`, and `TIMING:`.
|
- Use Markdown tables for `SUMMARY:`, `HOSTS:`, and `TIMING:`.
|
||||||
- Use one host per row in the `HOSTS:` section.
|
- Use one host per row in the `HOSTS:` section.
|
||||||
- For completed hosts, prefer:
|
- For completed hosts, prefer:
|
||||||
@@ -59,4 +72,6 @@ Use this as the default ATVM automation run-status template for:
|
|||||||
- `⏭️ SKIP`
|
- `⏭️ SKIP`
|
||||||
- Keep `Detail` concise.
|
- Keep `Detail` concise.
|
||||||
- Put broader context under `NOTES:`, not in the host table.
|
- Put broader context under `NOTES:`, not in the host table.
|
||||||
|
- `COVERAGE:` should describe what the run was intended to cover without listing target hosts.
|
||||||
|
- `FUNCTIONALLY:` should describe the intended workflow steps at a high level, even if the run failed before completing them.
|
||||||
- Use the same template for Mattermost and local operator-visible status output.
|
- Use the same template for Mattermost and local operator-visible status output.
|
||||||
|
|||||||
Reference in New Issue
Block a user