anthony.wen/cds-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refine categorized timing metrics and coverage output

Browse Source
This commit is contained in:
anthony.wen
2026-03-27 18:49:11 -04:00
parent 06b8098499
commit 6c7ba5212b
4 changed files with 34 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
atvm/docs/automation/guide.md
View File

@@ -243,6 +243,8 @@ Status-report expectations:
- Do not include internal watcher fallback notes in `NOTES:` such as `check-xml-files.ts` validation confirmations or reporter-artifact recovery details.
- The `HOSTS:` table includes `Host`, `Kernel`, `Status`, and `Detail` columns in that order.
- In `COVERAGE:`, describe the important `cmc-templates.py` command inputs such as template, categorize mode, datastore/config family, config filename, migration style, plugin/integration path, and other operator-relevant run options, but do not list target hosts there or include verbose prose scope descriptions.
- If `categorize mode: enabled` is already shown in `COVERAGE:`, do not also repeat `--categorize` under `run options`.
- When grouped categorized timing is reconstructed from host reporter artifacts, derive per-host quickest/longest/average durations from the sequence of recovered host timestamps and the grouped end time instead of leaving those metrics as `n/a`.
- In `TEST FLOW:`, show the template-specific numbered run flow once for the whole test, not per host.
- Resolve the flow from the run template name.
- `cmc-e2e` currently uses the 22-step migration flow documented in `/home/aw/code/cds/atvm/docs/automation/status-template.md`.

8
atvm/docs/automation/run-learnings.md
View File

@@ -321,6 +321,14 @@ This file stores run-specific examples only when a run produced a new learning r
- Store and display the exact `cmc-templates.py` command in `NOTES:`.
- Omit only the outer remote-execution wrapper.
## Run Learning: 2026-03-27 (Avoid redundant categorize flags and infer grouped timing stats)
- Observed requirement:
- When `categorize mode: enabled` is already shown in `COVERAGE:`, repeating `--categorize` under `run options` is redundant.
- Grouped categorized results should still show `quickest`, `longest`, and `average` when those values can be inferred from recovered host timing.
- Action for future runs:
- Do not repeat `--categorize` under `run options` when categorize mode is already shown separately.
- When grouped host results are reconstructed from reporter artifacts, infer per-host durations from the recovered host timestamp sequence and grouped end time so grouped timing stats do not default to `n/a` unnecessarily.
## Run Learning: 2026-03-27 (Do not auto-add blacklist excludes for explicitly specified VMs)
- Observed requirement:
- When the operator explicitly specifies the VM or VM list to run, they do not want the maintained `--exclude_partial_match` blacklist added automatically.

2
atvm/docs/automation/status-template.md
View File

@@ -80,6 +80,8 @@ Use this as the default ATVM automation run-status template for:
- Do not include internal fallback notes in `NOTES:` such as "`check-xml-files.ts` validation passed" or "host details were derived from reporter artifacts."
- `COVERAGE:` should describe what the run was intended to cover without listing target hosts.
- `COVERAGE:` should mostly mirror the important `cmc-templates.py` command inputs such as template, categorize mode, config filename, integration/plugin path, and important flags like `--ignore_force_shutdown`.
- If `categorize mode: enabled` is shown, do not also repeat `--categorize` under `run options`.
- When grouped categorized timing is reconstructed from host reporter artifacts, still populate `quickest`, `longest`, and `average` from inferred per-host durations when possible.
- `TEST FLOW:` should describe the template-specific numbered run flow once for the whole test, not per host.
- The watcher resolves `TEST FLOW:` from the run template name.
- `cmc-e2e` currently uses this flow: