Fix false ATVM failures from reporter txt fallback

This commit is contained in:
2026-03-30 17:10:25 -04:00
parent 0b379612a8
commit b45375dbbc
2 changed files with 13 additions and 2 deletions

View File

@@ -392,3 +392,12 @@ This file stores run-specific examples only when a run produced a new learning r
- Do not treat the presence of plugin-specific strings or code blocks in the generated `.ts` file as proof that those plugin steps will execute.
- For plugin-specific questions, determine expected behavior from the template/runtime gate and only call it a mismatch if the runtime logic would execute the wrong plugin path.
- Continue verifying that the requested VM set is present in the generated files and `specPattern`, but keep plugin-path validation separate from simple text-presence checks.
## Run Learning: 2026-03-30 (Do not classify reporter TXT logs as failed from generic `error` words)
- Observed failure mode:
- A completed `reboot-redhat8.10-iscsi` run actually passed in the launch log and `Cloud Run Finished` table, but the watcher saved it as failed.
- The TXT fallback matched generic strings such as `auth error encountered` and treated them as proof of host failure.
- Action for future runs:
- Do not classify a reporter TXT artifact as failed just because it contains the word `error`.
- For TXT fallback, require explicit terminal failure markers such as `cy:command error`, `cy:task error`, or real `Error:`/`AssertionError:`/timeout text.
- Prefer the parent run summary when available, because it is less prone to false failure signals than raw per-step console text.