fix(qa): make the quality harness work from a fresh clone

The harness resolved workflows and referenced skills only under
~/.hermes/skills/, so the commands documented in the README ("Quick
start") and in CONTRIBUTING ("Local checks before a PR") produced a
meaningless result on a fresh clone: every workflow scored 0, the gates
reported REPROVADO and the meta-tests failed.

- wf_quality_harness.py: resolve each workflow from the repo
  (workflows/<name>.md) first, then from the installed skills dir; resolve
  referenced skills from the vendored skills/ tree as well. Report the real
  per-workflow cap instead of a fixed 100 (scores were shown as >100%).
- test_quality.py: G3 compared the current score against 2x its own final
  value, so it could never pass and made the documented command exit 1.
  Moved G3 into the --strict family alongside G2, and updated the stale
  docstring (364.1 / 3 workflows -> 1613.2 / 4).
- test_tests.py: meta-test 3 pointed at a path that does not exist in the
  repo, so it was skipped in silence (false green). It now resolves the
  same path the harness uses and FAILS when the workflow is missing; also
  guards the None harness result that previously crashed test 2.

Verified from the clone: harness exit 0, score 1613.2 (matches the
committed snapshot exactly on CPython 3.10 and 3.11); test_quality exit 0
(APROVADO G1,G4,G5); test_tests exit 0 with both mutations detected.
This commit is contained in:
Evandro Fonseca Junior committed 2026-09-14 07:40:14 -03:00
1 parent c17192c9fe
commit eba8bb1d94
3 files changed
+123 -56

No files matched your search

+17 -10
View File
@@ -4,14 +4,18 @@ test_quality.py — testes validados + aprovados (gate de qualidade não-regress
Roda o harness, compara com o baseline snapshot e aplica gates:
Gate 1 (NÃO REGRESSÃO): score total atual >= baseline (364.1). Se caiu, exit 1.
Gate 2 (MELHORIA ≥2×): score total atual >= 2 * baseline (728.2). Meta do usuário.
Gate 3 (POR WORKFLOW): cada um dos 3 workflows >= 2 * seu baseline individual.
Gate 1 (NÃO REGRESSÃO): score total atual >= baseline (1613.2). Se caiu, exit 1.
Gate 2 (MELHORIA ≥2×): score total atual >= 2 * baseline (3226.4). Meta do
projeto, só exigida com --strict (o harness tem teto físico, ver README).
Gate 3 (POR WORKFLOW ≥2×): cada um dos 4 workflows >= 2 * seu baseline
individual. Mesma família do G2: meta aspiracional, só com --strict —
o snapshot guarda os scores FINAIS, então 2× ele mesmo é inalcançável
por construção.
Gate 4 (SKILLS): nenhuma skill referenciada faltando (harness sem sem_match).
Gate 5 (VERIFICAÇÃO POR FASE): toda fase tem >=3 verbos de verificação.
Uso: python3 test_quality.py -> exit 0 se G1,G3,G4,G5 ok (evidência)
python3 test_quality.py --strict -> também exige G2 (2x) — meta final
Uso: python3 test_quality.py -> exit 0 se G1,G4,G5 ok (evidência)
python3 test_quality.py --strict -> também exige G2 e G3 (meta final)
"""
import json, os, subprocess, sys
@@ -58,13 +62,16 @@ def main():
if strict:
ok &= g2
# Gate 3 — por workflow, 2x individual
print("[G3] POR WORKFLOW (2x individual):")
# Gate 3 — por workflow, 2x individual (mesma família do G2: só em --strict)
print("[G3] POR WORKFLOW (2x individual, só com --strict):")
all_g3_ok = True
for name, b in base_per.items():
cur = cur_per.get(name, 0)
g3i = cur >= 2 * max(b, 0.1)
print(f" {name:18} base={b:6.1f} atual={cur:6.1f} 2x={2*max(b,0.1):6.1f} -> {'OK' if g3i else 'FALHOU'}")
ok &= g3i
print(f" {name:18} base={b:6.1f} atual={cur:6.1f} 2x={2*max(b,0.1):6.1f} -> {'OK' if g3i else 'ainda não (meta final)'}")
all_g3_ok &= g3i
if strict:
ok &= all_g3_ok
# Gate 4 — skills
print("[G4] SKILLS REFERENCIADAS:")
@@ -84,7 +91,7 @@ def main():
print("="*72)
if ok:
print(f"RESULTADO: APROVADO (gates G1,G3,G4,G5 passaram{' + G2' if strict and g2 else ''}; score {cur_total:.1f})")
print(f"RESULTADO: APROVADO (gates G1,G4,G5 passaram{' + G2/G3' if strict else ''}; score {cur_total:.1f})")
sys.exit(0)
else:
print(f"RESULTADO: REPROVADO — gates abaixo falharam. Score {cur_total:.1f}. Rodar refinamento de novo.")