evandrodevbr
c17192c9fe
Copia local (nao fork, nao submodule) de safedeps, sca-audit, secrets-scan, code-review-security, api-security-review, web-security-review, cve-triage, patch-prioritization, dependency-scanning e hermaguard -- commit exato de origem e licenca de cada uma em skills/NOTICE.md. Elimina o passo de clone externo pro pilar de seguranca do wf-security-review: git clone deste repo ja traz tudo funcional. Corrige tambem o comando de instalacao pro Claude Code nos dois READMEs e no docs/SKILLS.md: Claude Code exige um diretorio por skill (skills/<nome>/SKILL.md), nao aceita os .md soltos que o comando antigo copiava -- confirmado testando a instalacao real.
34 lines
1.3 KiB
Markdown
34 lines
1.3 KiB
Markdown
# Hermaguard Benchmark
|
|
|
|
Score-only. It does **not** run HermaGuard (that's a skill, not a binary). Your
|
|
harness produces a `--json` report per bug; this scores it against ground truth.
|
|
|
|
## Layout
|
|
|
|
- `bugs/<id>/` — one case per planted bug: `code/app.py` (fixture, **no answer-key
|
|
comments**), `pre.diff` (the change that introduced it), `ground-truth.json`
|
|
(the findings a correct review should surface).
|
|
- `results/<id>.json` — a harness's review output for that bug (the shape the
|
|
scorer reads). The committed ones are a **reference run**, see below.
|
|
|
|
## Run
|
|
|
|
```bash
|
|
python3 benchmark.py --bugs bugs --results results --output results/report.html
|
|
```
|
|
|
|
## Reference run (committed in `results/`)
|
|
|
|
A single `claude-opus-4-8` harness reviewed the 5 fixtures. **Not blind:** the
|
|
corpus was authored and reviewed by the same agent, so recall here is an upper
|
|
bound. The corpus exists for blind runs by any harness, seed bugs authored by
|
|
someone other than the reviewer for an independent number.
|
|
|
|
| | Recall | Precision | F1 |
|
|
|---|---|---|---|
|
|
| 5 bugs | 100% | 90% | 93.3% |
|
|
|
|
The one sub-100% precision (bug 001) is a real out-of-ground-truth finding (an
|
|
unclosed DB connection), not a hallucination. Each result file carries a
|
|
`provenance` block (model, mode, blind flag).
|