Evandro Fonseca Junior eba8bb1d94 fix(qa): make the quality harness work from a fresh clone
The harness resolved workflows and referenced skills only under
~/.hermes/skills/, so the commands documented in the README ("Quick
start") and in CONTRIBUTING ("Local checks before a PR") produced a
meaningless result on a fresh clone: every workflow scored 0, the gates
reported REPROVADO and the meta-tests failed.

- wf_quality_harness.py: resolve each workflow from the repo
  (workflows/<name>.md) first, then from the installed skills dir; resolve
  referenced skills from the vendored skills/ tree as well. Report the real
  per-workflow cap instead of a fixed 100 (scores were shown as >100%).
- test_quality.py: G3 compared the current score against 2x its own final
  value, so it could never pass and made the documented command exit 1.
  Moved G3 into the --strict family alongside G2, and updated the stale
  docstring (364.1 / 3 workflows -> 1613.2 / 4).
- test_tests.py: meta-test 3 pointed at a path that does not exist in the
  repo, so it was skipped in silence (false green). It now resolves the
  same path the harness uses and FAILS when the workflow is missing; also
  guards the None harness result that previously crashed test 2.

Verified from the clone: harness exit 0, score 1613.2 (matches the
committed snapshot exactly on CPython 3.10 and 3.11); test_quality exit 0
(APROVADO G1,G4,G5); test_tests exit 0 with both mutations detected.
2026-09-14 07:40:14 -03:00

Dev Workflows

Agent workflows as SKILL.md files: frontend, backend, architecture and security code review that never skip verification.

License: MIT Python 3.10+ Agent Skills PRs Welcome

Turn your coding agent into a repeatable, verifiable process. Each workflow defines phases, concrete verification steps, real commands, and hard rules. The agent delivers work that can be checked, not just claimed.

🇧🇷 Ler em português


Why

Agents are great at generating code and terrible at admitting they don't know. This project fixes the second part.

Every workflow here forces the agent to prove the work at each phase:

  • Verification steps: run the build, run the tests, hit the API, consult the registry. Not "I think it works".
  • Acceptance criteria: checkbox gates that must be marked before a phase closes.
  • Hard rules: "never merge with a red build", "never claim 'no CVE' without querying OSV.dev".
  • A QA harness: a scorecard that measures workflow quality objectively, plus meta-tests that corrupt the harness to prove it detects regressions.

Table of contents

Workflows

Workflow File Covers
🎨 Frontend / UI workflows/wf-frontend.md Design direction before code, anti-AI-slop, React performance, visual QA for PDFs
⚙️ Backend / API workflows/wf-backend.md Contract before handler, safe defaults, OWASP audit, tests + adversarial bug hunt
🏗️ Architecture workflows/wf-architecture.md Measurable requirements, C4 diagrams, ADRs, threat modeling, review
🛡️ Security review workflows/wf-security-review.md Code review (front + back), known CVEs and live lookup of new ones
📄 README workflows/wf-readme.md Create/audit/rewrite READMEs grounded in real repo facts

Each file is a standard SKILL.md. The description in the frontmatter is the trigger: when the agent's context matches it, the workflow loads automatically. Works in Hermes Agent, Claude Code, and anything that reads the Agent Skills format.

Security: do not trust the training cutoff

A model's knowledge of CVEs stops at its training date. Anything disclosed after that, it has never seen, so "no vulnerabilities" from memory is a guess, not an answer.

The security workflow is built around that fact:

Code/diff → SCOPE → HUNT (known CVEs) → CONSULT (live sources) → CODE REVIEW → REPORT + GATE
                    │                      │
                    │                      └─ OSV.dev · GitHub Advisory DB · NVD
                    └──────────────┘

Every security review must consult live sources and stamp the date. A review with no source and no date is expired on arrival.

Skills used, and their owners

Workflows load these skills by context. The three pillars of the security review come from primary security sources (OWASP, OSV.dev, UnitOne SecuritySkills), not from a model's memory.

🛡️ Security review: the three pillars

Vendored under skills/ — see skills/NOTICE.md for exact origin and license.

Skill What it does Owner
safedeps Queries OSV.dev in real time; returns CVE + severity + fixed version for a package@version. Covers the gap a frozen model can't. Jeneidi
sca-audit Supply-chain audit of dependencies with reachability analysis and CWE mapping. OWASP
code-review-security Systematic security code review mapped to OWASP Top 10 + ASVS. OWASP

🛡️ Security review: supporting skills

Also vendored under skills/.

Skill What it does Owner
secrets-scan Finds hardcoded credentials / API keys in code and git history. OWASP
api-security-review API review against OWASP API Security Top 10. OWASP
web-security-review Web app review against OWASP Top 10. OWASP
cve-triage Prioritizes CVEs using CVSS 4.0 / EPSS / CISA KEV. UnitOneAI
patch-prioritization Decides remediation order. UnitOneAI
dependency-scanning Dependency tree scanning. UnitOneAI
hermaguard Adversarial review: static pre-scan + 3 specialist agents (edge cases, attack, blast radius). Sahil-SS9

🎨 Frontend / UI

Skill What it does Owner
frontend-design Intentional visual direction, typography, avoiding "generic AI" output. Anthropic
web-design Cohesive web aesthetics. KAOPU-XiaoPu
vercel-react-best-practices 40+ React/Next performance rules from Vercel engineering. Vercel Labs
vercel-composition-patterns Compound components, clean composition. Vercel Labs
animate Purpose-driven motion. emilkowalski
impeccable The missing design vocabulary for agents: 23 commands (craft, shape, audit, polish, animate, live) and 59 deterministic anti-slop rules. The most-used frontend design skill (230k+ installs). pbakaus
anti-ai-slop Detects "generated-by-AI" visual patterns (auto-loaded, local). community skill
avoid-ai-writing Removes AI-isms from microcopy, labels and docs. conorbronsdon

⚙️ Backend / API

Skill What it does Owner
secure-coding Secure coding patterns covering 15 stacks. securityreviewai
bola-detector Broken object-level authorization (OWASP API1). apisec-inc
auth-rbac-scaffold Authentication + RBAC (OWASP API2 / API5). apisec-inc
injection-checker SQL / ORM / shell / template injection (OWASP API8). apisec-inc
openapi-hardener Sanitizes OpenAPI / Zod / Pydantic schemas (OWASP API3). apisec-inc
api-security-review Full OWASP API Top 10 review. apisec-inc
security-test-generator Generates security test suites. apisec-inc

🏗️ Architecture

Skill What it does Owner
system-design HelloInterview system-design framework. Kotivskyi
c4-architecture C4 diagrams (Mermaid / Structurizr). Kotivskyi
isaqb-architecture-governance arc42 + ADRs. Kotivskyi
secure-architecture-governance STRIDE+CIA threat models, security ADRs. Kotivskyi

📖 Full install instructions for every skill live in docs/SKILLS.md.

Quick start

# 1. Copy workflows into your agent's skills directory
# Hermes Agent:
cp workflows/*.md ~/.hermes/skills/software-development/

# Claude Code: each skill needs its own directory (won't load a loose .md)
mkdir -p ~/.claude/skills
for f in workflows/*.md; do
  name=$(basename "$f" .md)
  mkdir -p "$HOME/.claude/skills/$name"
  cp "$f" "$HOME/.claude/skills/$name/SKILL.md"
done

# 2. (Optional, for security review) the supporting skills are already
# vendored in skills/ -- no external clone needed, just copy (Hermes; for
# Claude Code use the same loop as step 1, swapping "workflows" for "skills"):
cp -r skills/* ~/.hermes/skills/

# 3. Use it
# "review this PR for security"  -> wf-security-review loads
# "build a new page"              -> wf-frontend loads

The QA harness

The qa/ directory keeps the workflows themselves honest:

cd qa

python3 wf_quality_harness.py   # score each workflow on 30+ criteria
python3 test_quality.py         # gates: no regression, valid skills, phases verified
python3 test_tests.py           # meta-tests: corrupt the harness, prove it detects the corruption

The meta-tests are the interesting part. They deliberately break the harness and the workflows, then confirm the score drops. If the score didn't move, the detector would be useless. These tests stay green as a contract.

Quality tests and measured gains

These are real numbers from the harness runs, stored in qa/snapshots/. The baseline column is the state before the refinement work; the final column is the result after it.

Workflow quality score (same harness, before → after)

Workflow Baseline Final Gain
wf-frontend 201.0 385.5 +92%
wf-backend 215.0 424.0 +97%
wf-architecture 199.9 413.7 +107%
wf-security-review 332.0 390.0 +17%
Total 615.9 1223.2 +98.6%

That is almost exactly double the quality of the workflows, measured with the same ruler.

The ruler itself also grew stricter over the project (13 → 30+ criteria: verification per phase, real commands, anti-patterns, acceptance criteria, artifacts, flow). That is why the early snapshots show lower totals: base-364 (13 criteria) → baseline-v2 (536.7) → baseline-v3 (675.0) → baseline-v4 (884.6) → baseline-final (1613.2 with four workflows). The fair before/after comparison above uses the final ruler for both sides.

Gates (regression checks): all green

Gate What it checks Status
G1 Score did not drop below baseline ✅
G4 Every referenced skill exists ✅ 4/4 workflows
G5 Every phase has ≥3 verification verbs ✅ 4/4 workflows
Meta-tests Corrupt harness/workflows → score drops (proves detector works) ✅

What the refinement added

Every workflow gained the same batch of real improvements:

  • Verification steps with actual commands (curl, grep, python3, git, npm, pytest) instead of prose.
  • Per-phase acceptance checklists (markable [ ] gates).
  • Anti-pattern blocks per phase (what not to do, and why).
  • Executable-command sections and worked examples per phase.
  • Checkpoints that pause for user approval between phases.

The security workflow gained a hard rule: never claim "no CVE" without querying OSV.dev / GitHub Advisory. A model's knowledge stops at its training cutoff; the live sources don't.

Context on the "double" target

The harness has a physical ceiling (sum of all criterion caps). The numeric "2× of baseline" target would need a ruler with more resolution to be expressible, so inflating the score to hit it would be gaming the metric. What shipped is the honest version: real quality ~doubled on the same ruler (+98.6%), with meta-tests proving the ruler detects regressions.

Repository layout

dev-workflows/
├── README.md            # this file
├── README.pt-BR.md      # Portuguese version
├── workflows/           # the four SKILL.md workflows
│   ├── wf-frontend.md
│   ├── wf-backend.md
│   ├── wf-architecture.md
│   └── wf-security-review.md
├── docs/
│   └── SKILLS.md        # full skill inventory + install steps
├── CONTRIBUTING.md      # what to improve, local checks, PR process
└── qa/                  # quality harness, gates, meta-tests
    ├── wf_quality_harness.py
    ├── test_quality.py
    ├── test_tests.py
    └── snapshots/

License

MIT © 2026 Evandro Fonseca Junior


⭐ If this saves you from one "works on my machine" merge, star it.

S
Description
Convenções e automações compartilhadas entre os repositórios
Readme MIT
759 KiB
0 Stars 1 Watchers 0 Forks
Languages
Python 94.6%
HTML 5.2%
JavaScript 0.2%