evandro/dev-workflows
1
0 Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

chore(qa): replace the keyword-count harness with a structural lint and gate tests

Browse Source 
The old harness scored occurrences of words like VERIFICAR, so its score
measured wording, and it failed on a fresh clone. The lint checks frontmatter,
size limits, references and links; test_gate.py exercises the gate and hooks
on a throwaway repo; check_upstream.py flags vendored skills behind upstream.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
EvandroandClaude Opus 5.5 committed 2026-09-24 21:52:06 -03:00
1 parent 29b5ebf400
commit f77b0b440d
15 files changed
+560 -1549
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+81 -224
View File
@@ -2,14 +2,11 @@
# Dev Workflows
**Agent workflows as SKILL.md files**: frontend, backend, architecture and security code review that never skip verification.
**A Claude Code plugin (also for opencode and Hermes Agent) that sizes the process to the request**: a router picks the tier, least-privilege agents do the work, and a deterministic quality gate decides when it is done.
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/Python-3.10+-3776AB?logo=python&logoColor=white)](qa/wf_quality_harness.py)
[![Agent Skills](https://img.shields.io/badge/Agent%20Skills-1.0-8A2BE2)](https://agentskills.io)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/evandrodevbr/dev-workflows/pulls)
Turn your coding agent into a repeatable, verifiable process. Each workflow defines phases, concrete verification steps, real commands, and hard rules. The agent delivers work that can be *checked*, not just *claimed*.
[![CI](https://github.com/evandrodevbr/dev-workflows/actions/workflows/ci.yml/badge.svg)](.github/workflows/ci.yml)
**🇧🇷 [Ler em português](README.pt-BR.md)**
@@ -19,251 +16,111 @@ Turn your coding agent into a repeatable, verifiable process. Each workflow defi
## Why
Agents are great at generating code and terrible at admitting they don't know. This project fixes the second part.
Agents write code fast and are bad at two things: knowing when a task deserves more process, and
admitting what they did not verify. This plugin handles both:
Every workflow here forces the agent to **prove** the work at each phase:
- **Process proportional to the request.** A typo does not get a design review; an auth change does.
- **Proof, not claims.** "Done" requires a quality-gate report on the current diff. A tool that is not installed shows up as *unverified*, never as passed.
- **Least code.** Reuse before writing, delete before adding, no dependency without a reason.
- **Security from live sources.** A model's CVE knowledge stops at its training cutoff, so reviews query OSV.dev and stamp the date.
- **Verification steps**: run the build, run the tests, hit the API, consult the registry. Not "I think it works".
- **Acceptance criteria**: checkbox gates that must be marked before a phase closes.
- **Hard rules**: "never merge with a red build", "never claim 'no CVE' without querying OSV.dev".
- **A QA harness**: a scorecard that measures workflow quality objectively, plus meta-tests that corrupt the harness to prove it detects regressions.
## How it works
## Table of contents
`dev-router` loads first and declares a tier:
- [Workflows](#workflows)
- [Security: do not trust the training cutoff](#security-do-not-trust-the-training-cutoff)
- [Skills used, and their owners](#skills-used-and-their-owners)
- [Quick start](#quick-start)
- [The QA harness](#the-qa-harness)
- [Quality tests and measured gains](#quality-tests-and-measured-gains)
- [Repository layout](#repository-layout)
- [License](#license)
## Workflows
| Workflow | File | Covers |
| Tier | When | What runs |
|---|---|---|
| 🎨 **Frontend / UI** | [workflows/wf-frontend.md](workflows/wf-frontend.md) | Design direction before code, anti-AI-slop, React performance, visual QA for PDFs |
| ⚙️ **Backend / API** | [workflows/wf-backend.md](workflows/wf-backend.md) | Contract before handler, safe defaults, OWASP audit, tests + adversarial bug hunt |
| 🏗️ **Architecture** | [workflows/wf-architecture.md](workflows/wf-architecture.md) | Measurable requirements, C4 diagrams, ADRs, threat modeling, review |
| 🛡️ **Security review** | [workflows/wf-security-review.md](workflows/wf-security-review.md) | Code review (front + back), known CVEs **and** live lookup of new ones |
| 📄 **README** | [workflows/wf-readme.md](workflows/wf-readme.md) | Create/audit/rewrite READMEs grounded in real repo facts |
| 0, trivial | nothing changes behavior | the change + gate tier 0 |
| 1, small | 1–3 files, no new public surface | `wf-bugfix` or `wf-refactor`, regression test, gate tier 1 |
| 2, feature | new endpoint/screen, contract change, new dependency | explore, short plan, TDD, independent review, gate tier 2 |
| 3, large/risky | several modules, existing data, architecture, release | EARS spec + ADR approved by you, reviews, security audit, gate tier 3 |
Each file is a standard `SKILL.md`. The `description` in the frontmatter is the trigger: when the agent's context matches it, the workflow loads automatically. Works in [Hermes Agent](https://hermes-agent.nousresearch.com), Claude Code, and anything that reads the Agent Skills format.
Security triggers raise the tier at any point: auth, money, personal data, uploads, raw SQL,
shell/exec, deserialization, schema migrations, new dependencies, features that call an LLM.
## Security: do not trust the training cutoff
## What is inside
A model's knowledge of CVEs stops at its training date. Anything disclosed after that, it has never seen, so "no vulnerabilities" from memory is a guess, not an answer.
| Kind | Items |
|---|---|
| Router and rules | `dev-router`, `lean-code` |
| Workflows | `wf-bugfix`, `wf-refactor`, `wf-frontend`, `wf-backend`, `wf-architecture`, `wf-security-review`, `wf-readme` |
| Agents | `explorer` (haiku, read-only), `planner` (opus, read-only), `implementer` (sonnet), `reviewer` (opus, read-only, sees only the diff), `security-auditor` (opus, read-only), `ui-critic` (sonnet, read-only), `verifier` (haiku, runs the gate) |
| Hooks | `Stop`: blocks "done" while code changed without a fresh passing gate · `PreToolUse`: denies `--no-verify` and force-push, asks before `reset --hard`, `clean -f`, `DROP TABLE` · `PostToolUse`: formats edited files with the project's own formatter |
| Quality gate | [`scripts/quality-gate`](scripts/quality-gate): lint, typecheck, tests, gitleaks (tier 0); diff coverage, semgrep, dead code (tier 1); build, mutation testing, osv-scanner, architecture rules, duplication (tier 2); trivy, requirement→test traceability (tier 3) |
| Vendored skills | security pillar (OWASP secure-agent-playbook, UnitOneAI SecuritySkills, safedeps, hermaguard) and taste-skill for frontend. Origin and license: [`skills/NOTICE.md`](skills/NOTICE.md) |
The security workflow is built around that fact:
The gate runs only the checks of the requested tier, only on the diff, and needs no AI: it is
plain tooling. Checks that need the app running (Lighthouse CI budgets, Playwright + axe, schemathesis,
k6) are added per project in `.dev-workflows.toml`. Reference:
[`skills/dev-router/references/quality-gate.md`](skills/dev-router/references/quality-gate.md).
```
Code/diff → SCOPE → HUNT (known CVEs) → CONSULT (live sources) → CODE REVIEW → REPORT + GATE
│ │
│ └─ OSV.dev · GitHub Advisory DB · NVD
└──────────────┘
```
Every security review **must** consult live sources and stamp the date. A review with no source and no date is expired on arrival.
## Skills used, and their owners
Workflows load these skills by context. The three pillars of the security review come from primary security sources (OWASP, OSV.dev, UnitOne SecuritySkills), not from a model's memory.
### 🛡️ Security review: the three pillars
_Vendored under `skills/` — see [`skills/NOTICE.md`](skills/NOTICE.md) for exact origin and license._
| Skill | What it does | Owner |
|---|---|---|
| [`safedeps`](https://github.com/Jeneidi/safedeps) | Queries OSV.dev in real time; returns CVE + severity + fixed version for a `package@version`. Covers the gap a frozen model can't. | [Jeneidi](https://github.com/Jeneidi) |
| [`sca-audit`](https://github.com/OWASP/secure-agent-playbook) | Supply-chain audit of dependencies with reachability analysis and CWE mapping. | [OWASP](https://github.com/OWASP) |
| [`code-review-security`](https://github.com/OWASP/secure-agent-playbook) | Systematic security code review mapped to OWASP Top 10 + ASVS. | [OWASP](https://github.com/OWASP) |
### 🛡️ Security review: supporting skills
_Also vendored under `skills/`._
| Skill | What it does | Owner |
|---|---|---|
| [`secrets-scan`](https://github.com/OWASP/secure-agent-playbook) | Finds hardcoded credentials / API keys in code and git history. | [OWASP](https://github.com/OWASP) |
| [`api-security-review`](https://github.com/OWASP/secure-agent-playbook) | API review against OWASP API Security Top 10. | [OWASP](https://github.com/OWASP) |
| [`web-security-review`](https://github.com/OWASP/secure-agent-playbook) | Web app review against OWASP Top 10. | [OWASP](https://github.com/OWASP) |
| [`cve-triage`](https://github.com/UnitOneAI/SecuritySkills) | Prioritizes CVEs using CVSS 4.0 / EPSS / CISA KEV. | [UnitOneAI](https://github.com/UnitOneAI) |
| [`patch-prioritization`](https://github.com/UnitOneAI/SecuritySkills) | Decides remediation order. | [UnitOneAI](https://github.com/UnitOneAI) |
| [`dependency-scanning`](https://github.com/UnitOneAI/SecuritySkills) | Dependency tree scanning. | [UnitOneAI](https://github.com/UnitOneAI) |
| [`hermaguard`](https://github.com/Sahil-SS9/hermaguard) | Adversarial review: static pre-scan + 3 specialist agents (edge cases, attack, blast radius). | [Sahil-SS9](https://github.com/Sahil-SS9) |
### 🎨 Frontend / UI
| Skill | What it does | Owner |
|---|---|---|
| [`frontend-design`](https://github.com/anthropics/skills) | Intentional visual direction, typography, avoiding "generic AI" output. | [Anthropic](https://github.com/anthropics) |
| [`web-design`](https://github.com/KAOPU-XiaoPu/web-design) | Cohesive web aesthetics. | [KAOPU-XiaoPu](https://github.com/KAOPU-XiaoPu) |
| [`vercel-react-best-practices`](https://github.com/vercel-labs/agent-skills) | 40+ React/Next performance rules from Vercel engineering. | [Vercel Labs](https://github.com/vercel-labs) |
| [`vercel-composition-patterns`](https://github.com/vercel-labs/agent-skills) | Compound components, clean composition. | [Vercel Labs](https://github.com/vercel-labs) |
| [`animate`](https://github.com/emilkowalski/skill) | Purpose-driven motion. | [emilkowalski](https://github.com/emilkowalski) |
| [`impeccable`](https://github.com/pbakaus/impeccable) | The missing design vocabulary for agents: 23 commands (craft, shape, audit, polish, animate, live) and 59 deterministic anti-slop rules. The most-used frontend design skill (230k+ installs). | [pbakaus](https://github.com/pbakaus) |
| [`design-taste-frontend`](https://github.com/Leonxlnx/taste-skill) | Generates new UI with a direction inferred from the brief; three 1–10 dials (variance, motion, density). Vendored in `skills/`. | [Leonxlnx](https://github.com/Leonxlnx) |
| [`redesign-existing-projects`](https://github.com/Leonxlnx/taste-skill) | Audits an existing UI and upgrades it without breaking behavior. Vendored in `skills/`. | [Leonxlnx](https://github.com/Leonxlnx) |
| [`anti-ai-slop`](https://github.com/evandrodevbr/dev-workflows) | Detects "generated-by-AI" visual patterns (auto-loaded, local). | community skill |
| [`avoid-ai-writing`](https://github.com/conorbronsdon/avoid-ai-writing) | Removes AI-isms from microcopy, labels and docs. | [conorbronsdon](https://github.com/conorbronsdon) |
### ⚙️ Backend / API
| Skill | What it does | Owner |
|---|---|---|
| [`secure-coding`](https://github.com/securityreviewai/secure-coding-skill) | Secure coding patterns covering 15 stacks. | [securityreviewai](https://github.com/securityreviewai) |
| [`bola-detector`](https://github.com/apisec-inc/apisec-skills) | Broken object-level authorization (OWASP API1). | [apisec-inc](https://github.com/apisec-inc) |
| [`auth-rbac-scaffold`](https://github.com/apisec-inc/apisec-skills) | Authentication + RBAC (OWASP API2 / API5). | [apisec-inc](https://github.com/apisec-inc) |
| [`injection-checker`](https://github.com/apisec-inc/apisec-skills) | SQL / ORM / shell / template injection (OWASP API8). | [apisec-inc](https://github.com/apisec-inc) |
| [`openapi-hardener`](https://github.com/apisec-inc/apisec-skills) | Sanitizes OpenAPI / Zod / Pydantic schemas (OWASP API3). | [apisec-inc](https://github.com/apisec-inc) |
| [`api-security-review`](https://github.com/apisec-inc/apisec-skills) | Full OWASP API Top 10 review. | [apisec-inc](https://github.com/apisec-inc) |
| [`security-test-generator`](https://github.com/apisec-inc/apisec-skills) | Generates security test suites. | [apisec-inc](https://github.com/apisec-inc) |
> ⚠️ `api-security-review` is two unrelated skills sharing one name: the OWASP one vendored in `skills/` (security pillar, above) and the apisec-inc one in this table. Only the OWASP copy ships in this repository. Install the one your workflow needs — with both present, agents that resolve by skill name report a collision.
### 🏗️ Architecture
| Skill | What it does | Owner |
|---|---|---|
| [`system-design`](https://github.com/Kotivskyi/architecture-governance-skills) | HelloInterview system-design framework. | [Kotivskyi](https://github.com/Kotivskyi) |
| [`c4-architecture`](https://github.com/Kotivskyi/architecture-governance-skills) | C4 diagrams (Mermaid / Structurizr). | [Kotivskyi](https://github.com/Kotivskyi) |
| [`isaqb-architecture-governance`](https://github.com/Kotivskyi/architecture-governance-skills) | arc42 + ADRs. | [Kotivskyi](https://github.com/Kotivskyi) |
| [`secure-architecture-governance`](https://github.com/Kotivskyi/architecture-governance-skills) | STRIDE+CIA threat models, security ADRs. | [Kotivskyi](https://github.com/Kotivskyi) |
> 📖 Full install instructions for every skill live in [docs/SKILLS.md](docs/SKILLS.md).
## Quick start
## Install
```bash
# 1. Copy the workflows into your agent's skills directory.
# Each skill needs its own directory containing a SKILL.md -- a loose .md
# file sitting in a category folder is not indexed (neither Hermes nor
# Claude Code loads it), so the loop below builds the right shape.
SKILLS_DIR="$HOME/.hermes/skills/software-development"
# Claude Code: SKILLS_DIR="$HOME/.claude/skills"
git clone https://github.com/evandrodevbr/dev-workflows && cd dev-workflows
for f in workflows/*.md; do
name=$(basename "$f" .md)
mkdir -p "$SKILLS_DIR/$name"
cp "$f" "$SKILLS_DIR/$name/SKILL.md"
done
# 2. (Optional, for security review) the supporting skills are already
# vendored in skills/ -- no external clone needed, just copy:
cp -r skills/* "$HOME/.hermes/skills/"
# 3. Use it
# "review this PR for security" -> wf-security-review loads
# "build a new page" -> wf-frontend loads
python3 scripts/install.py claude # plugin: skills + agents + hooks
python3 scripts/install.py opencode # skills, agents and a plugin in ~/.config/opencode
python3 scripts/install.py hermes # skills only, in ~/.hermes/skills
```
Verify the install took: the workflows should show up in your agent's skill
list (`hermes skills list` for Hermes Agent).
Claude Code without cloning:
## The QA harness
```text
/plugin marketplace add evandrodevbr/dev-workflows
/plugin install dev-workflows@dev-workflows
```
The `qa/` directory keeps the workflows themselves honest:
Differences between agents:
- **opencode** cannot block the end of a turn, so a missing or failing gate is logged instead of blocking. Agents are converted to opencode subagents with the same permissions.
- **Hermes** gets the skills only; run the gate yourself.
- For opencode and Hermes the copies point at your checkout, so keep it in place (or re-run the install after moving it).
Turn the `Stop` hook off for one session with `DW_GATE=off`.
## External skills (optional)
Workflows use these when installed and fall back to their own rules when not. Install commands:
[`docs/SKILLS.md`](docs/SKILLS.md).
| Area | Skills |
|---|---|
| Frontend | `frontend-design` (Anthropic), `impeccable` (pbakaus), `vercel-react-best-practices`, `vercel-composition-patterns` (Vercel Labs), `animate` (emilkowalski), `web-design`, `avoid-ai-writing` |
| Backend | `secure-coding`, `bola-detector`, `auth-rbac-scaffold`, `injection-checker`, `openapi-hardener`, `security-test-generator` (apisec-inc) |
| Architecture | `system-design`, `c4-architecture`, `isaqb-architecture-governance`, `secure-architecture-governance` (Kotivskyi) |
| README | `readme-crafter`, `good-readme`, `curating-readme` |
## Quality of this repository
```bash
cd qa
python3 wf_quality_harness.py # score each workflow on 30+ criteria
python3 test_quality.py # gates: no regression, valid skills, phases verified
python3 test_tests.py # meta-tests: corrupt the harness, prove it detects the corruption
python3 qa/lint_skills.py # frontmatter, size limits, references, links, machine-specific paths
python3 qa/test_gate.py # quality gate and hooks on a throwaway repo
python3 qa/check_upstream.py # vendored skills behind their upstream
claude plugin validate --strict .claude-plugin/plugin.json
```
The meta-tests are the interesting part. They deliberately break the harness and the workflows, then confirm the score *drops*. If the score didn't move, the detector would be useless. These tests stay green as a contract.
CI runs the first two on every PR; the upstream check runs monthly.
## Quality tests and measured gains
**Not measured yet:** whether the kit improves the code an agent produces. Public studies
([SkillsBench](https://arxiv.org/abs/2602.12670), [SWE-Skills-Bench](https://arxiv.org/abs/2603.15401))
show skills can also make results worse, so the next step is an A/B benchmark (with vs. without the
kit) on [Harbor](https://github.com/harbor-framework/harbor) with hidden tests, mutation score and
security checks.
These are real numbers from the harness runs, stored in `qa/snapshots/baseline-final.json`. The baseline column is the state before the refinement work; the final column is the result after it.
### Workflow quality score (same harness, before → after)
| Workflow | Baseline | Final | Gain |
|---|---|---|---|
| wf-frontend | 201.0 | 385.5 | +92% |
| wf-backend | 215.0 | 424.0 | +97% |
| wf-architecture | 199.9 | 413.7 | +107% |
| wf-security-review | 332.0 | 390.0 | +17% |
| **Total** | **615.9** | **1223.2** | **+98.6%** |
That is almost exactly double the quality of the workflows, measured with the same ruler.
The ruler itself also grew stricter over the project (13 → 30+ criteria: verification per phase, real commands, anti-patterns, acceptance criteria, artifacts, flow). That is why the early runs show lower totals: `base-364` (13 criteria) → `baseline-v2` (536.7) → `baseline-v3` (675.0) → `baseline-v4` (884.6) → `baseline-final` (1613.2 across the four scored workflows). Only the final snapshot is committed, as `qa/snapshots/baseline-final.json` — that is the file the gates read and the file to update when scoring changes. The fair before/after comparison above uses the final ruler for both sides.
### Gates (regression checks): all green
| Gate | What it checks | Status |
|---|---|---|
| G1 | Score did not drop below baseline | ✅ |
| G4 | Every referenced skill exists | ✅ 4/4 workflows |
| G5 | Every phase has ≥3 verification verbs | ✅ 4/4 workflows |
| Meta-tests | Corrupt harness/workflows → score drops (proves detector works) | ✅ |
### What the refinement added
Every workflow gained the same batch of real improvements:
- **Verification steps** with actual commands (`curl`, `grep`, `python3`, `git`, `npm`, `pytest`) instead of prose.
- **Per-phase acceptance checklists** (markable `[ ]` gates).
- **Anti-pattern blocks** per phase (what not to do, and why).
- **Executable-command sections** and **worked examples per phase**.
- **Checkpoints** that pause for user approval between phases.
The security workflow gained a hard rule: *never claim "no CVE" without querying OSV.dev / GitHub Advisory*. A model's knowledge stops at its training cutoff; the live sources don't.
### Context on the "double" target
The harness has a physical ceiling (sum of all criterion caps). The numeric "2× of baseline" target would need a ruler with more resolution to be expressible, so inflating the score to hit it would be gaming the metric. What shipped is the honest version: real quality ~doubled on the same ruler (+98.6%), with meta-tests proving the ruler detects regressions.
## Repository layout
## Layout
```
dev-workflows/
├── README.md # this file
├── README.pt-BR.md # Portuguese version
├── workflows/ # the five SKILL.md workflows
│ ├── wf-frontend.md
│ ├── wf-backend.md
│ ├── wf-architecture.md
│ ├── wf-security-review.md
│ └── wf-readme.md
├── skills/ # the security pillar, vendored (see skills/NOTICE.md)
│ ├── safedeps/ # + plays/, templates/, schemas/, docs/ support files
│ ├── sca-audit/
│ ├── code-review-security/
│ ├── api-security-review/
│ ├── web-security-review/
│ ├── secrets-scan/
│ ├── cve-triage/
│ ├── patch-prioritization/
│ ├── dependency-scanning/
│ ├── hermaguard/
│ ├── design-taste-frontend/ # taste-skill (frontend)
│ └── redesign-existing-projects/ # taste-skill (frontend)
├── docs/
│ └── SKILLS.md # full skill inventory + install steps
├── CONTRIBUTING.md # what to improve, local checks, PR process
└── qa/ # quality harness, gates, meta-tests
├── wf_quality_harness.py
├── test_quality.py
├── test_tests.py
├── baseline.json
└── snapshots/
└── baseline-final.json
.claude-plugin/ plugin.json, marketplace.json
skills/ dev-router, lean-code, wf-* workflows, vendored skills (NOTICE.md)
agents/ explorer, planner, implementer, reviewer, security-auditor, ui-critic, verifier
hooks/hooks.json Stop, PreToolUse, PostToolUse
scripts/ quality-gate, install.py, hooks/*.py
adapters/opencode/ opencode plugin
qa/ lint, gate tests, upstream check
docs/SKILLS.md skill inventory and install commands
```
## License
[MIT](LICENSE) © 2026 [Evandro Fonseca Junior](https://github.com/evandrodevbr)
---
<div align="center">
⭐ If this saves you from one "works on my machine" merge, star it.
</div>
[MIT](LICENSE) © 2026 [Evandro Fonseca Junior](https://github.com/evandrodevbr). Vendored skills keep their own licenses ([`skills/NOTICE.md`](skills/NOTICE.md)).