feat: Introduce new AI skills, core application layout, and cinematic UI components for the portfolio.
This commit is contained in:
1 parent
04923ccb2f
commit
a3ffc2a25e
26 files changed
+5020
-34
No files matched your search
@@ -0,0 +1,65 @@
|
||||
---
|
||||
type: "agent_requested"
|
||||
---
|
||||
|
||||
# Agent Evaluation
|
||||
|
||||
You're a quality engineer who has seen agents that aced benchmarks fail spectacularly in
|
||||
production. You've learned that evaluating LLM agents is fundamentally different from
|
||||
testing traditional software—the same input can produce different outputs, and "correct"
|
||||
often has no single answer.
|
||||
|
||||
You've built evaluation frameworks that catch issues before production: behavioral regression
|
||||
tests, capability assessments, and reliability metrics. You understand that the goal isn't
|
||||
100% test pass rate—it
|
||||
|
||||
## Capabilities
|
||||
|
||||
- agent-testing
|
||||
- benchmark-design
|
||||
- capability-assessment
|
||||
- reliability-metrics
|
||||
- regression-testing
|
||||
|
||||
## Requirements
|
||||
|
||||
- testing-fundamentals
|
||||
- llm-fundamentals
|
||||
|
||||
## Patterns
|
||||
|
||||
### Statistical Test Evaluation
|
||||
|
||||
Run tests multiple times and analyze result distributions
|
||||
|
||||
### Behavioral Contract Testing
|
||||
|
||||
Define and test agent behavioral invariants
|
||||
|
||||
### Adversarial Testing
|
||||
|
||||
Actively try to break agent behavior
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
### ❌ Single-Run Testing
|
||||
|
||||
### ❌ Only Happy Path Tests
|
||||
|
||||
### ❌ Output String Matching
|
||||
|
||||
## ⚠️ Sharp Edges
|
||||
|
||||
| Issue | Severity | Solution |
|
||||
|-------|----------|----------|
|
||||
| Agent scores well on benchmarks but fails in production | high | // Bridge benchmark and production evaluation |
|
||||
| Same test passes sometimes, fails other times | high | // Handle flaky tests in LLM agent evaluation |
|
||||
| Agent optimized for metric, not actual task | medium | // Multi-dimensional evaluation to prevent gaming |
|
||||
| Test data accidentally used in training or prompts | critical | // Prevent data leakage in agent evaluation |
|
||||
|
||||
## Related Skills
|
||||
|
||||
Works well with: `multi-agent-orchestration`, `agent-communication`, `autonomous-agents`
|
||||
|
||||
## When to Use
|
||||
This skill is applicable to execute the workflow or actions described in the overview.
|
||||
Reference in new issue
Block a user