[autoresearch] baseline: system prompt quality tests (5/5 passing)

.agents/rules/monitoring-and-alerting-principles.md

@@ -0,0 +1,86 @@
+---
+trigger: model_decision
+description: When implementing health checks, metrics instrumentation, error tracking integration, or production observability code
+---
+
+## Monitoring and Alerting Principles
+
+> **Scope:** This rule covers what the agent **implements in code**. Organizational 
+> concerns (SLO/SLI definitions, on-call rotation, alert escalation) are out of scope —
+> those are team/org decisions, not code generation concerns.
+
+### Health Checks
+
+**Every service must expose health check endpoints:**
+
+- **`/health` (Liveness)** — "Is the process alive?"
+  - Returns 200 if the process is running
+  - No dependency checks (database, cache, etc.)
+  - Used by orchestrators to decide whether to restart
+
+- **`/ready` (Readiness)** — "Can the service accept traffic?"
+  - Checks all critical dependencies (database, cache, message queue)
+  - Returns 503 if any dependency is unavailable
+  - Used by load balancers to route traffic
+
+**Rules:**
+- Health checks must be fast (< 1 second)
+- Health checks must not have side effects
+- Separate liveness from readiness — they serve different purposes
+
+### Metrics Instrumentation
+
+**Instrument code using the RED method for services:**
+- **Rate** — requests per second
+- **Errors** — error count/rate
+- **Duration** — request latency (use histograms, not averages)
+
+**Instrument code using the USE method for resources:**
+- **Utilization** — how much of the resource is used
+- **Saturation** — how much work is queued
+- **Errors** — error count for the resource
+
+**Rules:**
+- Use counters for things that only go up (requests, errors)
+- Use gauges for things that go up and down (connections, queue depth)
+- Use histograms for distributions (latency, response size)
+- Label metrics consistently (service, method, status_code)
+- Don't create high-cardinality labels (no user IDs as labels)
+
+### Error Tracking Integration
+
+- **Capture unhandled exceptions** with full stack traces
+- **Include context** — user ID, request ID, correlation ID
+- **Group errors** by root cause, not by instance
+- **Set severity levels** based on user impact
+
+### Graceful Degradation
+
+- **Circuit breakers** for external dependencies — stop calling failing services
+- **Fallbacks** for non-critical features — serve cached data, show reduced UI
+- **Timeouts** on all external calls — never wait indefinitely
+- **Retry with backoff** for transient failures — exponential backoff with jitter
+
+### Implementation Notes
+
+This rule is tool-agnostic. Whether the project uses Datadog, LGTM stack, Sentry,
+New Relic, or CloudWatch — the code patterns (health checks, metrics, error tracking)
+are the same. The specific client library belongs in project-level configuration.
+
+### Monitoring Checklist
+
+- [ ] Health check endpoints implemented (/health and /ready)?
+- [ ] Liveness probe has no dependency checks?
+- [ ] Readiness probe checks all critical dependencies?
+- [ ] Key operations instrumented with RED/USE metrics?
+- [ ] No high-cardinality metric labels?
+- [ ] Unhandled exceptions captured with context?
+- [ ] Circuit breakers on external dependencies?
+- [ ] Timeouts on all external calls?
+
+### Related Principles
+- Logging and Observability Mandate @logging-and-observability-mandate.md
+- Logging and Observability Principles @logging-and-observability-principles.md
+- Error Handling Principles @error-handling-principles.md
+- Resource and Memory Management Principles @resources-and-memory-management-principles.md
+- Concurrency and Threading Principles @concurrency-and-threading-principles.md