87 lines
3.5 KiB
Markdown
87 lines
3.5 KiB
Markdown
---
|
|
trigger: model_decision
|
|
description: When implementing health checks, metrics instrumentation, error tracking integration, or production observability code
|
|
---
|
|
|
|
## Monitoring and Alerting Principles
|
|
|
|
> **Scope:** This rule covers what the agent **implements in code**. Organizational
|
|
> concerns (SLO/SLI definitions, on-call rotation, alert escalation) are out of scope —
|
|
> those are team/org decisions, not code generation concerns.
|
|
|
|
### Health Checks
|
|
|
|
**Every service must expose health check endpoints:**
|
|
|
|
- **`/health` (Liveness)** — "Is the process alive?"
|
|
- Returns 200 if the process is running
|
|
- No dependency checks (database, cache, etc.)
|
|
- Used by orchestrators to decide whether to restart
|
|
|
|
- **`/ready` (Readiness)** — "Can the service accept traffic?"
|
|
- Checks all critical dependencies (database, cache, message queue)
|
|
- Returns 503 if any dependency is unavailable
|
|
- Used by load balancers to route traffic
|
|
|
|
**Rules:**
|
|
- Health checks must be fast (< 1 second)
|
|
- Health checks must not have side effects
|
|
- Separate liveness from readiness — they serve different purposes
|
|
|
|
### Metrics Instrumentation
|
|
|
|
**Instrument code using the RED method for services:**
|
|
- **Rate** — requests per second
|
|
- **Errors** — error count/rate
|
|
- **Duration** — request latency (use histograms, not averages)
|
|
|
|
**Instrument code using the USE method for resources:**
|
|
- **Utilization** — how much of the resource is used
|
|
- **Saturation** — how much work is queued
|
|
- **Errors** — error count for the resource
|
|
|
|
**Rules:**
|
|
- Use counters for things that only go up (requests, errors)
|
|
- Use gauges for things that go up and down (connections, queue depth)
|
|
- Use histograms for distributions (latency, response size)
|
|
- Label metrics consistently (service, method, status_code)
|
|
- Don't create high-cardinality labels (no user IDs as labels)
|
|
|
|
### Error Tracking Integration
|
|
|
|
- **Capture unhandled exceptions** with full stack traces
|
|
- **Include context** — user ID, request ID, correlation ID
|
|
- **Group errors** by root cause, not by instance
|
|
- **Set severity levels** based on user impact
|
|
|
|
### Graceful Degradation
|
|
|
|
- **Circuit breakers** for external dependencies — stop calling failing services
|
|
- **Fallbacks** for non-critical features — serve cached data, show reduced UI
|
|
- **Timeouts** on all external calls — never wait indefinitely
|
|
- **Retry with backoff** for transient failures — exponential backoff with jitter
|
|
|
|
### Implementation Notes
|
|
|
|
This rule is tool-agnostic. Whether the project uses Datadog, LGTM stack, Sentry,
|
|
New Relic, or CloudWatch — the code patterns (health checks, metrics, error tracking)
|
|
are the same. The specific client library belongs in project-level configuration.
|
|
|
|
### Monitoring Checklist
|
|
|
|
- [ ] Health check endpoints implemented (/health and /ready)?
|
|
- [ ] Liveness probe has no dependency checks?
|
|
- [ ] Readiness probe checks all critical dependencies?
|
|
- [ ] Key operations instrumented with RED/USE metrics?
|
|
- [ ] No high-cardinality metric labels?
|
|
- [ ] Unhandled exceptions captured with context?
|
|
- [ ] Circuit breakers on external dependencies?
|
|
- [ ] Timeouts on all external calls?
|
|
|
|
### Related Principles
|
|
- Logging and Observability Mandate @logging-and-observability-mandate.md
|
|
- Logging and Observability Principles @logging-and-observability-principles.md
|
|
- Error Handling Principles @error-handling-principles.md
|
|
- Resource and Memory Management Principles @resources-and-memory-management-principles.md
|
|
- Concurrency and Threading Principles @concurrency-and-threading-principles.md
|