# Autoresearch: latency-p95-chat-completions

## Objective
Optimize end-to-end p95 latency for local OpenAI-compatible `POST /v1/chat/completions`
requests against the Rust inference server, using the local Qwen GGUF model already
present in `models/`. The benchmark targets a short coding-oriented request that is
representative of editor integrations such as VSCode, OpenCode, Cline, and Roo.

## Metrics
- **Primary**: p95_ms (milliseconds, lower is better)
- **Secondary**: p50_ms, mean_ms, completion_tps_mean, success_count

## How to Run
`./autoresearch.sh` - builds the release server, starts it locally, runs a fixed HTTP benchmark, and outputs `METRIC name=number` lines.

## Files in Scope
- `crates/inference-server/src/compat.rs` - request shaping, prompt preparation, auth/CORS/config, streaming mode decisions
- `crates/inference-server/src/routes.rs` - HTTP handlers, blocking/streaming response path, usage/reporting
- `crates/inference-server/src/main.rs` - runtime/server boot configuration
- `crates/inference-core/src/engine.rs` - engine load/warmup/session creation path
- `crates/inference-core/src/session.rs` - prompt prefill and token generation hot path
- `crates/inference-core/src/profile.rs` - runtime defaults that affect latency
- `crates/inference-core/src/sampler.rs` - per-request generation config
- `crates/metrics/src/*` - measurement plumbing if needed
- `experiments/bench_chat_latency.py` - benchmark harness for the autoresearch loop
- `autoresearch.sh` - loop benchmark entrypoint and fast preflight

## Off Limits
- `models/` model binaries
- `vendor/` third-party sources and prebuilt llama binaries
- `registry/` Python registry service
- `ui/` frontend files
- broad API-surface feature additions unrelated to latency unless they directly reduce p95

## Constraints
- Keep the benchmark reproducible and deterministic enough for comparison.
- No new external dependencies.
- Preserve OpenAI-compatible behavior already implemented.
- Tests must pass for keeps.
- Prefer changes that improve p95 without making code materially harder to maintain.
- Aggressive refactors are allowed if they stay within the in-scope server/runtime path.

## What's Been Tried
- Baseline session initialized; no latency experiments logged yet.
- Existing work before this autoresearch session already improved token detokenization and broadened OpenAI-compatible request/response handling, but this loop focuses specifically on p95 latency.