46 lines
2.4 KiB
Markdown
46 lines
2.4 KiB
Markdown
# Autoresearch: latency-p95-chat-completions
|
|
|
|
## Objective
|
|
Optimize end-to-end p95 latency for local OpenAI-compatible `POST /v1/chat/completions`
|
|
requests against the Rust inference server, using the local Qwen GGUF model already
|
|
present in `models/`. The benchmark targets a short coding-oriented request that is
|
|
representative of editor integrations such as VSCode, OpenCode, Cline, and Roo.
|
|
|
|
## Metrics
|
|
- **Primary**: p95_ms (milliseconds, lower is better)
|
|
- **Secondary**: p50_ms, mean_ms, completion_tps_mean, success_count
|
|
|
|
## How to Run
|
|
`./autoresearch.sh` - builds the release server, starts it locally, runs a fixed HTTP benchmark, and outputs `METRIC name=number` lines.
|
|
|
|
## Files in Scope
|
|
- `crates/inference-server/src/compat.rs` - request shaping, prompt preparation, auth/CORS/config, streaming mode decisions
|
|
- `crates/inference-server/src/routes.rs` - HTTP handlers, blocking/streaming response path, usage/reporting
|
|
- `crates/inference-server/src/main.rs` - runtime/server boot configuration
|
|
- `crates/inference-core/src/engine.rs` - engine load/warmup/session creation path
|
|
- `crates/inference-core/src/session.rs` - prompt prefill and token generation hot path
|
|
- `crates/inference-core/src/profile.rs` - runtime defaults that affect latency
|
|
- `crates/inference-core/src/sampler.rs` - per-request generation config
|
|
- `crates/metrics/src/*` - measurement plumbing if needed
|
|
- `experiments/bench_chat_latency.py` - benchmark harness for the autoresearch loop
|
|
- `autoresearch.sh` - loop benchmark entrypoint and fast preflight
|
|
|
|
## Off Limits
|
|
- `models/` model binaries
|
|
- `vendor/` third-party sources and prebuilt llama binaries
|
|
- `registry/` Python registry service
|
|
- `ui/` frontend files
|
|
- broad API-surface feature additions unrelated to latency unless they directly reduce p95
|
|
|
|
## Constraints
|
|
- Keep the benchmark reproducible and deterministic enough for comparison.
|
|
- No new external dependencies.
|
|
- Preserve OpenAI-compatible behavior already implemented.
|
|
- Tests must pass for keeps.
|
|
- Prefer changes that improve p95 without making code materially harder to maintain.
|
|
- Aggressive refactors are allowed if they stay within the in-scope server/runtime path.
|
|
|
|
## What's Been Tried
|
|
- Baseline session initialized; no latency experiments logged yet.
|
|
- Existing work before this autoresearch session already improved token detokenization and broadened OpenAI-compatible request/response handling, but this loop focuses specifically on p95 latency.
|