ForgeLM/autoresearch.md

Autoresearch: latency-p95-chat-completions

Objective

Optimize end-to-end p95 latency for local OpenAI-compatible POST /v1/chat/completions requests against the Rust inference server, using the local Qwen GGUF model already present in models/. The benchmark targets a short coding-oriented request that is representative of editor integrations such as VSCode, OpenCode, Cline, and Roo.

Metrics

• Primary: p95_ms (milliseconds, lower is better)
• Secondary: p50_ms, mean_ms, completion_tps_mean, success_count

How to Run

./autoresearch.sh - builds the release server, starts it locally, runs a fixed HTTP benchmark, and outputs METRIC name=number lines.

Files in Scope

• crates/inference-server/src/compat.rs - request shaping, prompt preparation, auth/CORS/config, streaming mode decisions
• crates/inference-server/src/routes.rs - HTTP handlers, blocking/streaming response path, usage/reporting
• crates/inference-server/src/main.rs - runtime/server boot configuration
• crates/inference-core/src/engine.rs - engine load/warmup/session creation path
• crates/inference-core/src/session.rs - prompt prefill and token generation hot path
• crates/inference-core/src/profile.rs - runtime defaults that affect latency
• crates/inference-core/src/sampler.rs - per-request generation config
• crates/metrics/src/* - measurement plumbing if needed
• experiments/bench_chat_latency.py - benchmark harness for the autoresearch loop
• autoresearch.sh - loop benchmark entrypoint and fast preflight

Off Limits

• models/ model binaries
• vendor/ third-party sources and prebuilt llama binaries
• registry/ Python registry service
• ui/ frontend files
• broad API-surface feature additions unrelated to latency unless they directly reduce p95

Constraints

• Keep the benchmark reproducible and deterministic enough for comparison.
• No new external dependencies.
• Preserve OpenAI-compatible behavior already implemented.
• Tests must pass for keeps.
• Prefer changes that improve p95 without making code materially harder to maintain.
• Aggressive refactors are allowed if they stay within the in-scope server/runtime path.

What's Been Tried

• Baseline session initialized; no latency experiments logged yet.
• Existing work before this autoresearch session already improved token detokenization and broadened OpenAI-compatible request/response handling, but this loop focuses specifically on p95 latency.