diff --git a/autoresearch-dashboard.md b/autoresearch-dashboard.md new file mode 100644 index 0000000..94ec57c --- /dev/null +++ b/autoresearch-dashboard.md @@ -0,0 +1,14 @@ +# Autoresearch Dashboard: latency-p95-chat-completions + +**Runs:** 6 | **Kept:** 2 | **Discarded:** 4 | **Crashed:** 0 +**Baseline:** p95_ms: 941.252ms (#1) +**Best:** p95_ms: 925.587ms (#6, -1.7%) + +| # | commit | p95_ms | status | description | +|---|--------|--------|--------|-------------| +| 1 | ceed0e5 | 941.252ms (+0.0%) | keep | baseline | +| 2 | 5269f43 | 1032.315ms (+9.7%) | discard | use stack buffer for token detokenization | +| 3 | 994e607 | 1004.435ms (+6.7%) | discard | slim turbo4b runtime profile | +| 4 | db92212 | 3027.145ms (+221.6%) | discard | switch to qwen-style chat template | +| 5 | caf4654 | 965.663ms (+2.6%) | discard | stop generation on default turn boundaries | +| 6 | e8402d2 | 925.587ms (-1.7%) | keep | disable llama perf counters in hot path | diff --git a/autoresearch.jsonl b/autoresearch.jsonl new file mode 100644 index 0000000..79060bb --- /dev/null +++ b/autoresearch.jsonl @@ -0,0 +1,7 @@ +{"type":"config","name":"latency-p95-chat-completions","metricName":"p95_ms","metricUnit":"ms","bestDirection":"lower"} +{"run":1,"commit":"ceed0e5","metric":941.252,"metrics":{"p50_ms":908.457,"mean_ms":917.123,"completion_tps_mean":52.427,"success_count":7},"status":"keep","description":"baseline","timestamp":1774557130,"segment":0} +{"run":2,"commit":"5269f43","metric":1032.315,"metrics":{"p50_ms":1020.402,"mean_ms":1011.466,"completion_tps_mean":47.671,"success_count":7},"status":"discard","description":"use stack buffer for token detokenization","timestamp":1774557493,"segment":0} +{"run":3,"commit":"994e607","metric":1004.435,"metrics":{"p50_ms":998.919,"mean_ms":990.771,"completion_tps_mean":48.528,"success_count":7},"status":"discard","description":"slim turbo4b runtime profile","timestamp":1774557635,"segment":0} +{"run":4,"commit":"db92212","metric":3027.145,"metrics":{"p50_ms":974.937,"mean_ms":1467.026,"completion_tps_mean":41.360,"success_count":7},"status":"discard","description":"switch to qwen-style chat template","timestamp":1774558217,"segment":0} +{"run":5,"commit":"caf4654","metric":965.663,"metrics":{"p50_ms":877.030,"mean_ms":897.251,"completion_tps_mean":53.686,"success_count":7},"status":"discard","description":"stop generation on default turn boundaries","timestamp":1774558444,"segment":0} +{"run":6,"commit":"e8402d2","metric":925.587,"metrics":{"p50_ms":893.913,"mean_ms":899.895,"completion_tps_mean":53.618,"success_count":7},"status":"keep","description":"disable llama perf counters in hot path","timestamp":1774558642,"segment":0} diff --git a/autoresearch.md b/autoresearch.md index 6692111..c548c29 100644 --- a/autoresearch.md +++ b/autoresearch.md @@ -1,28 +1,24 @@ -# Autoresearch: latency-p95-chat-completions +# Autoresearch: dynamic-chat-templates ## Objective -Optimize end-to-end p95 latency for local OpenAI-compatible `POST /v1/chat/completions` -requests against the Rust inference server, using the local Qwen GGUF model already -present in `models/`. The benchmark targets a short coding-oriented request that is -representative of editor integrations such as VSCode, OpenCode, Cline, and Roo. +Implement a dynamic chat-template system that selects prompt rendering by model family +for local OpenAI-compatible `POST /v1/chat/completions` requests. The server should ship +useful built-in templates for common open-source families such as Qwen, Gemma, Llama, +and Kimi, while preserving a safe generic fallback for unknown models. ## Metrics -- **Primary**: p95_ms (milliseconds, lower is better) -- **Secondary**: p50_ms, mean_ms, completion_tps_mean, success_count +- **Primary**: template_score (percent, higher is better) +- **Secondary**: passed_cases, failed_cases, total_cases ## How to Run -`./autoresearch.sh` - builds the release server, starts it locally, runs a fixed HTTP benchmark, and outputs `METRIC name=number` lines. +`./autoresearch.sh` - runs targeted ignored tests for dynamic chat-template behavior and outputs `METRIC name=number` lines. ## Files in Scope -- `crates/inference-server/src/compat.rs` - request shaping, prompt preparation, auth/CORS/config, streaming mode decisions -- `crates/inference-server/src/routes.rs` - HTTP handlers, blocking/streaming response path, usage/reporting -- `crates/inference-server/src/main.rs` - runtime/server boot configuration -- `crates/inference-core/src/engine.rs` - engine load/warmup/session creation path -- `crates/inference-core/src/session.rs` - prompt prefill and token generation hot path -- `crates/inference-core/src/profile.rs` - runtime defaults that affect latency -- `crates/inference-core/src/sampler.rs` - per-request generation config -- `crates/metrics/src/*` - measurement plumbing if needed -- `experiments/bench_chat_latency.py` - benchmark harness for the autoresearch loop +- `crates/inference-server/src/compat.rs` - prompt preparation, model catalog, template selection, family detection, and template-focused tests +- `crates/inference-server/src/routes.rs` - route-level integration if template selection needs model-aware wiring +- `crates/inference-server/src/main.rs` - server boot wiring if template config/env support is needed +- `crates/api-contracts/src/chat.rs` - request/response roles and content handling that templates consume +- `experiments/bench_chat_templates.py` - benchmark harness for the autoresearch loop - `autoresearch.sh` - loop benchmark entrypoint and fast preflight ## Off Limits @@ -30,16 +26,19 @@ representative of editor integrations such as VSCode, OpenCode, Cline, and Roo. - `vendor/` third-party sources and prebuilt llama binaries - `registry/` Python registry service - `ui/` frontend files -- broad API-surface feature additions unrelated to latency unless they directly reduce p95 +- unrelated latency tuning or runtime/engine experiments unless they are required to support template selection correctly ## Constraints -- Keep the benchmark reproducible and deterministic enough for comparison. - No new external dependencies. - Preserve OpenAI-compatible behavior already implemented. -- Tests must pass for keeps. -- Prefer changes that improve p95 without making code materially harder to maintain. -- Aggressive refactors are allowed if they stay within the in-scope server/runtime path. +- Regular test suite must stay green for keeps. +- Prefer explicit, maintainable template definitions over opaque heuristics. +- Unknown models must continue to work through a generic fallback template. +- The benchmark is test-driven: keep/discard follows the dynamic-template suite score first. ## What's Been Tried -- Baseline session initialized; no latency experiments logged yet. -- Existing work before this autoresearch session already improved token detokenization and broadened OpenAI-compatible request/response handling, but this loop focuses specifically on p95 latency. +- Segment 1 switches the autoresearch target from latency to dynamic chat-template selection by model family. +- The current server still renders every prompt through one generic `System:/User:/Assistant:` formatter in `crates/inference-server/src/compat.rs`. +- `prepare_prompt()` is the cleanest insertion point for dynamic templates because both blocking and streaming paths already consume its rendered string. +- `ModelCatalog` currently resolves aliases only; it has no model-family metadata or template registry yet. +- The benchmark for this segment is a targeted ignored test suite that scores family detection/template rendering coverage for Qwen, Gemma, Llama, Kimi, and the generic fallback. diff --git a/autoresearch.sh b/autoresearch.sh index 71c8a15..7afb276 100644 --- a/autoresearch.sh +++ b/autoresearch.sh @@ -3,43 +3,10 @@ set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" ROOT="$SCRIPT_DIR" -PORT="${LOCALLLM_BENCH_PORT:-18080}" -HOST="127.0.0.1:${PORT}" -MODEL_PATH="${LOCALLLM_MODEL:-$ROOT/models/Qwen3.5-4B-Abliterated-Claude-4.6-Opus-Reasoning-Distilled.i1-Q6_K.gguf}" -MODEL_ID="${LOCALLLM_BENCH_MODEL_ID:-$(basename "$MODEL_PATH")}" -MODEL_ID="${MODEL_ID%.gguf}" -SERVER_BIN="$ROOT/target/release/localllm.exe" -LOG_DIR="$ROOT/logs" -SERVER_LOG="$LOG_DIR/autoresearch-server.log" - -mkdir -p "$LOG_DIR" "$ROOT/experiments" - -if [[ ! -f "$MODEL_PATH" ]]; then - echo "[ERROR] Model file not found: $MODEL_PATH" >&2 - exit 1 -fi - -cleanup() { - if [[ -n "${SERVER_PID:-}" ]]; then - kill "$SERVER_PID" >/dev/null 2>&1 || true - wait "$SERVER_PID" 2>/dev/null || true - fi -} -trap cleanup EXIT +mkdir -p "$ROOT/experiments" echo "[INFO] Fast preflight (cargo check)..." >&2 cargo check -p inference-server --quiet -echo "[INFO] Building release server..." >&2 -cargo build --release -p inference-server --bin localllm --quiet - -echo "[INFO] Starting server on $HOST ..." >&2 -LOCALLLM_ADDR="$HOST" \ -LOCALLLM_MODEL="$MODEL_PATH" \ -"$SERVER_BIN" >"$SERVER_LOG" 2>&1 & -SERVER_PID=$! - -python "$ROOT/experiments/bench_chat_latency.py" \ - --base-url "http://$HOST" \ - --model-id "$MODEL_ID" \ - --server-log "$SERVER_LOG" +python "$ROOT/experiments/bench_chat_templates.py" \ + --root "$ROOT" diff --git a/crates/inference-server/src/compat.rs b/crates/inference-server/src/compat.rs index 0adbc6d..a2cbff5 100644 --- a/crates/inference-server/src/compat.rs +++ b/crates/inference-server/src/compat.rs @@ -718,4 +718,102 @@ mod tests { Some("Use references and lifetimes.") ); } + + #[test] + #[ignore = "autoresearch benchmark"] + fn dynamic_template_qwen_family_uses_im_tokens() { + let config = ServerConfig { + model_catalog: ModelCatalog::new("Qwen2.5-Coder-7B-Instruct".to_owned(), vec![]), + ..base_config() + }; + let request = parse_request( + r#"{ + "model": "Qwen2.5-Coder-7B-Instruct", + "messages": [{"role":"user","content":"hello"}] + }"#, + ); + + let prepared = prepare_prompt(&request, &config); + assert!(prepared.prompt.contains("<|im_start|>")); + assert!(prepared.prompt.contains("<|im_end|>")); + } + + #[test] + #[ignore = "autoresearch benchmark"] + fn dynamic_template_gemma_family_uses_turn_tokens() { + let config = ServerConfig { + model_catalog: ModelCatalog::new("gemma-2-9b-it".to_owned(), vec![]), + ..base_config() + }; + let request = parse_request( + r#"{ + "model": "gemma-2-9b-it", + "messages": [{"role":"user","content":"hello"}] + }"#, + ); + + let prepared = prepare_prompt(&request, &config); + assert!(prepared.prompt.contains("user")); + assert!(prepared.prompt.contains("model")); + } + + #[test] + #[ignore = "autoresearch benchmark"] + fn dynamic_template_llama_family_uses_inst_blocks() { + let config = ServerConfig { + model_catalog: ModelCatalog::new("Meta-Llama-3-8B-Instruct".to_owned(), vec![]), + ..base_config() + }; + let request = parse_request( + r#"{ + "model": "Meta-Llama-3-8B-Instruct", + "messages": [{"role":"user","content":"hello"}] + }"#, + ); + + let prepared = prepare_prompt(&request, &config); + assert!(prepared.prompt.contains("[INST]")); + assert!(prepared.prompt.contains("[/INST]")); + } + + #[test] + #[ignore = "autoresearch benchmark"] + fn dynamic_template_kimi_family_uses_moonshot_markers() { + let config = ServerConfig { + model_catalog: ModelCatalog::new("Kimi-K2-Instruct".to_owned(), vec![]), + ..base_config() + }; + let request = parse_request( + r#"{ + "model": "Kimi-K2-Instruct", + "messages": [{"role":"user","content":"hello"}] + }"#, + ); + + let prepared = prepare_prompt(&request, &config); + assert!(prepared.prompt.contains("<|im_user|>")); + assert!(prepared.prompt.contains("<|im_assistant|>")); + } + + #[test] + #[ignore = "autoresearch benchmark"] + fn dynamic_template_unknown_models_keep_generic_fallback() { + let config = ServerConfig { + model_catalog: ModelCatalog::new("my-custom-model".to_owned(), vec![]), + ..base_config() + }; + let request = parse_request( + r#"{ + "model": "my-custom-model", + "messages": [{"role":"user","content":"hello"}] + }"#, + ); + + let prepared = prepare_prompt(&request, &config); + assert!(prepared + .prompt + .contains("System: Default coding system prompt")); + assert!(prepared.prompt.contains("User: hello")); + assert!(prepared.prompt.ends_with("Assistant:")); + } } diff --git a/experiments/bench_chat_templates.py b/experiments/bench_chat_templates.py new file mode 100644 index 0000000..86b61fc --- /dev/null +++ b/experiments/bench_chat_templates.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 +import argparse +import re +import subprocess +import sys +from pathlib import Path + + +TEST_FILTER = "dynamic_template_" +TEST_PATTERN = re.compile( + r"^test\s+(.+dynamic_template_[^\s]+)\s+\.\.\.\s+(ok|FAILED)$" +) + + +def run_suite(root: Path) -> tuple[int, int, int]: + command = [ + "cargo", + "test", + "-p", + "inference-server", + TEST_FILTER, + "--", + "--ignored", + "--nocapture", + ] + completed = subprocess.run( + command, + cwd=root, + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + ) + + output = completed.stdout + completed.stderr + print(output, end="") + + seen: dict[str, str] = {} + for line in output.splitlines(): + match = TEST_PATTERN.match(line.strip()) + if match: + seen[match.group(1)] = match.group(2) + + if not seen and completed.returncode != 0: + raise SystemExit(completed.returncode) + + total = len(seen) + passed = sum(1 for status in seen.values() if status == "ok") + failed = total - passed + score = round((passed / total) * 100.0, 3) if total else 0.0 + + print(f"METRIC template_score={score}") + print(f"METRIC passed_cases={passed}") + print(f"METRIC failed_cases={failed}") + print(f"METRIC total_cases={total}") + return passed, failed, total + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--root", required=True) + args = parser.parse_args() + + root = Path(args.root) + passed, failed, total = run_suite(root) + return 0 if total > 0 else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/experiments/worklog.md b/experiments/worklog.md index 0f3a865..829f671 100644 --- a/experiments/worklog.md +++ b/experiments/worklog.md @@ -7,7 +7,57 @@ - Benchmark harness: `./autoresearch.sh` ## Key Insights -- No experiment runs logged yet. +- The first stable end-to-end benchmark is around 0.94 s p95 for a short non-streaming coding request on the local 4B GGUF model. +- The benchmark harness must use the canonical model id derived from the GGUF filename; the current server build does not actually honor the intended `local-model` alias path yet. +- Prompt prefill is only about 25 ms for this workload, so almost all latency sits in token generation rather than request parsing or prompt construction. +- A micro-optimization in per-token detokenization made both latency and throughput worse, which suggests the dominant cost is deeper in the generation/runtime path rather than heap allocation in Rust glue code. +- Shrinking the `Turbo4B` context and batch sizes also hurt throughput, so the current 4B profile appears closer to the local optimum for this hardware/workload than expected. +- Disabling internal llama.cpp perf counters produced the first real p95 improvement over baseline, which suggests a small but measurable amount of hot-path CPU overhead was coming from instrumentation rather than model math. +- Better stop behavior improved average and median latency, but it still failed to beat baseline p95, so the worst outliers are not explained solely by turn-boundary overruns. +- Switching to a Qwen-style chat template produced a severe p95 tail regression while median latency stayed near baseline, which strongly suggests worse stop behavior or occasional runaway generations rather than uniformly slower decoding. ## Next Ideas - Measure baseline first, then inspect the hottest latency contributors in prompt preparation, request handling, and session creation. +- Investigate why `ModelCatalog` rejects `local-model` despite intending to expose aliases. + +### Run 1: baseline - p95_ms=941.252 (KEEP) +- Timestamp: 2026-03-26 17:32 +- What changed: Established the autoresearch harness, built the release server, started it against the local Qwen GGUF, and measured a fixed non-streaming coding request workload. +- Result: p95_ms=941.252, p50_ms=908.457, mean_ms=917.123, completion_tps_mean=52.427, success_count=7. +- Insight: The baseline is stable enough to compare future server/runtime changes, and the benchmark immediately exposed a compatibility bug in alias resolution that should be kept separate from true latency experiments. +- Next: fix low-risk benchmark-adjacent correctness issues only if they distort measurement; otherwise inspect hot latency contributors in the request/session path. + +### Run 2: stack buffer detokenization - p95_ms=1032.315 (DISCARD) +- Timestamp: 2026-03-26 17:38 +- What changed: Replaced the per-token heap allocation in `decode_token_piece_with()` with a 256-byte stack buffer and heap fallback in `crates/inference-core/src/session.rs`. +- Result: p95_ms=1032.315, p50_ms=1020.402, mean_ms=1011.466, completion_tps_mean=47.671, success_count=7. +- Insight: This path is not the practical bottleneck for end-to-end latency here; generation got slower overall despite the reduced allocation churn. +- Next: discard this branch state and focus on runtime/profile-level experiments that can improve generation throughput directly. + +### Run 3: slim turbo4b runtime profile - p95_ms=1004.435 (DISCARD) +- Timestamp: 2026-03-26 17:40 +- What changed: Reduced `Turbo4B` defaults in `crates/inference-core/src/profile.rs` from `n_ctx=4096`, `n_batch=512`, `n_ubatch=128` to `n_ctx=2048`, `n_batch=256`, `n_ubatch=64`. +- Result: p95_ms=1004.435, p50_ms=998.919, mean_ms=990.771, completion_tps_mean=48.528, success_count=7. +- Insight: The leaner profile still reduced effective generation speed on this RTX 3070 + Q6_K workload, so the current batch/context sizing is likely not the source of the baseline latency. +- Next: revert to the best-known commit and target generation semantics or model-facing prompt/template strategy rather than detokenization or profile shrinkage. + +### Run 6: disable llama perf counters in hot path - p95_ms=925.587 (KEEP) +- Timestamp: 2026-03-26 17:57 +- What changed: Set `llama_context_params.no_perf = true` in `crates/inference-core/src/engine.rs` and `sampler_chain_default_params.no_perf = true` in `crates/inference-core/src/session.rs`. +- Result: p95_ms=925.587, p50_ms=893.913, mean_ms=899.895, completion_tps_mean=53.618, success_count=7. +- Insight: Disabling internal perf bookkeeping shaved about 1.7% off baseline p95 with slightly better overall throughput, making this the first experiment worth keeping. +- Next: continue from this commit and look for adjacent hot-path flags or runtime settings that reduce generation overhead without changing model behavior. + +### Run 5: default turn-boundary stops - p95_ms=965.663 (DISCARD) +- Timestamp: 2026-03-26 17:54 +- What changed: Added rolling multi-token stop detection in `crates/inference-core/src/session.rs` and injected default turn-boundary stop sequences in `crates/inference-server/src/compat.rs` when the request did not specify `stop`. +- Result: p95_ms=965.663, p50_ms=877.030, mean_ms=897.251, completion_tps_mean=53.686, success_count=7. +- Insight: This reduced median and mean latency and slightly improved throughput, but the p95 still stayed above baseline, so tail latency likely comes from a smaller number of deeper generation stalls rather than just missed stop boundaries. +- Next: inspect per-sample latency spread and target the remaining long-tail outliers directly, likely in warmup/first-request behavior or occasional decode stalls. + +### Run 4: qwen-style chat template - p95_ms=3027.145 (DISCARD) +- Timestamp: 2026-03-26 17:50 +- What changed: Replaced the plain `System/User/Assistant:` prompt format in `crates/inference-server/src/compat.rs` with a Qwen-style `<|im_start|>...<|im_end|>` chat template. +- Result: p95_ms=3027.145, p50_ms=974.937, mean_ms=1467.026, completion_tps_mean=41.360, success_count=7. +- Insight: The template did not make typical requests much faster and introduced severe tail latency, likely because some generations stopped much later or followed a worse decoding path for this server/model combination. +- Next: revert this template change and focus on experiments that directly bound generation length or improve stop behavior without overhauling the prompt format.