chore: initialize template autoresearch segment

autoresearch-dashboard.md

@@ -0,0 +1,14 @@
+# Autoresearch Dashboard: latency-p95-chat-completions
+
+**Runs:** 6 | **Kept:** 2 | **Discarded:** 4 | **Crashed:** 0
+**Baseline:** p95_ms: 941.252ms (#1)
+**Best:** p95_ms: 925.587ms (#6, -1.7%)
+
+| # | commit | p95_ms | status | description |
+|---|--------|--------|--------|-------------|
+| 1 | ceed0e5 | 941.252ms (+0.0%) | keep | baseline |
+| 2 | 5269f43 | 1032.315ms (+9.7%) | discard | use stack buffer for token detokenization |
+| 3 | 994e607 | 1004.435ms (+6.7%) | discard | slim turbo4b runtime profile |
+| 4 | db92212 | 3027.145ms (+221.6%) | discard | switch to qwen-style chat template |
+| 5 | caf4654 | 965.663ms (+2.6%) | discard | stop generation on default turn boundaries |
+| 6 | e8402d2 | 925.587ms (-1.7%) | keep | disable llama perf counters in hot path |

autoresearch.jsonl

@@ -0,0 +1,7 @@
+{"type":"config","name":"latency-p95-chat-completions","metricName":"p95_ms","metricUnit":"ms","bestDirection":"lower"}
+{"run":1,"commit":"ceed0e5","metric":941.252,"metrics":{"p50_ms":908.457,"mean_ms":917.123,"completion_tps_mean":52.427,"success_count":7},"status":"keep","description":"baseline","timestamp":1774557130,"segment":0}
+{"run":2,"commit":"5269f43","metric":1032.315,"metrics":{"p50_ms":1020.402,"mean_ms":1011.466,"completion_tps_mean":47.671,"success_count":7},"status":"discard","description":"use stack buffer for token detokenization","timestamp":1774557493,"segment":0}
+{"run":3,"commit":"994e607","metric":1004.435,"metrics":{"p50_ms":998.919,"mean_ms":990.771,"completion_tps_mean":48.528,"success_count":7},"status":"discard","description":"slim turbo4b runtime profile","timestamp":1774557635,"segment":0}
+{"run":4,"commit":"db92212","metric":3027.145,"metrics":{"p50_ms":974.937,"mean_ms":1467.026,"completion_tps_mean":41.360,"success_count":7},"status":"discard","description":"switch to qwen-style chat template","timestamp":1774558217,"segment":0}
+{"run":5,"commit":"caf4654","metric":965.663,"metrics":{"p50_ms":877.030,"mean_ms":897.251,"completion_tps_mean":53.686,"success_count":7},"status":"discard","description":"stop generation on default turn boundaries","timestamp":1774558444,"segment":0}
+{"run":6,"commit":"e8402d2","metric":925.587,"metrics":{"p50_ms":893.913,"mean_ms":899.895,"completion_tps_mean":53.618,"success_count":7},"status":"keep","description":"disable llama perf counters in hot path","timestamp":1774558642,"segment":0}

autoresearch.md

@@ -1,28 +1,24 @@
-# Autoresearch: latency-p95-chat-completions
+# Autoresearch: dynamic-chat-templates
 
 ## Objective
-Optimize end-to-end p95 latency for local OpenAI-compatible `POST /v1/chat/completions`
+Implement a dynamic chat-template system that selects prompt rendering by model family
-requests against the Rust inference server, using the local Qwen GGUF model already
+for local OpenAI-compatible `POST /v1/chat/completions` requests. The server should ship
-present in `models/`. The benchmark targets a short coding-oriented request that is
+useful built-in templates for common open-source families such as Qwen, Gemma, Llama,
-representative of editor integrations such as VSCode, OpenCode, Cline, and Roo.
+and Kimi, while preserving a safe generic fallback for unknown models.
 
 ## Metrics
-- **Primary**: p95_ms (milliseconds, lower is better)
+- **Primary**: template_score (percent, higher is better)
-- **Secondary**: p50_ms, mean_ms, completion_tps_mean, success_count
+- **Secondary**: passed_cases, failed_cases, total_cases
 
 ## How to Run
-`./autoresearch.sh` - builds the release server, starts it locally, runs a fixed HTTP benchmark, and outputs `METRIC name=number` lines.
+`./autoresearch.sh` - runs targeted ignored tests for dynamic chat-template behavior and outputs `METRIC name=number` lines.
 
 ## Files in Scope
-- `crates/inference-server/src/compat.rs` - request shaping, prompt preparation, auth/CORS/config, streaming mode decisions
+- `crates/inference-server/src/compat.rs` - prompt preparation, model catalog, template selection, family detection, and template-focused tests
-- `crates/inference-server/src/routes.rs` - HTTP handlers, blocking/streaming response path, usage/reporting
+- `crates/inference-server/src/routes.rs` - route-level integration if template selection needs model-aware wiring
-- `crates/inference-server/src/main.rs` - runtime/server boot configuration
+- `crates/inference-server/src/main.rs` - server boot wiring if template config/env support is needed
-- `crates/inference-core/src/engine.rs` - engine load/warmup/session creation path
+- `crates/api-contracts/src/chat.rs` - request/response roles and content handling that templates consume
-- `crates/inference-core/src/session.rs` - prompt prefill and token generation hot path
+- `experiments/bench_chat_templates.py` - benchmark harness for the autoresearch loop
-- `crates/inference-core/src/profile.rs` - runtime defaults that affect latency
-- `crates/inference-core/src/sampler.rs` - per-request generation config
-- `crates/metrics/src/*` - measurement plumbing if needed
-- `experiments/bench_chat_latency.py` - benchmark harness for the autoresearch loop
 - `autoresearch.sh` - loop benchmark entrypoint and fast preflight
 
 ## Off Limits
@@ -30,16 +26,19 @@ representative of editor integrations such as VSCode, OpenCode, Cline, and Roo.
 - `vendor/` third-party sources and prebuilt llama binaries
 - `registry/` Python registry service
 - `ui/` frontend files
-- broad API-surface feature additions unrelated to latency unless they directly reduce p95
+- unrelated latency tuning or runtime/engine experiments unless they are required to support template selection correctly
 
 ## Constraints
-- Keep the benchmark reproducible and deterministic enough for comparison.
 - No new external dependencies.
 - Preserve OpenAI-compatible behavior already implemented.
-- Tests must pass for keeps.
+- Regular test suite must stay green for keeps.
-- Prefer changes that improve p95 without making code materially harder to maintain.
+- Prefer explicit, maintainable template definitions over opaque heuristics.
-- Aggressive refactors are allowed if they stay within the in-scope server/runtime path.
+- Unknown models must continue to work through a generic fallback template.
+- The benchmark is test-driven: keep/discard follows the dynamic-template suite score first.
 
 ## What's Been Tried
-- Baseline session initialized; no latency experiments logged yet.
+- Segment 1 switches the autoresearch target from latency to dynamic chat-template selection by model family.
-- Existing work before this autoresearch session already improved token detokenization and broadened OpenAI-compatible request/response handling, but this loop focuses specifically on p95 latency.
+- The current server still renders every prompt through one generic `System:/User:/Assistant:` formatter in `crates/inference-server/src/compat.rs`.
+- `prepare_prompt()` is the cleanest insertion point for dynamic templates because both blocking and streaming paths already consume its rendered string.
+- `ModelCatalog` currently resolves aliases only; it has no model-family metadata or template registry yet.
+- The benchmark for this segment is a targeted ignored test suite that scores family detection/template rendering coverage for Qwen, Gemma, Llama, Kimi, and the generic fallback.

autoresearch.sh

@@ -3,43 +3,10 @@ set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 ROOT="$SCRIPT_DIR"
-PORT="${LOCALLLM_BENCH_PORT:-18080}"
+mkdir -p "$ROOT/experiments"
-HOST="127.0.0.1:${PORT}"
-MODEL_PATH="${LOCALLLM_MODEL:-$ROOT/models/Qwen3.5-4B-Abliterated-Claude-4.6-Opus-Reasoning-Distilled.i1-Q6_K.gguf}"
-MODEL_ID="${LOCALLLM_BENCH_MODEL_ID:-$(basename "$MODEL_PATH")}" 
-MODEL_ID="${MODEL_ID%.gguf}"
-SERVER_BIN="$ROOT/target/release/localllm.exe"
-LOG_DIR="$ROOT/logs"
-SERVER_LOG="$LOG_DIR/autoresearch-server.log"
-
-mkdir -p "$LOG_DIR" "$ROOT/experiments"
-
-if [[ ! -f "$MODEL_PATH" ]]; then
-    echo "[ERROR] Model file not found: $MODEL_PATH" >&2
-    exit 1
-fi
-
-cleanup() {
-    if [[ -n "${SERVER_PID:-}" ]]; then
-        kill "$SERVER_PID" >/dev/null 2>&1 || true
-        wait "$SERVER_PID" 2>/dev/null || true
-    fi
-}
-trap cleanup EXIT
 
 echo "[INFO] Fast preflight (cargo check)..." >&2
 cargo check -p inference-server --quiet
 
-echo "[INFO] Building release server..." >&2
+python "$ROOT/experiments/bench_chat_templates.py" \
-cargo build --release -p inference-server --bin localllm --quiet
+    --root "$ROOT"
-
-echo "[INFO] Starting server on $HOST ..." >&2
-LOCALLLM_ADDR="$HOST" \
-LOCALLLM_MODEL="$MODEL_PATH" \
-"$SERVER_BIN" >"$SERVER_LOG" 2>&1 &
-SERVER_PID=$!
-
-python "$ROOT/experiments/bench_chat_latency.py" \
-    --base-url "http://$HOST" \
-    --model-id "$MODEL_ID" \
-    --server-log "$SERVER_LOG"

crates/inference-server/src/compat.rs

@@ -718,4 +718,102 @@ mod tests {
             Some("Use references and lifetimes.")
         );
     }
+
+    #[test]
+    #[ignore = "autoresearch benchmark"]
+    fn dynamic_template_qwen_family_uses_im_tokens() {
+        let config = ServerConfig {
+            model_catalog: ModelCatalog::new("Qwen2.5-Coder-7B-Instruct".to_owned(), vec![]),
+            ..base_config()
+        };
+        let request = parse_request(
+            r#"{
+                "model": "Qwen2.5-Coder-7B-Instruct",
+                "messages": [{"role":"user","content":"hello"}]
+            }"#,
+        );
+
+        let prepared = prepare_prompt(&request, &config);
+        assert!(prepared.prompt.contains("<|im_start|>"));
+        assert!(prepared.prompt.contains("<|im_end|>"));
+    }
+
+    #[test]
+    #[ignore = "autoresearch benchmark"]
+    fn dynamic_template_gemma_family_uses_turn_tokens() {
+        let config = ServerConfig {
+            model_catalog: ModelCatalog::new("gemma-2-9b-it".to_owned(), vec![]),
+            ..base_config()
+        };
+        let request = parse_request(
+            r#"{
+                "model": "gemma-2-9b-it",
+                "messages": [{"role":"user","content":"hello"}]
+            }"#,
+        );
+
+        let prepared = prepare_prompt(&request, &config);
+        assert!(prepared.prompt.contains("<start_of_turn>user"));
+        assert!(prepared.prompt.contains("<start_of_turn>model"));
+    }
+
+    #[test]
+    #[ignore = "autoresearch benchmark"]
+    fn dynamic_template_llama_family_uses_inst_blocks() {
+        let config = ServerConfig {
+            model_catalog: ModelCatalog::new("Meta-Llama-3-8B-Instruct".to_owned(), vec![]),
+            ..base_config()
+        };
+        let request = parse_request(
+            r#"{
+                "model": "Meta-Llama-3-8B-Instruct",
+                "messages": [{"role":"user","content":"hello"}]
+            }"#,
+        );
+
+        let prepared = prepare_prompt(&request, &config);
+        assert!(prepared.prompt.contains("[INST]"));
+        assert!(prepared.prompt.contains("[/INST]"));
+    }
+
+    #[test]
+    #[ignore = "autoresearch benchmark"]
+    fn dynamic_template_kimi_family_uses_moonshot_markers() {
+        let config = ServerConfig {
+            model_catalog: ModelCatalog::new("Kimi-K2-Instruct".to_owned(), vec![]),
+            ..base_config()
+        };
+        let request = parse_request(
+            r#"{
+                "model": "Kimi-K2-Instruct",
+                "messages": [{"role":"user","content":"hello"}]
+            }"#,
+        );
+
+        let prepared = prepare_prompt(&request, &config);
+        assert!(prepared.prompt.contains("<|im_user|>"));
+        assert!(prepared.prompt.contains("<|im_assistant|>"));
+    }
+
+    #[test]
+    #[ignore = "autoresearch benchmark"]
+    fn dynamic_template_unknown_models_keep_generic_fallback() {
+        let config = ServerConfig {
+            model_catalog: ModelCatalog::new("my-custom-model".to_owned(), vec![]),
+            ..base_config()
+        };
+        let request = parse_request(
+            r#"{
+                "model": "my-custom-model",
+                "messages": [{"role":"user","content":"hello"}]
+            }"#,
+        );
+
+        let prepared = prepare_prompt(&request, &config);
+        assert!(prepared
+            .prompt
+            .contains("System: Default coding system prompt"));
+        assert!(prepared.prompt.contains("User: hello"));
+        assert!(prepared.prompt.ends_with("Assistant:"));
+    }
 }

experiments/bench_chat_templates.py

@@ -0,0 +1,70 @@
+#!/usr/bin/env python3
+import argparse
+import re
+import subprocess
+import sys
+from pathlib import Path
+
+
+TEST_FILTER = "dynamic_template_"
+TEST_PATTERN = re.compile(
+    r"^test\s+(.+dynamic_template_[^\s]+)\s+\.\.\.\s+(ok|FAILED)$"
+)
+
+
+def run_suite(root: Path) -> tuple[int, int, int]:
+    command = [
+        "cargo",
+        "test",
+        "-p",
+        "inference-server",
+        TEST_FILTER,
+        "--",
+        "--ignored",
+        "--nocapture",
+    ]
+    completed = subprocess.run(
+        command,
+        cwd=root,
+        capture_output=True,
+        text=True,
+        encoding="utf-8",
+        errors="replace",
+    )
+
+    output = completed.stdout + completed.stderr
+    print(output, end="")
+
+    seen: dict[str, str] = {}
+    for line in output.splitlines():
+        match = TEST_PATTERN.match(line.strip())
+        if match:
+            seen[match.group(1)] = match.group(2)
+
+    if not seen and completed.returncode != 0:
+        raise SystemExit(completed.returncode)
+
+    total = len(seen)
+    passed = sum(1 for status in seen.values() if status == "ok")
+    failed = total - passed
+    score = round((passed / total) * 100.0, 3) if total else 0.0
+
+    print(f"METRIC template_score={score}")
+    print(f"METRIC passed_cases={passed}")
+    print(f"METRIC failed_cases={failed}")
+    print(f"METRIC total_cases={total}")
+    return passed, failed, total
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--root", required=True)
+    args = parser.parse_args()
+
+    root = Path(args.root)
+    passed, failed, total = run_suite(root)
+    return 0 if total > 0 else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

experiments/worklog.md

@@ -7,7 +7,57 @@
 - Benchmark harness: `./autoresearch.sh`
 
 ## Key Insights
-- No experiment runs logged yet.
+- The first stable end-to-end benchmark is around 0.94 s p95 for a short non-streaming coding request on the local 4B GGUF model.
+- The benchmark harness must use the canonical model id derived from the GGUF filename; the current server build does not actually honor the intended `local-model` alias path yet.
+- Prompt prefill is only about 25 ms for this workload, so almost all latency sits in token generation rather than request parsing or prompt construction.
+- A micro-optimization in per-token detokenization made both latency and throughput worse, which suggests the dominant cost is deeper in the generation/runtime path rather than heap allocation in Rust glue code.
+- Shrinking the `Turbo4B` context and batch sizes also hurt throughput, so the current 4B profile appears closer to the local optimum for this hardware/workload than expected.
+- Disabling internal llama.cpp perf counters produced the first real p95 improvement over baseline, which suggests a small but measurable amount of hot-path CPU overhead was coming from instrumentation rather than model math.
+- Better stop behavior improved average and median latency, but it still failed to beat baseline p95, so the worst outliers are not explained solely by turn-boundary overruns.
+- Switching to a Qwen-style chat template produced a severe p95 tail regression while median latency stayed near baseline, which strongly suggests worse stop behavior or occasional runaway generations rather than uniformly slower decoding.
 
 ## Next Ideas
 - Measure baseline first, then inspect the hottest latency contributors in prompt preparation, request handling, and session creation.
+- Investigate why `ModelCatalog` rejects `local-model` despite intending to expose aliases.
+
+### Run 1: baseline - p95_ms=941.252 (KEEP)
+- Timestamp: 2026-03-26 17:32
+- What changed: Established the autoresearch harness, built the release server, started it against the local Qwen GGUF, and measured a fixed non-streaming coding request workload.
+- Result: p95_ms=941.252, p50_ms=908.457, mean_ms=917.123, completion_tps_mean=52.427, success_count=7.
+- Insight: The baseline is stable enough to compare future server/runtime changes, and the benchmark immediately exposed a compatibility bug in alias resolution that should be kept separate from true latency experiments.
+- Next: fix low-risk benchmark-adjacent correctness issues only if they distort measurement; otherwise inspect hot latency contributors in the request/session path.
+
+### Run 2: stack buffer detokenization - p95_ms=1032.315 (DISCARD)
+- Timestamp: 2026-03-26 17:38
+- What changed: Replaced the per-token heap allocation in `decode_token_piece_with()` with a 256-byte stack buffer and heap fallback in `crates/inference-core/src/session.rs`.
+- Result: p95_ms=1032.315, p50_ms=1020.402, mean_ms=1011.466, completion_tps_mean=47.671, success_count=7.
+- Insight: This path is not the practical bottleneck for end-to-end latency here; generation got slower overall despite the reduced allocation churn.
+- Next: discard this branch state and focus on runtime/profile-level experiments that can improve generation throughput directly.
+
+### Run 3: slim turbo4b runtime profile - p95_ms=1004.435 (DISCARD)
+- Timestamp: 2026-03-26 17:40
+- What changed: Reduced `Turbo4B` defaults in `crates/inference-core/src/profile.rs` from `n_ctx=4096`, `n_batch=512`, `n_ubatch=128` to `n_ctx=2048`, `n_batch=256`, `n_ubatch=64`.
+- Result: p95_ms=1004.435, p50_ms=998.919, mean_ms=990.771, completion_tps_mean=48.528, success_count=7.
+- Insight: The leaner profile still reduced effective generation speed on this RTX 3070 + Q6_K workload, so the current batch/context sizing is likely not the source of the baseline latency.
+- Next: revert to the best-known commit and target generation semantics or model-facing prompt/template strategy rather than detokenization or profile shrinkage.
+
+### Run 6: disable llama perf counters in hot path - p95_ms=925.587 (KEEP)
+- Timestamp: 2026-03-26 17:57
+- What changed: Set `llama_context_params.no_perf = true` in `crates/inference-core/src/engine.rs` and `sampler_chain_default_params.no_perf = true` in `crates/inference-core/src/session.rs`.
+- Result: p95_ms=925.587, p50_ms=893.913, mean_ms=899.895, completion_tps_mean=53.618, success_count=7.
+- Insight: Disabling internal perf bookkeeping shaved about 1.7% off baseline p95 with slightly better overall throughput, making this the first experiment worth keeping.
+- Next: continue from this commit and look for adjacent hot-path flags or runtime settings that reduce generation overhead without changing model behavior.
+
+### Run 5: default turn-boundary stops - p95_ms=965.663 (DISCARD)
+- Timestamp: 2026-03-26 17:54
+- What changed: Added rolling multi-token stop detection in `crates/inference-core/src/session.rs` and injected default turn-boundary stop sequences in `crates/inference-server/src/compat.rs` when the request did not specify `stop`.
+- Result: p95_ms=965.663, p50_ms=877.030, mean_ms=897.251, completion_tps_mean=53.686, success_count=7.
+- Insight: This reduced median and mean latency and slightly improved throughput, but the p95 still stayed above baseline, so tail latency likely comes from a smaller number of deeper generation stalls rather than just missed stop boundaries.
+- Next: inspect per-sample latency spread and target the remaining long-tail outliers directly, likely in warmup/first-request behavior or occasional decode stalls.
+
+### Run 4: qwen-style chat template - p95_ms=3027.145 (DISCARD)
+- Timestamp: 2026-03-26 17:50
+- What changed: Replaced the plain `System/User/Assistant:` prompt format in `crates/inference-server/src/compat.rs` with a Qwen-style `<|im_start|>...<|im_end|>` chat template.
+- Result: p95_ms=3027.145, p50_ms=974.937, mean_ms=1467.026, completion_tps_mean=41.360, success_count=7.
+- Insight: The template did not make typical requests much faster and introduced severe tail latency, likely because some generations stopped much later or followed a worse decoding path for this server/model combination.
+- Next: revert this template change and focus on experiments that directly bound generation length or improve stop behavior without overhauling the prompt format.