ForgeLM/autoresearch.md

Autoresearch: dynamic-chat-templates

Objective

Implement a dynamic chat-template system that selects prompt rendering by model family for local OpenAI-compatible POST /v1/chat/completions requests. The server should ship useful built-in templates for common open-source families such as Qwen, Gemma, Llama, and Kimi, while preserving a safe generic fallback for unknown models.

Metrics

• Primary: template_score (percent, higher is better)
• Secondary: passed_cases, failed_cases, total_cases

How to Run

./autoresearch.sh - runs targeted ignored tests for dynamic chat-template behavior and outputs METRIC name=number lines.

Files in Scope

• crates/inference-server/src/compat.rs - prompt preparation, model catalog, template selection, family detection, and template-focused tests
• crates/inference-server/src/routes.rs - route-level integration if template selection needs model-aware wiring
• crates/inference-server/src/main.rs - server boot wiring if template config/env support is needed
• crates/api-contracts/src/chat.rs - request/response roles and content handling that templates consume
• experiments/bench_chat_templates.py - benchmark harness for the autoresearch loop
• autoresearch.sh - loop benchmark entrypoint and fast preflight

Off Limits

• models/ model binaries
• vendor/ third-party sources and prebuilt llama binaries
• registry/ Python registry service
• ui/ frontend files
• unrelated latency tuning or runtime/engine experiments unless they are required to support template selection correctly

Constraints

• No new external dependencies.
• Preserve OpenAI-compatible behavior already implemented.
• Regular test suite must stay green for keeps.
• Prefer explicit, maintainable template definitions over opaque heuristics.
• Unknown models must continue to work through a generic fallback template.
• The benchmark is test-driven: keep/discard follows the dynamic-template suite score first.

What's Been Tried

• Segment 1 switches the autoresearch target from latency to dynamic chat-template selection by model family.
• The current server still renders every prompt through one generic System:/User:/Assistant: formatter in crates/inference-server/src/compat.rs.
• prepare_prompt() is the cleanest insertion point for dynamic templates because both blocking and streaming paths already consume its rendered string.
• ModelCatalog currently resolves aliases only; it has no model-family metadata or template registry yet.
• The benchmark for this segment is a targeted ignored test suite that scores family detection/template rendering coverage for Qwen, Gemma, Llama, Kimi, and the generic fallback.
• Baseline run 7 scored only 20.0% because just the generic fallback path passed; Qwen, Gemma, Llama, and Kimi were still rendered through the old generic formatter.
• Run 8 (ceb83f2) added family detection plus built-in template selection inside compat.rs and immediately raised coverage to 100.0% (5/5 passing), making it the current best for segment 1.
• Segment 2 raises the bar by checking stronger Llama structure, especially whether system+user are merged into the first instruction block and whether the prompt avoids leaving an open [INST] prefix at generation time.
• Run 10 (fd6d35e) hardened the Llama renderer with a dedicated formatting path that folds pending system/developer context into the next instruction block and eliminates the dangling open [INST] suffix, raising segment 2 from 71.429% to 100.0% (7/7 passing).
• Segment 3 raises the bar again around alias-driven family detection, specifically ensuring a family hint carried only by an alias still selects the right template even when the canonical/root model id is generic.
• Run 12 (140c9b2) fixed alias-driven family detection by checking the requested model string before falling back to resolved alias/root metadata, raising segment 3 from 88.889% to 100.0% (9/9 passing).
• Segment 4 raises the bar around family-name variants, starting with Moonshot-branded Kimi aliases so common moonshot names select Kimi markers while unrelated lookalikes still fall back safely to Generic.
• Run 14 (f200733) extended family detection so moonshot variant names map to the Kimi template, raising segment 4 from 90.909% to 100.0% (11/11 passing) without weakening generic fallback behavior.