53 lines
4.3 KiB
Markdown
53 lines
4.3 KiB
Markdown
# Autoresearch: dynamic-chat-templates
|
|
|
|
## Objective
|
|
Implement a dynamic chat-template system that selects prompt rendering by model family
|
|
for local OpenAI-compatible `POST /v1/chat/completions` requests. The server should ship
|
|
useful built-in templates for common open-source families such as Qwen, Gemma, Llama,
|
|
and Kimi, while preserving a safe generic fallback for unknown models.
|
|
|
|
## Metrics
|
|
- **Primary**: template_score (percent, higher is better)
|
|
- **Secondary**: passed_cases, failed_cases, total_cases
|
|
|
|
## How to Run
|
|
`./autoresearch.sh` - runs targeted ignored tests for dynamic chat-template behavior and outputs `METRIC name=number` lines.
|
|
|
|
## Files in Scope
|
|
- `crates/inference-server/src/compat.rs` - prompt preparation, model catalog, template selection, family detection, and template-focused tests
|
|
- `crates/inference-server/src/routes.rs` - route-level integration if template selection needs model-aware wiring
|
|
- `crates/inference-server/src/main.rs` - server boot wiring if template config/env support is needed
|
|
- `crates/api-contracts/src/chat.rs` - request/response roles and content handling that templates consume
|
|
- `experiments/bench_chat_templates.py` - benchmark harness for the autoresearch loop
|
|
- `autoresearch.sh` - loop benchmark entrypoint and fast preflight
|
|
|
|
## Off Limits
|
|
- `models/` model binaries
|
|
- `vendor/` third-party sources and prebuilt llama binaries
|
|
- `registry/` Python registry service
|
|
- `ui/` frontend files
|
|
- unrelated latency tuning or runtime/engine experiments unless they are required to support template selection correctly
|
|
|
|
## Constraints
|
|
- No new external dependencies.
|
|
- Preserve OpenAI-compatible behavior already implemented.
|
|
- Regular test suite must stay green for keeps.
|
|
- Prefer explicit, maintainable template definitions over opaque heuristics.
|
|
- Unknown models must continue to work through a generic fallback template.
|
|
- The benchmark is test-driven: keep/discard follows the dynamic-template suite score first.
|
|
|
|
## What's Been Tried
|
|
- Segment 1 switches the autoresearch target from latency to dynamic chat-template selection by model family.
|
|
- The current server still renders every prompt through one generic `System:/User:/Assistant:` formatter in `crates/inference-server/src/compat.rs`.
|
|
- `prepare_prompt()` is the cleanest insertion point for dynamic templates because both blocking and streaming paths already consume its rendered string.
|
|
- `ModelCatalog` currently resolves aliases only; it has no model-family metadata or template registry yet.
|
|
- The benchmark for this segment is a targeted ignored test suite that scores family detection/template rendering coverage for Qwen, Gemma, Llama, Kimi, and the generic fallback.
|
|
- Baseline run 7 scored only `20.0%` because just the generic fallback path passed; Qwen, Gemma, Llama, and Kimi were still rendered through the old generic formatter.
|
|
- Run 8 (`ceb83f2`) added family detection plus built-in template selection inside `compat.rs` and immediately raised coverage to `100.0%` (5/5 passing), making it the current best for segment 1.
|
|
- Segment 2 raises the bar by checking stronger Llama structure, especially whether system+user are merged into the first instruction block and whether the prompt avoids leaving an open `[INST]` prefix at generation time.
|
|
- Run 10 (`fd6d35e`) hardened the Llama renderer with a dedicated formatting path that folds pending system/developer context into the next instruction block and eliminates the dangling open `[INST]` suffix, raising segment 2 from `71.429%` to `100.0%` (7/7 passing).
|
|
- Segment 3 raises the bar again around alias-driven family detection, specifically ensuring a family hint carried only by an alias still selects the right template even when the canonical/root model id is generic.
|
|
- Run 12 (`140c9b2`) fixed alias-driven family detection by checking the requested model string before falling back to resolved alias/root metadata, raising segment 3 from `88.889%` to `100.0%` (9/9 passing).
|
|
- Segment 4 raises the bar around family-name variants, starting with Moonshot-branded Kimi aliases so common `moonshot` names select Kimi markers while unrelated lookalikes still fall back safely to `Generic`.
|
|
- Run 14 (`f200733`) extended family detection so `moonshot` variant names map to the Kimi template, raising segment 4 from `90.909%` to `100.0%` (11/11 passing) without weakening generic fallback behavior.
|