diff --git a/autoresearch-dashboard.md b/autoresearch-dashboard.md index 77b8e17..3605502 100644 --- a/autoresearch-dashboard.md +++ b/autoresearch-dashboard.md @@ -1,9 +1,10 @@ # Autoresearch Dashboard: dynamic-chat-templates -**Runs:** 1 | **Kept:** 1 | **Discarded:** 0 | **Crashed:** 0 +**Runs:** 2 | **Kept:** 2 | **Discarded:** 0 | **Crashed:** 0 **Baseline:** template_score: 20.0% (#7) -**Best:** template_score: 20.0% (#7, +0.0%) +**Best:** template_score: 100.0% (#8, +400.0%) | # | commit | template_score | status | description | |---|--------|----------------|--------|-------------| | 7 | 89ef220 | 20.0% (+0.0%) | keep | baseline | +| 8 | ceb83f2 | 100.0% (+400.0%) | keep | add family-based chat templates | diff --git a/autoresearch.jsonl b/autoresearch.jsonl index 573ee49..9c0e0bc 100644 --- a/autoresearch.jsonl +++ b/autoresearch.jsonl @@ -7,3 +7,4 @@ {"run":6,"commit":"e8402d2","metric":925.587,"metrics":{"p50_ms":893.913,"mean_ms":899.895,"completion_tps_mean":53.618,"success_count":7},"status":"keep","description":"disable llama perf counters in hot path","timestamp":1774558642,"segment":0} {"type":"config","name":"dynamic-chat-templates","metricName":"template_score","metricUnit":"%","bestDirection":"higher"} {"run":7,"commit":"89ef220","metric":20.0,"metrics":{"passed_cases":1,"failed_cases":4,"total_cases":5},"status":"keep","description":"baseline","timestamp":1774559683,"segment":1} +{"run":8,"commit":"ceb83f2","metric":100.0,"metrics":{"passed_cases":5,"failed_cases":0,"total_cases":5},"status":"keep","description":"add family-based chat templates","timestamp":1774559814,"segment":1} diff --git a/autoresearch.md b/autoresearch.md index c548c29..a84160d 100644 --- a/autoresearch.md +++ b/autoresearch.md @@ -42,3 +42,5 @@ and Kimi, while preserving a safe generic fallback for unknown models. - `prepare_prompt()` is the cleanest insertion point for dynamic templates because both blocking and streaming paths already consume its rendered string. - `ModelCatalog` currently resolves aliases only; it has no model-family metadata or template registry yet. - The benchmark for this segment is a targeted ignored test suite that scores family detection/template rendering coverage for Qwen, Gemma, Llama, Kimi, and the generic fallback. +- Baseline run 7 scored only `20.0%` because just the generic fallback path passed; Qwen, Gemma, Llama, and Kimi were still rendered through the old generic formatter. +- Run 8 (`ceb83f2`) added family detection plus built-in template selection inside `compat.rs` and immediately raised coverage to `100.0%` (5/5 passing), making it the current best for segment 1. diff --git a/crates/inference-server/src/compat.rs b/crates/inference-server/src/compat.rs index be94b8a..01ceef9 100644 --- a/crates/inference-server/src/compat.rs +++ b/crates/inference-server/src/compat.rs @@ -929,6 +929,44 @@ mod tests { assert!(prepared.prompt.contains("[/INST]")); } + #[test] + #[ignore = "autoresearch benchmark"] + fn dynamic_template_llama_family_combines_system_and_user_in_first_inst_block() { + let config = ServerConfig { + model_catalog: ModelCatalog::new("Meta-Llama-3-8B-Instruct".to_owned(), vec![]), + ..base_config() + }; + let request = parse_request( + r#"{ + "model": "Meta-Llama-3-8B-Instruct", + "messages": [{"role":"user","content":"hello"}] + }"#, + ); + + let prepared = prepare_prompt(&request, &config); + assert!(prepared.prompt.contains("[INST] <>")); + assert!(prepared.prompt.contains("hello [/INST]")); + assert_eq!(prepared.prompt.matches("[INST]").count(), 1); + } + + #[test] + #[ignore = "autoresearch benchmark"] + fn dynamic_template_llama_family_does_not_end_with_open_inst_prefix() { + let config = ServerConfig { + model_catalog: ModelCatalog::new("Meta-Llama-3-8B-Instruct".to_owned(), vec![]), + ..base_config() + }; + let request = parse_request( + r#"{ + "model": "Meta-Llama-3-8B-Instruct", + "messages": [{"role":"user","content":"hello"}] + }"#, + ); + + let prepared = prepare_prompt(&request, &config); + assert!(!prepared.prompt.ends_with("[INST] ")); + } + #[test] #[ignore = "autoresearch benchmark"] fn dynamic_template_kimi_family_uses_moonshot_markers() { diff --git a/experiments/worklog.md b/experiments/worklog.md index a91f397..af8a7fa 100644 --- a/experiments/worklog.md +++ b/experiments/worklog.md @@ -66,6 +66,7 @@ - New objective: add dynamic prompt templates by model family with built-in coverage for Qwen, Gemma, Llama, Kimi, and a generic fallback. - Benchmark driver: `./autoresearch.sh` now runs ignored template-specific tests and reports `template_score`, `passed_cases`, `failed_cases`, and `total_cases`. - Keep/discard rule for this segment is test-driven: higher `template_score` wins, while the regular non-ignored test suite must remain green. +- A minimal family detector plus embedded template registry in `compat.rs` was enough to move coverage from 1/5 to 5/5 without touching the runtime. ### Run 7: baseline - template_score=20.0 (KEEP) - Timestamp: 2026-03-26 18:14 @@ -73,3 +74,10 @@ - Result: template_score=20.0, passed_cases=1, failed_cases=4, total_cases=5. - Insight: Only the generic fallback works today; all named family-template expectations fail because prompt rendering is still generic and model-agnostic. - Next: add a minimal model-family detector and template registry in `crates/inference-server/src/compat.rs`, then route `prepare_prompt()` through it. + +### Run 8: add family-based chat templates - template_score=100.0 (KEEP) +- Timestamp: 2026-03-26 18:16 +- What changed: Added model-family detection and template-aware prompt rendering in `crates/inference-server/src/compat.rs` for Qwen, Gemma, Llama, Kimi, and generic fallback, then routed system/tool/JSON-mode instructions and assistant prefixes through the selected template. +- Result: template_score=100.0, passed_cases=5, failed_cases=0, total_cases=5. +- Insight: The current seam in `prepare_prompt()` is sufficient for broad family-based template support; no runtime or routing changes were needed for the first successful implementation. +- Next: harden the design by adding alias-driven family detection, safer fallback behavior for mixed-case/variant model names, and more coverage for tool-call and response-format rendering under non-generic templates.