chore: expand template benchmark coverage

autoresearch-dashboard.md

@@ -1,9 +1,10 @@
 # Autoresearch Dashboard: dynamic-chat-templates
 
-**Runs:** 1 | **Kept:** 1 | **Discarded:** 0 | **Crashed:** 0
+**Runs:** 2 | **Kept:** 2 | **Discarded:** 0 | **Crashed:** 0
 **Baseline:** template_score: 20.0% (#7)
-**Best:** template_score: 20.0% (#7, +0.0%)
+**Best:** template_score: 100.0% (#8, +400.0%)
 
 | # | commit | template_score | status | description |
 |---|--------|----------------|--------|-------------|
 | 7 | 89ef220 | 20.0% (+0.0%) | keep | baseline |
+| 8 | ceb83f2 | 100.0% (+400.0%) | keep | add family-based chat templates |

autoresearch.jsonl

@@ -7,3 +7,4 @@
 {"run":6,"commit":"e8402d2","metric":925.587,"metrics":{"p50_ms":893.913,"mean_ms":899.895,"completion_tps_mean":53.618,"success_count":7},"status":"keep","description":"disable llama perf counters in hot path","timestamp":1774558642,"segment":0}
 {"type":"config","name":"dynamic-chat-templates","metricName":"template_score","metricUnit":"%","bestDirection":"higher"}
 {"run":7,"commit":"89ef220","metric":20.0,"metrics":{"passed_cases":1,"failed_cases":4,"total_cases":5},"status":"keep","description":"baseline","timestamp":1774559683,"segment":1}
+{"run":8,"commit":"ceb83f2","metric":100.0,"metrics":{"passed_cases":5,"failed_cases":0,"total_cases":5},"status":"keep","description":"add family-based chat templates","timestamp":1774559814,"segment":1}

autoresearch.md

@@ -42,3 +42,5 @@ and Kimi, while preserving a safe generic fallback for unknown models.
 - `prepare_prompt()` is the cleanest insertion point for dynamic templates because both blocking and streaming paths already consume its rendered string.
 - `ModelCatalog` currently resolves aliases only; it has no model-family metadata or template registry yet.
 - The benchmark for this segment is a targeted ignored test suite that scores family detection/template rendering coverage for Qwen, Gemma, Llama, Kimi, and the generic fallback.
+- Baseline run 7 scored only `20.0%` because just the generic fallback path passed; Qwen, Gemma, Llama, and Kimi were still rendered through the old generic formatter.
+- Run 8 (`ceb83f2`) added family detection plus built-in template selection inside `compat.rs` and immediately raised coverage to `100.0%` (5/5 passing), making it the current best for segment 1.

crates/inference-server/src/compat.rs

@@ -929,6 +929,44 @@ mod tests {
         assert!(prepared.prompt.contains("[/INST]"));
     }
 
+    #[test]
+    #[ignore = "autoresearch benchmark"]
+    fn dynamic_template_llama_family_combines_system_and_user_in_first_inst_block() {
+        let config = ServerConfig {
+            model_catalog: ModelCatalog::new("Meta-Llama-3-8B-Instruct".to_owned(), vec![]),
+            ..base_config()
+        };
+        let request = parse_request(
+            r#"{
+                "model": "Meta-Llama-3-8B-Instruct",
+                "messages": [{"role":"user","content":"hello"}]
+            }"#,
+        );
+
+        let prepared = prepare_prompt(&request, &config);
+        assert!(prepared.prompt.contains("[INST] <<SYS>>"));
+        assert!(prepared.prompt.contains("hello [/INST]"));
+        assert_eq!(prepared.prompt.matches("[INST]").count(), 1);
+    }
+
+    #[test]
+    #[ignore = "autoresearch benchmark"]
+    fn dynamic_template_llama_family_does_not_end_with_open_inst_prefix() {
+        let config = ServerConfig {
+            model_catalog: ModelCatalog::new("Meta-Llama-3-8B-Instruct".to_owned(), vec![]),
+            ..base_config()
+        };
+        let request = parse_request(
+            r#"{
+                "model": "Meta-Llama-3-8B-Instruct",
+                "messages": [{"role":"user","content":"hello"}]
+            }"#,
+        );
+
+        let prepared = prepare_prompt(&request, &config);
+        assert!(!prepared.prompt.ends_with("[INST] "));
+    }
+
     #[test]
     #[ignore = "autoresearch benchmark"]
     fn dynamic_template_kimi_family_uses_moonshot_markers() {

experiments/worklog.md

@@ -66,6 +66,7 @@
 - New objective: add dynamic prompt templates by model family with built-in coverage for Qwen, Gemma, Llama, Kimi, and a generic fallback.
 - Benchmark driver: `./autoresearch.sh` now runs ignored template-specific tests and reports `template_score`, `passed_cases`, `failed_cases`, and `total_cases`.
 - Keep/discard rule for this segment is test-driven: higher `template_score` wins, while the regular non-ignored test suite must remain green.
+- A minimal family detector plus embedded template registry in `compat.rs` was enough to move coverage from 1/5 to 5/5 without touching the runtime.
 
 ### Run 7: baseline - template_score=20.0 (KEEP)
 - Timestamp: 2026-03-26 18:14
@@ -73,3 +74,10 @@
 - Result: template_score=20.0, passed_cases=1, failed_cases=4, total_cases=5.
 - Insight: Only the generic fallback works today; all named family-template expectations fail because prompt rendering is still generic and model-agnostic.
 - Next: add a minimal model-family detector and template registry in `crates/inference-server/src/compat.rs`, then route `prepare_prompt()` through it.
+
+### Run 8: add family-based chat templates - template_score=100.0 (KEEP)
+- Timestamp: 2026-03-26 18:16
+- What changed: Added model-family detection and template-aware prompt rendering in `crates/inference-server/src/compat.rs` for Qwen, Gemma, Llama, Kimi, and generic fallback, then routed system/tool/JSON-mode instructions and assistant prefixes through the selected template.
+- Result: template_score=100.0, passed_cases=5, failed_cases=0, total_cases=5.
+- Insight: The current seam in `prepare_prompt()` is sufficient for broad family-based template support; no runtime or routing changes were needed for the first successful implementation.
+- Next: harden the design by adding alias-driven family detection, safer fallback behavior for mixed-case/variant model names, and more coverage for tool-call and response-format rendering under non-generic templates.