chore: expand template benchmark coverage

This commit is contained in:
Evandro Fonseca Junior committed 2026-03-26 18:20:35 -03:00
1 parent bad2bb33a5
commit f96b42c356
5 files changed
+52 -2

No files matched your search

+8
View File
@@ -66,6 +66,7 @@
- New objective: add dynamic prompt templates by model family with built-in coverage for Qwen, Gemma, Llama, Kimi, and a generic fallback.
- Benchmark driver: `./autoresearch.sh` now runs ignored template-specific tests and reports `template_score`, `passed_cases`, `failed_cases`, and `total_cases`.
- Keep/discard rule for this segment is test-driven: higher `template_score` wins, while the regular non-ignored test suite must remain green.
- A minimal family detector plus embedded template registry in `compat.rs` was enough to move coverage from 1/5 to 5/5 without touching the runtime.
### Run 7: baseline - template_score=20.0 (KEEP)
- Timestamp: 2026-03-26 18:14
@@ -73,3 +74,10 @@
- Result: template_score=20.0, passed_cases=1, failed_cases=4, total_cases=5.
- Insight: Only the generic fallback works today; all named family-template expectations fail because prompt rendering is still generic and model-agnostic.
- Next: add a minimal model-family detector and template registry in `crates/inference-server/src/compat.rs`, then route `prepare_prompt()` through it.
### Run 8: add family-based chat templates - template_score=100.0 (KEEP)
- Timestamp: 2026-03-26 18:16
- What changed: Added model-family detection and template-aware prompt rendering in `crates/inference-server/src/compat.rs` for Qwen, Gemma, Llama, Kimi, and generic fallback, then routed system/tool/JSON-mode instructions and assistant prefixes through the selected template.
- Result: template_score=100.0, passed_cases=5, failed_cases=0, total_cases=5.
- Insight: The current seam in `prepare_prompt()` is sufficient for broad family-based template support; no runtime or routing changes were needed for the first successful implementation.
- Next: harden the design by adding alias-driven family detection, safer fallback behavior for mixed-case/variant model names, and more coverage for tool-call and response-format rendering under non-generic templates.