chore: expand template benchmark coverage
This commit is contained in:
1 parent
bad2bb33a5
commit
f96b42c356
5 files changed
+52
-2
No files matched your search
@@ -66,6 +66,7 @@
|
||||
- New objective: add dynamic prompt templates by model family with built-in coverage for Qwen, Gemma, Llama, Kimi, and a generic fallback.
|
||||
- Benchmark driver: `./autoresearch.sh` now runs ignored template-specific tests and reports `template_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
||||
- Keep/discard rule for this segment is test-driven: higher `template_score` wins, while the regular non-ignored test suite must remain green.
|
||||
- A minimal family detector plus embedded template registry in `compat.rs` was enough to move coverage from 1/5 to 5/5 without touching the runtime.
|
||||
|
||||
### Run 7: baseline - template_score=20.0 (KEEP)
|
||||
- Timestamp: 2026-03-26 18:14
|
||||
@@ -73,3 +74,10 @@
|
||||
- Result: template_score=20.0, passed_cases=1, failed_cases=4, total_cases=5.
|
||||
- Insight: Only the generic fallback works today; all named family-template expectations fail because prompt rendering is still generic and model-agnostic.
|
||||
- Next: add a minimal model-family detector and template registry in `crates/inference-server/src/compat.rs`, then route `prepare_prompt()` through it.
|
||||
|
||||
### Run 8: add family-based chat templates - template_score=100.0 (KEEP)
|
||||
- Timestamp: 2026-03-26 18:16
|
||||
- What changed: Added model-family detection and template-aware prompt rendering in `crates/inference-server/src/compat.rs` for Qwen, Gemma, Llama, Kimi, and generic fallback, then routed system/tool/JSON-mode instructions and assistant prefixes through the selected template.
|
||||
- Result: template_score=100.0, passed_cases=5, failed_cases=0, total_cases=5.
|
||||
- Insight: The current seam in `prepare_prompt()` is sufficient for broad family-based template support; no runtime or routing changes were needed for the first successful implementation.
|
||||
- Next: harden the design by adding alias-driven family detection, safer fallback behavior for mixed-case/variant model names, and more coverage for tool-call and response-format rendering under non-generic templates.
|
||||
Reference in new issue
Block a user