chore: expand alias template benchmark coverage

This commit is contained in:
Evandro Fonseca Junior committed 2026-03-26 18:26:40 -03:00
1 parent 28d3fa7b51
commit 9bf79ff68c
5 files changed
+76 -5

No files matched your search

+19
View File
@@ -81,3 +81,22 @@
- Result: template_score=100.0, passed_cases=5, failed_cases=0, total_cases=5.
- Insight: The current seam in `prepare_prompt()` is sufficient for broad family-based template support; no runtime or routing changes were needed for the first successful implementation.
- Next: harden the design by adding alias-driven family detection, safer fallback behavior for mixed-case/variant model names, and more coverage for tool-call and response-format rendering under non-generic templates.
## Segment 2: Dynamic Chat Template Hardening
- New objective: keep family-based template selection, but harden structure-sensitive formatting for instruct-style models, starting with Llama.
- Benchmark driver: `./autoresearch.sh` still runs the ignored template suite, now with stricter Llama assertions.
- Keep/discard rule remains test-driven: higher `template_score` wins while the regular non-ignored suite stays green.
### Run 9: baseline - template_score=71.429 (KEEP)
- Timestamp: 2026-03-26 18:21
- What changed: Expanded the ignored benchmark suite with two stronger Llama assertions that require system+user to be merged into the first instruction block and forbid leaving an open `[INST]` prefix at generation time.
- Result: template_score=71.429, passed_cases=5, failed_cases=2, total_cases=7.
- Insight: The current family detector is good enough, but the Llama renderer is still structurally naive; it over-produces instruction blocks and leaves the prompt in an invalid generation state.
- Next: refactor Llama rendering into a dedicated formatter that folds system context into the first user turn and emits a proper assistant generation prefix.
### Run 10: harden llama chat template rendering - template_score=100.0 (KEEP)
- Timestamp: 2026-03-26 18:23
- What changed: Reworked the Llama path in `crates/inference-server/src/compat.rs` into a dedicated renderer that accumulates pending system/developer content, folds it into the next user/tool instruction block, and stops emitting a dangling open assistant `[INST]` prefix.
- Result: template_score=100.0, passed_cases=7, failed_cases=0, total_cases=7.
- Insight: Llama needed a specialized formatter instead of the generic per-message helper; once the prompt was rendered as proper closed instruction blocks with folded system context, the hardening suite passed cleanly.
- Next: raise the benchmark again around multi-turn continuity, tool-role placement, or alias-driven family detection so the next experiments target real edge cases instead of already-solved happy paths.