feat: add family-based chat templates

This commit is contained in:
Evandro Fonseca Junior committed 2026-03-26 18:16:54 -03:00
1 parent b3001f6234
commit bad2bb33a5
4 files changed
+223 -61

No files matched your search

+12
View File
@@ -61,3 +61,15 @@
- Result: p95_ms=3027.145, p50_ms=974.937, mean_ms=1467.026, completion_tps_mean=41.360, success_count=7.
- Insight: The template did not make typical requests much faster and introduced severe tail latency, likely because some generations stopped much later or followed a worse decoding path for this server/model combination.
- Next: revert this template change and focus on experiments that directly bound generation length or improve stop behavior without overhauling the prompt format.
## Segment 1: Dynamic Chat Templates
- New objective: add dynamic prompt templates by model family with built-in coverage for Qwen, Gemma, Llama, Kimi, and a generic fallback.
- Benchmark driver: `./autoresearch.sh` now runs ignored template-specific tests and reports `template_score`, `passed_cases`, `failed_cases`, and `total_cases`.
- Keep/discard rule for this segment is test-driven: higher `template_score` wins, while the regular non-ignored test suite must remain green.
### Run 7: baseline - template_score=20.0 (KEEP)
- Timestamp: 2026-03-26 18:14
- What changed: Re-initialized autoresearch to a new segment focused on dynamic chat templates, added a dedicated ignored test suite for Qwen, Gemma, Llama, Kimi, and generic fallback behavior, and switched the benchmark harness to score that suite.
- Result: template_score=20.0, passed_cases=1, failed_cases=4, total_cases=5.
- Insight: Only the generic fallback works today; all named family-template expectations fail because prompt rendering is still generic and model-agnostic.
- Next: add a minimal model-family detector and template registry in `crates/inference-server/src/compat.rs`, then route `prepare_prompt()` through it.