docs: finalize autoresearch template coverage
This commit is contained in:
1 parent
79dd5c6bb6
commit
aa0161a67c
4 files changed
+53
-5
No files matched your search
@@ -100,3 +100,41 @@
|
||||
- Result: template_score=100.0, passed_cases=7, failed_cases=0, total_cases=7.
|
||||
- Insight: Llama needed a specialized formatter instead of the generic per-message helper; once the prompt was rendered as proper closed instruction blocks with folded system context, the hardening suite passed cleanly.
|
||||
- Next: raise the benchmark again around multi-turn continuity, tool-role placement, or alias-driven family detection so the next experiments target real edge cases instead of already-solved happy paths.
|
||||
|
||||
## Segment 3: Alias-Driven Template Selection
|
||||
- New objective: harden family detection so template selection can still infer the right family from an alias even when the canonical/root model id is generic.
|
||||
- Benchmark driver: `./autoresearch.sh` still runs the ignored template suite, now expanded with alias-sensitive cases.
|
||||
- Keep/discard rule remains test-driven: higher `template_score` wins while the regular non-ignored suite stays green.
|
||||
|
||||
### Run 11: baseline - template_score=88.889 (KEEP)
|
||||
- Timestamp: 2026-03-26 18:27
|
||||
- What changed: Expanded the ignored benchmark suite with alias-sensitive family-selection coverage, especially a case where the canonical model id is generic but the requested alias carries the `qwen` family hint.
|
||||
- Result: template_score=88.889, passed_cases=8, failed_cases=1, total_cases=9.
|
||||
- Insight: The family templates and fallback behavior are otherwise stable; the remaining gap is specifically that `select_chat_template()` loses the alias hint when it resolves through a generic canonical/root id.
|
||||
- Next: make template selection consider both the requested model string and the resolved catalog/root metadata, preferring whichever one contains a recognized family marker.
|
||||
|
||||
### Run 12: detect template family from aliases - template_score=100.0 (KEEP)
|
||||
- Timestamp: 2026-03-26 18:28
|
||||
- What changed: Updated `select_chat_template()` in `crates/inference-server/src/compat.rs` to evaluate the requested model name first, then resolved alias/canonical/root ids, so family hints present only in aliases still drive template selection.
|
||||
- Result: template_score=100.0, passed_cases=9, failed_cases=0, total_cases=9.
|
||||
- Insight: Alias-aware selection closes the last known family-detection gap without needing extra catalog metadata; simple ordered candidate matching is enough here.
|
||||
- Next: raise the benchmark again around multi-turn continuity, tool-call/tool-role rendering, or family-specific response-format instructions to target subtler correctness gaps.
|
||||
|
||||
## Segment 4: Variant Family Detection
|
||||
- New objective: harden family detection for common model-name variants, starting with Moonshot-branded Kimi aliases.
|
||||
- Benchmark driver: `./autoresearch.sh` still runs the ignored template suite, now expanded with variant-sensitive cases.
|
||||
- Keep/discard rule remains test-driven: higher `template_score` wins while the regular non-ignored suite stays green.
|
||||
|
||||
### Run 13: baseline - template_score=90.909 (KEEP)
|
||||
- Timestamp: 2026-03-26 18:32
|
||||
- What changed: Expanded the ignored template suite with Moonshot/Kimi variant coverage, including one positive Moonshot alias case and one near-name generic fallback case.
|
||||
- Result: template_score=90.909, passed_cases=10, failed_cases=1, total_cases=11.
|
||||
- Insight: The only remaining gap is that `select_chat_template()` recognizes `kimi` but not the common `moonshot` naming variant; nearby names like `moonshine` already fall back safely.
|
||||
- Next: extend family detection so Moonshot-branded aliases map to the Kimi template without broadening the match so far that unrelated names stop using the generic fallback.
|
||||
|
||||
### Run 14: detect moonshot as kimi family - template_score=100.0 (KEEP)
|
||||
- Timestamp: 2026-03-26 18:33
|
||||
- What changed: Extended `select_chat_template()` in `crates/inference-server/src/compat.rs` so candidate names containing `moonshot` are treated as the Kimi family alongside direct `kimi` matches.
|
||||
- Result: template_score=100.0, passed_cases=11, failed_cases=0, total_cases=11.
|
||||
- Insight: A narrow Moonshot variant match closes the Kimi-family naming gap cleanly without breaking safe generic fallback for unrelated names such as `moonshine-local`.
|
||||
- Next: raise the benchmark around multi-turn continuity, tool-role rendering, or family-specific response-format instructions.
|
||||
Reference in new issue
Block a user