From aa0161a67cf1e000f2d0c7f6b4aae4317871d2ea Mon Sep 17 00:00:00 2001 From: Evandro Fonseca Junior Date: Thu, 26 Mar 2026 18:44:42 -0300 Subject: [PATCH] docs: finalize autoresearch template coverage --- autoresearch-dashboard.md | 10 +++++----- autoresearch.jsonl | 6 ++++++ autoresearch.md | 4 ++++ experiments/worklog.md | 38 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 53 insertions(+), 5 deletions(-) diff --git a/autoresearch-dashboard.md b/autoresearch-dashboard.md index bc93cb6..058fb9d 100644 --- a/autoresearch-dashboard.md +++ b/autoresearch-dashboard.md @@ -1,10 +1,10 @@ -# Autoresearch Dashboard: dynamic-chat-templates-hardening +# Autoresearch Dashboard: dynamic-chat-templates-variants **Runs:** 2 | **Kept:** 2 | **Discarded:** 0 | **Crashed:** 0 -**Baseline:** template_score: 71.429% (#9) -**Best:** template_score: 100.0% (#10, +40.0%) +**Baseline:** template_score: 90.909% (#13) +**Best:** template_score: 100.0% (#14, +10.0%) | # | commit | template_score | status | description | |---|--------|----------------|--------|-------------| -| 9 | 5f3ed5e | 71.429% (+0.0%) | keep | baseline | -| 10 | fd6d35e | 100.0% (+40.0%) | keep | harden llama chat template rendering | +| 13 | ffab766 | 90.909% (+0.0%) | keep | baseline | +| 14 | f200733 | 100.0% (+10.0%) | keep | detect moonshot as kimi family | diff --git a/autoresearch.jsonl b/autoresearch.jsonl index 06a95e1..c78b380 100644 --- a/autoresearch.jsonl +++ b/autoresearch.jsonl @@ -11,3 +11,9 @@ {"type":"config","name":"dynamic-chat-templates-hardening","metricName":"template_score","metricUnit":"%","bestDirection":"higher"} {"run":9,"commit":"5f3ed5e","metric":71.429,"metrics":{"passed_cases":5,"failed_cases":2,"total_cases":7},"status":"keep","description":"baseline","timestamp":1774560085,"segment":2} {"run":10,"commit":"fd6d35e","metric":100.0,"metrics":{"passed_cases":7,"failed_cases":0,"total_cases":7},"status":"keep","description":"harden llama chat template rendering","timestamp":1774560229,"segment":2} +{"type":"config","name":"dynamic-chat-templates-aliases","metricName":"template_score","metricUnit":"%","bestDirection":"higher"} +{"run":11,"commit":"2efe601","metric":88.889,"metrics":{"passed_cases":8,"failed_cases":1,"total_cases":9},"status":"keep","description":"baseline","timestamp":1774560452,"segment":3} +{"run":12,"commit":"140c9b2","metric":100.0,"metrics":{"passed_cases":9,"failed_cases":0,"total_cases":9},"status":"keep","description":"detect template family from aliases","timestamp":1774560533,"segment":3} +{"type":"config","name":"dynamic-chat-templates-variants","metricName":"template_score","metricUnit":"%","bestDirection":"higher"} +{"run":13,"commit":"ffab766","metric":90.909,"metrics":{"passed_cases":10,"failed_cases":1,"total_cases":11},"status":"keep","description":"baseline","timestamp":1774560738,"segment":4} +{"run":14,"commit":"f200733","metric":100.0,"metrics":{"passed_cases":11,"failed_cases":0,"total_cases":11},"status":"keep","description":"detect moonshot as kimi family","timestamp":1774560829,"segment":4} diff --git a/autoresearch.md b/autoresearch.md index 4c7618e..a2c1285 100644 --- a/autoresearch.md +++ b/autoresearch.md @@ -46,3 +46,7 @@ and Kimi, while preserving a safe generic fallback for unknown models. - Run 8 (`ceb83f2`) added family detection plus built-in template selection inside `compat.rs` and immediately raised coverage to `100.0%` (5/5 passing), making it the current best for segment 1. - Segment 2 raises the bar by checking stronger Llama structure, especially whether system+user are merged into the first instruction block and whether the prompt avoids leaving an open `[INST]` prefix at generation time. - Run 10 (`fd6d35e`) hardened the Llama renderer with a dedicated formatting path that folds pending system/developer context into the next instruction block and eliminates the dangling open `[INST]` suffix, raising segment 2 from `71.429%` to `100.0%` (7/7 passing). +- Segment 3 raises the bar again around alias-driven family detection, specifically ensuring a family hint carried only by an alias still selects the right template even when the canonical/root model id is generic. +- Run 12 (`140c9b2`) fixed alias-driven family detection by checking the requested model string before falling back to resolved alias/root metadata, raising segment 3 from `88.889%` to `100.0%` (9/9 passing). +- Segment 4 raises the bar around family-name variants, starting with Moonshot-branded Kimi aliases so common `moonshot` names select Kimi markers while unrelated lookalikes still fall back safely to `Generic`. +- Run 14 (`f200733`) extended family detection so `moonshot` variant names map to the Kimi template, raising segment 4 from `90.909%` to `100.0%` (11/11 passing) without weakening generic fallback behavior. diff --git a/experiments/worklog.md b/experiments/worklog.md index a71106d..0a093ef 100644 --- a/experiments/worklog.md +++ b/experiments/worklog.md @@ -100,3 +100,41 @@ - Result: template_score=100.0, passed_cases=7, failed_cases=0, total_cases=7. - Insight: Llama needed a specialized formatter instead of the generic per-message helper; once the prompt was rendered as proper closed instruction blocks with folded system context, the hardening suite passed cleanly. - Next: raise the benchmark again around multi-turn continuity, tool-role placement, or alias-driven family detection so the next experiments target real edge cases instead of already-solved happy paths. + +## Segment 3: Alias-Driven Template Selection +- New objective: harden family detection so template selection can still infer the right family from an alias even when the canonical/root model id is generic. +- Benchmark driver: `./autoresearch.sh` still runs the ignored template suite, now expanded with alias-sensitive cases. +- Keep/discard rule remains test-driven: higher `template_score` wins while the regular non-ignored suite stays green. + +### Run 11: baseline - template_score=88.889 (KEEP) +- Timestamp: 2026-03-26 18:27 +- What changed: Expanded the ignored benchmark suite with alias-sensitive family-selection coverage, especially a case where the canonical model id is generic but the requested alias carries the `qwen` family hint. +- Result: template_score=88.889, passed_cases=8, failed_cases=1, total_cases=9. +- Insight: The family templates and fallback behavior are otherwise stable; the remaining gap is specifically that `select_chat_template()` loses the alias hint when it resolves through a generic canonical/root id. +- Next: make template selection consider both the requested model string and the resolved catalog/root metadata, preferring whichever one contains a recognized family marker. + +### Run 12: detect template family from aliases - template_score=100.0 (KEEP) +- Timestamp: 2026-03-26 18:28 +- What changed: Updated `select_chat_template()` in `crates/inference-server/src/compat.rs` to evaluate the requested model name first, then resolved alias/canonical/root ids, so family hints present only in aliases still drive template selection. +- Result: template_score=100.0, passed_cases=9, failed_cases=0, total_cases=9. +- Insight: Alias-aware selection closes the last known family-detection gap without needing extra catalog metadata; simple ordered candidate matching is enough here. +- Next: raise the benchmark again around multi-turn continuity, tool-call/tool-role rendering, or family-specific response-format instructions to target subtler correctness gaps. + +## Segment 4: Variant Family Detection +- New objective: harden family detection for common model-name variants, starting with Moonshot-branded Kimi aliases. +- Benchmark driver: `./autoresearch.sh` still runs the ignored template suite, now expanded with variant-sensitive cases. +- Keep/discard rule remains test-driven: higher `template_score` wins while the regular non-ignored suite stays green. + +### Run 13: baseline - template_score=90.909 (KEEP) +- Timestamp: 2026-03-26 18:32 +- What changed: Expanded the ignored template suite with Moonshot/Kimi variant coverage, including one positive Moonshot alias case and one near-name generic fallback case. +- Result: template_score=90.909, passed_cases=10, failed_cases=1, total_cases=11. +- Insight: The only remaining gap is that `select_chat_template()` recognizes `kimi` but not the common `moonshot` naming variant; nearby names like `moonshine` already fall back safely. +- Next: extend family detection so Moonshot-branded aliases map to the Kimi template without broadening the match so far that unrelated names stop using the generic fallback. + +### Run 14: detect moonshot as kimi family - template_score=100.0 (KEEP) +- Timestamp: 2026-03-26 18:33 +- What changed: Extended `select_chat_template()` in `crates/inference-server/src/compat.rs` so candidate names containing `moonshot` are treated as the Kimi family alongside direct `kimi` matches. +- Result: template_score=100.0, passed_cases=11, failed_cases=0, total_cases=11. +- Insight: A narrow Moonshot variant match closes the Kimi-family naming gap cleanly without breaking safe generic fallback for unrelated names such as `moonshine-local`. +- Next: raise the benchmark around multi-turn continuity, tool-role rendering, or family-specific response-format instructions.