ForgeLM/experiments/worklog.md

Autoresearch Worklog: latency-p95-chat-completions

Session Summary

• Objective: reduce end-to-end p95 latency for local POST /v1/chat/completions
• Primary metric: p95_ms (lower is better)
• Secondary metrics: p50_ms, mean_ms, completion_tps_mean, success_count
• Benchmark harness: ./autoresearch.sh

Key Insights

• The first stable end-to-end benchmark is around 0.94 s p95 for a short non-streaming coding request on the local 4B GGUF model.
• The benchmark harness must use the canonical model id derived from the GGUF filename; the current server build does not actually honor the intended local-model alias path yet.
• Prompt prefill is only about 25 ms for this workload, so almost all latency sits in token generation rather than request parsing or prompt construction.
• A micro-optimization in per-token detokenization made both latency and throughput worse, which suggests the dominant cost is deeper in the generation/runtime path rather than heap allocation in Rust glue code.
• Shrinking the Turbo4B context and batch sizes also hurt throughput, so the current 4B profile appears closer to the local optimum for this hardware/workload than expected.
• Disabling internal llama.cpp perf counters produced the first real p95 improvement over baseline, which suggests a small but measurable amount of hot-path CPU overhead was coming from instrumentation rather than model math.
• Better stop behavior improved average and median latency, but it still failed to beat baseline p95, so the worst outliers are not explained solely by turn-boundary overruns.
• Switching to a Qwen-style chat template produced a severe p95 tail regression while median latency stayed near baseline, which strongly suggests worse stop behavior or occasional runaway generations rather than uniformly slower decoding.

Next Ideas

• Measure baseline first, then inspect the hottest latency contributors in prompt preparation, request handling, and session creation.
• Investigate why ModelCatalog rejects local-model despite intending to expose aliases.

Run 1: baseline - p95_ms=941.252 (KEEP)

• Timestamp: 2026-03-26 17:32
• What changed: Established the autoresearch harness, built the release server, started it against the local Qwen GGUF, and measured a fixed non-streaming coding request workload.
• Result: p95_ms=941.252, p50_ms=908.457, mean_ms=917.123, completion_tps_mean=52.427, success_count=7.
• Insight: The baseline is stable enough to compare future server/runtime changes, and the benchmark immediately exposed a compatibility bug in alias resolution that should be kept separate from true latency experiments.
• Next: fix low-risk benchmark-adjacent correctness issues only if they distort measurement; otherwise inspect hot latency contributors in the request/session path.

Run 2: stack buffer detokenization - p95_ms=1032.315 (DISCARD)

• Timestamp: 2026-03-26 17:38
• What changed: Replaced the per-token heap allocation in decode_token_piece_with() with a 256-byte stack buffer and heap fallback in crates/inference-core/src/session.rs.
• Result: p95_ms=1032.315, p50_ms=1020.402, mean_ms=1011.466, completion_tps_mean=47.671, success_count=7.
• Insight: This path is not the practical bottleneck for end-to-end latency here; generation got slower overall despite the reduced allocation churn.
• Next: discard this branch state and focus on runtime/profile-level experiments that can improve generation throughput directly.

Run 3: slim turbo4b runtime profile - p95_ms=1004.435 (DISCARD)

• Timestamp: 2026-03-26 17:40
• What changed: Reduced Turbo4B defaults in crates/inference-core/src/profile.rs from n_ctx=4096, n_batch=512, n_ubatch=128 to n_ctx=2048, n_batch=256, n_ubatch=64.
• Result: p95_ms=1004.435, p50_ms=998.919, mean_ms=990.771, completion_tps_mean=48.528, success_count=7.
• Insight: The leaner profile still reduced effective generation speed on this RTX 3070 + Q6_K workload, so the current batch/context sizing is likely not the source of the baseline latency.
• Next: revert to the best-known commit and target generation semantics or model-facing prompt/template strategy rather than detokenization or profile shrinkage.

Run 6: disable llama perf counters in hot path - p95_ms=925.587 (KEEP)

• Timestamp: 2026-03-26 17:57
• What changed: Set llama_context_params.no_perf = true in crates/inference-core/src/engine.rs and sampler_chain_default_params.no_perf = true in crates/inference-core/src/session.rs.
• Result: p95_ms=925.587, p50_ms=893.913, mean_ms=899.895, completion_tps_mean=53.618, success_count=7.
• Insight: Disabling internal perf bookkeeping shaved about 1.7% off baseline p95 with slightly better overall throughput, making this the first experiment worth keeping.
• Next: continue from this commit and look for adjacent hot-path flags or runtime settings that reduce generation overhead without changing model behavior.

Run 5: default turn-boundary stops - p95_ms=965.663 (DISCARD)

• Timestamp: 2026-03-26 17:54
• What changed: Added rolling multi-token stop detection in crates/inference-core/src/session.rs and injected default turn-boundary stop sequences in crates/inference-server/src/compat.rs when the request did not specify stop.
• Result: p95_ms=965.663, p50_ms=877.030, mean_ms=897.251, completion_tps_mean=53.686, success_count=7.
• Insight: This reduced median and mean latency and slightly improved throughput, but the p95 still stayed above baseline, so tail latency likely comes from a smaller number of deeper generation stalls rather than just missed stop boundaries.
• Next: inspect per-sample latency spread and target the remaining long-tail outliers directly, likely in warmup/first-request behavior or occasional decode stalls.

Run 4: qwen-style chat template - p95_ms=3027.145 (DISCARD)

• Timestamp: 2026-03-26 17:50
• What changed: Replaced the plain System/User/Assistant: prompt format in crates/inference-server/src/compat.rs with a Qwen-style <|im_start|>...<|im_end|> chat template.
• Result: p95_ms=3027.145, p50_ms=974.937, mean_ms=1467.026, completion_tps_mean=41.360, success_count=7.
• Insight: The template did not make typical requests much faster and introduced severe tail latency, likely because some generations stopped much later or followed a worse decoding path for this server/model combination.
• Next: revert this template change and focus on experiments that directly bound generation length or improve stop behavior without overhauling the prompt format.

Segment 1: Dynamic Chat Templates

• New objective: add dynamic prompt templates by model family with built-in coverage for Qwen, Gemma, Llama, Kimi, and a generic fallback.
• Benchmark driver: ./autoresearch.sh now runs ignored template-specific tests and reports template_score, passed_cases, failed_cases, and total_cases.
• Keep/discard rule for this segment is test-driven: higher template_score wins, while the regular non-ignored test suite must remain green.
• A minimal family detector plus embedded template registry in compat.rs was enough to move coverage from 1/5 to 5/5 without touching the runtime.

Run 7: baseline - template_score=20.0 (KEEP)

• Timestamp: 2026-03-26 18:14
• What changed: Re-initialized autoresearch to a new segment focused on dynamic chat templates, added a dedicated ignored test suite for Qwen, Gemma, Llama, Kimi, and generic fallback behavior, and switched the benchmark harness to score that suite.
• Result: template_score=20.0, passed_cases=1, failed_cases=4, total_cases=5.
• Insight: Only the generic fallback works today; all named family-template expectations fail because prompt rendering is still generic and model-agnostic.
• Next: add a minimal model-family detector and template registry in crates/inference-server/src/compat.rs, then route prepare_prompt() through it.

Run 8: add family-based chat templates - template_score=100.0 (KEEP)

• Timestamp: 2026-03-26 18:16
• What changed: Added model-family detection and template-aware prompt rendering in crates/inference-server/src/compat.rs for Qwen, Gemma, Llama, Kimi, and generic fallback, then routed system/tool/JSON-mode instructions and assistant prefixes through the selected template.
• Result: template_score=100.0, passed_cases=5, failed_cases=0, total_cases=5.
• Insight: The current seam in prepare_prompt() is sufficient for broad family-based template support; no runtime or routing changes were needed for the first successful implementation.
• Next: harden the design by adding alias-driven family detection, safer fallback behavior for mixed-case/variant model names, and more coverage for tool-call and response-format rendering under non-generic templates.

Segment 2: Dynamic Chat Template Hardening

• New objective: keep family-based template selection, but harden structure-sensitive formatting for instruct-style models, starting with Llama.
• Benchmark driver: ./autoresearch.sh still runs the ignored template suite, now with stricter Llama assertions.
• Keep/discard rule remains test-driven: higher template_score wins while the regular non-ignored suite stays green.

Run 9: baseline - template_score=71.429 (KEEP)

• Timestamp: 2026-03-26 18:21
• What changed: Expanded the ignored benchmark suite with two stronger Llama assertions that require system+user to be merged into the first instruction block and forbid leaving an open [INST] prefix at generation time.
• Result: template_score=71.429, passed_cases=5, failed_cases=2, total_cases=7.
• Insight: The current family detector is good enough, but the Llama renderer is still structurally naive; it over-produces instruction blocks and leaves the prompt in an invalid generation state.
• Next: refactor Llama rendering into a dedicated formatter that folds system context into the first user turn and emits a proper assistant generation prefix.

Run 10: harden llama chat template rendering - template_score=100.0 (KEEP)

• Timestamp: 2026-03-26 18:23
• What changed: Reworked the Llama path in crates/inference-server/src/compat.rs into a dedicated renderer that accumulates pending system/developer content, folds it into the next user/tool instruction block, and stops emitting a dangling open assistant [INST] prefix.
• Result: template_score=100.0, passed_cases=7, failed_cases=0, total_cases=7.
• Insight: Llama needed a specialized formatter instead of the generic per-message helper; once the prompt was rendered as proper closed instruction blocks with folded system context, the hardening suite passed cleanly.
• Next: raise the benchmark again around multi-turn continuity, tool-role placement, or alias-driven family detection so the next experiments target real edge cases instead of already-solved happy paths.

Segment 3: Alias-Driven Template Selection

• New objective: harden family detection so template selection can still infer the right family from an alias even when the canonical/root model id is generic.
• Benchmark driver: ./autoresearch.sh still runs the ignored template suite, now expanded with alias-sensitive cases.
• Keep/discard rule remains test-driven: higher template_score wins while the regular non-ignored suite stays green.

Run 11: baseline - template_score=88.889 (KEEP)

• Timestamp: 2026-03-26 18:27
• What changed: Expanded the ignored benchmark suite with alias-sensitive family-selection coverage, especially a case where the canonical model id is generic but the requested alias carries the qwen family hint.
• Result: template_score=88.889, passed_cases=8, failed_cases=1, total_cases=9.
• Insight: The family templates and fallback behavior are otherwise stable; the remaining gap is specifically that select_chat_template() loses the alias hint when it resolves through a generic canonical/root id.
• Next: make template selection consider both the requested model string and the resolved catalog/root metadata, preferring whichever one contains a recognized family marker.

Run 12: detect template family from aliases - template_score=100.0 (KEEP)

• Timestamp: 2026-03-26 18:28
• What changed: Updated select_chat_template() in crates/inference-server/src/compat.rs to evaluate the requested model name first, then resolved alias/canonical/root ids, so family hints present only in aliases still drive template selection.
• Result: template_score=100.0, passed_cases=9, failed_cases=0, total_cases=9.
• Insight: Alias-aware selection closes the last known family-detection gap without needing extra catalog metadata; simple ordered candidate matching is enough here.
• Next: raise the benchmark again around multi-turn continuity, tool-call/tool-role rendering, or family-specific response-format instructions to target subtler correctness gaps.

Segment 4: Variant Family Detection

• New objective: harden family detection for common model-name variants, starting with Moonshot-branded Kimi aliases.
• Benchmark driver: ./autoresearch.sh still runs the ignored template suite, now expanded with variant-sensitive cases.
• Keep/discard rule remains test-driven: higher template_score wins while the regular non-ignored suite stays green.

Run 13: baseline - template_score=90.909 (KEEP)

• Timestamp: 2026-03-26 18:32
• What changed: Expanded the ignored template suite with Moonshot/Kimi variant coverage, including one positive Moonshot alias case and one near-name generic fallback case.
• Result: template_score=90.909, passed_cases=10, failed_cases=1, total_cases=11.
• Insight: The only remaining gap is that select_chat_template() recognizes kimi but not the common moonshot naming variant; nearby names like moonshine already fall back safely.
• Next: extend family detection so Moonshot-branded aliases map to the Kimi template without broadening the match so far that unrelated names stop using the generic fallback.

Run 14: detect moonshot as kimi family - template_score=100.0 (KEEP)

• Timestamp: 2026-03-26 18:33
• What changed: Extended select_chat_template() in crates/inference-server/src/compat.rs so candidate names containing moonshot are treated as the Kimi family alongside direct kimi matches.
• Result: template_score=100.0, passed_cases=11, failed_cases=0, total_cases=11.
• Insight: A narrow Moonshot variant match closes the Kimi-family naming gap cleanly without breaking safe generic fallback for unrelated names such as moonshine-local.
• Next: raise the benchmark around multi-turn continuity, tool-role rendering, or family-specific response-format instructions.

Segment 5: CLI Control Surface

• New objective: build a complete management CLI first, ahead of the Tauri app, on top of the current backend version.
• Benchmark driver: ./autoresearch.sh now runs ignored CLI-surface tests and reports cli_surface_score, passed_cases, failed_cases, and total_cases.
• Keep/discard rule is test-driven: higher cli_surface_score wins while the regular non-ignored suite remains green.
• This segment starts by benchmarking the command surface itself before deeper execution and backend-admin semantics are implemented.

Run 15: baseline - cli_surface_score=18.182 (KEEP)

• Timestamp: 2026-03-26 18:53
• What changed: Initialized a new crates/localllm-cli crate with a minimal Clap-based CLI scaffold, switched the autoresearch harness from template tests to CLI surface tests, and defined the first ignored benchmark suite for professional command coverage.
• Result: cli_surface_score=18.182, passed_cases=2, failed_cases=9, total_cases=11.
• Insight: The scaffold already covers global flags plus status and models list, but almost the entire promised management surface is still missing.
• Next: add the remaining top-level command groups and model/auth/config/bench/logs/doctor subcommands so the CLI surface matches the intended product contract before wiring execution internals.

Run 16: expand cli command surface - cli_surface_score=100.0 (KEEP)

• Timestamp: 2026-03-26 18:55
• What changed: Expanded crates/localllm-cli/src/main.rs into a broad Clap parser with top-level status, models, auth, config, bench, logs, and doctor commands plus the expected model/auth/config/throughput subcommands and global flags.
• Result: cli_surface_score=100.0, passed_cases=11, failed_cases=0, total_cases=11.
• Insight: The professional CLI contract can be locked in quickly at the parser layer before execution internals exist; Clap gives a maintainable foundation for the full management surface.
• Next: raise the benchmark from command presence to semantic correctness, starting with command-specific arguments, defaults, and output-contract expectations for the highest-value workflows.

Segment 6: CLI Semantic Contracts

• New objective: keep the CLI surface stable, but raise the benchmark so key commands preserve useful semantic detail such as defaults, high-value arguments, and summary/intention contracts.
• Benchmark driver: ./autoresearch.sh still runs ignored localllm-cli tests and reports cli_surface_score, passed_cases, failed_cases, and total_cases.
• Keep/discard rule remains test-driven: higher cli_surface_score wins while the normal non-ignored suites remain green.

Segment 7: CLI Execution Plans

• New objective: raise the CLI benchmark from human-readable semantic summaries to machine-usable execution plans for key workflows.
• Benchmark driver: ./autoresearch.sh still runs ignored localllm-cli tests and reports cli_surface_score, passed_cases, failed_cases, and total_cases.
• Keep/discard rule remains test-driven: higher cli_surface_score wins while the normal non-ignored suites remain green.

Run 19: baseline - cli_surface_score=69.565 (KEEP)

• Timestamp: 2026-03-26 19:04
• What changed: Expanded the CLI benchmark with plan-oriented ignored tests that require key commands to map into machine-usable CommandPlan variants instead of only semantic summaries.
• Result: cli_surface_score=69.565, passed_cases=16, failed_cases=7, total_cases=23.
• Insight: The semantic summary layer is already solid, but plan_command() is still a stub that collapses everything to SummaryOnly, so execution planning has not been wired yet.
• Next: implement concrete CommandPlan mapping for status, models scan/load, auth enable, config set, bench throughput, and logs.

Run 20: add cli execution plans - cli_surface_score=100.0 (KEEP)

• Timestamp: 2026-03-26 19:05
• What changed: Implemented concrete CommandPlan mapping in crates/localllm-cli/src/main.rs for status, model scan/load, auth enable, config set, throughput bench, and logs while leaving lower-priority commands on SummaryOnly.
• Result: cli_surface_score=100.0, passed_cases=23, failed_cases=0, total_cases=23.
• Insight: The CLI now has a machine-usable planning layer for the highest-value workflows, which is a stronger base for later real handlers and Tauri reuse than summary strings alone.
• Next: raise the benchmark from structured plans to execution-ready request payloads or output contracts at the backend/control-plane seam.

Segment 8: CLI Execution Contracts

• New objective: turn the highest-value CommandPlan variants into concrete execution contracts that describe the exact HTTP or local operation each workflow intends to perform.
• Benchmark driver: ./autoresearch.sh still runs ignored localllm-cli tests and reports cli_surface_score, passed_cases, failed_cases, and total_cases.
• Keep/discard rule remains test-driven: higher cli_surface_score wins while the normal non-ignored suites remain green.

Run 21: baseline - cli_surface_score=76.667 (KEEP)

• Timestamp: 2026-03-26 19:11
• What changed: Expanded the benchmark from CommandPlan semantics to ExecutionContract semantics, requiring the CLI to describe the concrete HTTP or local operation each high-value workflow intends to perform.
• Result: cli_surface_score=76.667, passed_cases=23, failed_cases=7, total_cases=30.
• Insight: The planning layer is now solid, but execution_contract() is still stubbed and returns SummaryOnly { command: "unplanned" } for every concrete plan, so the CLI cannot yet describe its backend/control-plane seam accurately.
• Next: map the high-value CommandPlan variants into concrete HTTP/local execution contracts for status, models scan/load, auth enable, config set, throughput bench, and logs.

Run 22: add cli execution contracts - cli_surface_score=100.0 (KEEP)

• Timestamp: 2026-03-26 19:14
• What changed: Implemented concrete execution_contract() mapping in crates/localllm-cli/src/main.rs, turning the highest-value CommandPlan variants into explicit HTTP or local contracts for health probing, model scan/load, auth/config mutation, throughput requests, and log reads.
• Result: cli_surface_score=100.0, passed_cases=30, failed_cases=0, total_cases=30.
• Insight: The CLI now has a clean progression from parser -> semantic summary -> machine-usable plan -> concrete execution contract for the highest-value workflows, which is a strong base for wiring real handlers next.
• Next: raise the benchmark from contract shape to actual execution stubs or backend request payload semantics, especially for status, throughput bench, and model operations.

Run 17: baseline - cli_surface_score=68.75 (KEEP)

• Timestamp: 2026-03-26 18:59
• What changed: Raised the benchmark with summary-focused ignored tests that require render_command_summary() to preserve output format, models directory, command-specific arguments, and key defaults for status, models load, auth enable, bench throughput, and logs.
• Result: cli_surface_score=68.75, passed_cases=11, failed_cases=5, total_cases=16.
• Insight: The parser surface is solid, but the current summary layer is too shallow; it only reports the top-level command and server, losing the semantic detail needed for later execution planning.
• Next: upgrade render_command_summary() so it emits nested command paths and the highest-value arguments/defaults for each management workflow.

Run 18: enrich cli semantic summaries - cli_surface_score=100.0 (KEEP)

• Timestamp: 2026-03-26 19:00
• What changed: Reworked render_command_summary() in crates/localllm-cli/src/main.rs to emit nested command paths plus key defaults and high-value arguments for status, models, auth, config, bench, and logs workflows.
• Result: cli_surface_score=100.0, passed_cases=16, failed_cases=0, total_cases=16.
• Insight: A lightweight semantic/intention layer is enough to make the CLI contract meaningfully richer before wiring real handlers; the CLI now preserves actionable detail instead of collapsing everything to top-level groups.
• Next: raise the benchmark again toward execution planning or output contracts, e.g. structured action plans for backend/admin operations rather than just human-readable summaries.

Segment 9: CLI Execution Stubs

• New objective: raise the CLI benchmark from abstract execution contracts to concrete execution stubs that preserve HTTP method/path/body semantics or named local actions for the highest-value workflows.
• Benchmark driver: ./autoresearch.sh still runs ignored localllm-cli tests and reports cli_surface_score, passed_cases, failed_cases, and total_cases.
• Keep/discard rule remains test-driven: higher cli_surface_score wins while the normal non-ignored suites remain green.

Run 23: baseline - cli_surface_score=83.333 (KEEP)

• Timestamp: 2026-03-26 19:20
• What changed: Expanded the benchmark from execution contracts to concrete execution stubs, adding ignored tests for HTTP method/path/body semantics and named local actions for status, model scan/load, auth enable, throughput bench, and logs.
• Result: cli_surface_score=83.333, passed_cases=30, failed_cases=6, total_cases=36.
• Insight: The CLI already knows how to parse commands and describe execution contracts, but execution_stub() is still just a placeholder returning SummaryOnly { command: "unimplemented" } for every high-value workflow.
• Next: implement concrete stub mapping so status and throughput produce execution-ready HTTP request shapes and local workflows emit named local actions with preserved detail.

Run 24: add cli execution stubs - cli_surface_score=100.0 (KEEP)

• Timestamp: 2026-03-26 19:22
• What changed: Implemented concrete execution_stub() mapping in crates/localllm-cli/src/main.rs, turning the highest-value execution contracts into execution-ready HTTP requests and named local actions with preserved detail.
• Result: cli_surface_score=100.0, passed_cases=36, failed_cases=0, total_cases=36.
• Insight: The CLI now has a full design ladder for the key workflows - parser, semantic summary, machine-usable plan, execution contract, and finally execution stub - which is a strong handoff point for wiring real handlers next.
• Next: raise the benchmark from stub shape to actual handler behavior or request payload construction for the same high-value workflows.

Segment 10: CLI Execution Previews

• New objective: raise the CLI benchmark from execution stubs to dry-run previews that expand the same high-value workflows into full previewable HTTP URLs or local-effect descriptions.
• Benchmark driver: ./autoresearch.sh still runs ignored localllm-cli tests and reports cli_surface_score, passed_cases, failed_cases, and total_cases.
• Keep/discard rule remains test-driven: higher cli_surface_score wins while the normal non-ignored suites remain green.

Run 25: baseline - cli_surface_score=85.714 (KEEP)

• Timestamp: 2026-03-26 19:27
• What changed: Expanded the benchmark from execution stubs to dry-run previews, adding ignored tests for full preview URLs and human-readable local-effect descriptions across status, models scan/load, auth enable, throughput bench, and logs.
• Result: cli_surface_score=85.714, passed_cases=36, failed_cases=6, total_cases=42.
• Insight: The CLI already models execution stubs correctly, but execution_preview() still collapses every high-value workflow to SummaryOnly { command: "preview-unimplemented" }, so there is no user-facing dry-run layer yet.
• Next: map the highest-value execution stubs into concrete HttpPreview and LocalPreview values using the resolved --server base URL and readable local-effect text.

Run 26: add cli execution previews - cli_surface_score=100.0 (KEEP)

• Timestamp: 2026-03-26 19:28
• What changed: Implemented concrete execution_preview() mapping in crates/localllm-cli/src/main.rs, expanding HTTP stubs into full URLs with preserved payloads and local actions into readable effect descriptions via preview_effect().
• Result: cli_surface_score=100.0, passed_cases=42, failed_cases=0, total_cases=42.
• Insight: The CLI now has a complete dry-run preview layer for the highest-value workflows, which is a strong bridge from internal planning seams to future real handlers and --dry-run UX.
• Next: raise the benchmark from preview shape to actual handler behavior or request-body construction details for status, throughput, and model-management workflows.

Segment 11: CLI Dry-Run Output

• New objective: raise the CLI benchmark from internal preview objects to final user-facing --dry-run output in both text and json, so the same high-value workflows are ready for a real preview UX at the command line.
• Benchmark driver: ./autoresearch.sh still runs ignored localllm-cli tests and reports cli_surface_score, passed_cases, failed_cases, and total_cases.
• Keep/discard rule remains test-driven: higher cli_surface_score wins while the normal non-ignored suites remain green.

Run 27: baseline - cli_surface_score=87.5 (KEEP)

• Timestamp: 2026-03-26 19:32
• What changed: Expanded the benchmark from internal ExecutionPreview objects to final --dry-run UX in both text and JSON, while keeping render_execution_preview() intentionally shallow as DRY RUN {summary}.
• Result: cli_surface_score=87.5, passed_cases=42, failed_cases=6, total_cases=48.
• Insight: The CLI already knows how to build the preview object graph, but the final presentation layer still collapses to summaries instead of rendering health/chat HTTP previews and local-effect dry-run text.
• Next: render the existing preview layer into final text/json output using the current ExecutionPreview data rather than introducing a new execution seam.

Run 28: render cli dry-run previews - cli_surface_score=100.0 (KEEP)

• Timestamp: 2026-03-26 19:34
• What changed: Implemented final --dry-run rendering in crates/localllm-cli/src/main.rs, mapping ExecutionPreview values to user-facing text and JSON output and reusing the existing preview pipeline (plan_command -> execution_contract -> execution_stub -> execution_preview).
• Result: cli_surface_score=100.0, passed_cases=48, failed_cases=0, total_cases=48.
• Insight: The CLI now has a complete dry-run stack for the highest-value workflows, from parser through final user-visible text/json previews, without yet executing real side effects.
• Next: raise the bar from dry-run rendering to real handler behavior or deterministic request/body construction for backend-facing commands.

Segment 12: CLI Backend Invocations

• New objective: raise the CLI benchmark from dry-run rendering to the first real backend-call specifications for commands that can already work today, especially status, models list, and bench throughput.
• Benchmark driver: ./autoresearch.sh still runs ignored localllm-cli tests and reports cli_surface_score, passed_cases, failed_cases, and total_cases.
• Keep/discard rule remains test-driven: higher cli_surface_score wins while the normal non-ignored suites remain green.

Segment 13: CLI Backend Requests

• New objective: raise the CLI benchmark from backend invocation specs to executable backend request construction for the commands that already align with live server endpoints today.
• Benchmark driver: ./autoresearch.sh still runs ignored localllm-cli tests and reports cli_surface_score, passed_cases, failed_cases, and total_cases.
• Keep/discard rule remains test-driven: higher cli_surface_score wins while the normal non-ignored suites remain green.

Run 29: baseline - cli_surface_score=94.118 (KEEP)

• Timestamp: 2026-03-26 19:39
• What changed: Expanded the benchmark from final dry-run rendering to backend invocation specs for commands that can already work today, adding ignored tests for live GET /health, live GET /v1/models, and a throughput POST /v1/chat/completions request that preserves the selected model and token budget.
• Result: cli_surface_score=94.118, passed_cases=48, failed_cases=3, total_cases=51.
• Insight: The CLI already handled every previous parser/planning/dry-run layer, but backend_invocation() was still a placeholder, and models list had not yet been promoted into the real backend-facing seam.
• Next: implement concrete backend invocation specs for status, models list, and bench throughput using the resolved server base URL and auth presence.

Run 30: add cli backend invocation specs - cli_surface_score=100.0 (KEEP)

• Timestamp: 2026-03-26 19:41
• What changed: Implemented backend_invocation(&Cli) in crates/localllm-cli/src/main.rs so status maps to live GET /health, models list maps to live GET /v1/models, and bench throughput maps to a live POST /v1/chat/completions invocation that preserves --model, samples, token budget, and auth presence.
• Result: cli_surface_score=100.0, passed_cases=51, failed_cases=0, total_cases=51.
• Insight: The CLI now crosses from dry-run-only UX into concrete backend-facing invocation specs for commands that can already work against the current server today, without needing the future control-plane endpoints yet.
• Next: raise the bar from backend invocation specs to actual request payload construction or the first executable handlers for these already-supported commands.

Run 31: baseline - cli_surface_score=94.444 (KEEP)

• Timestamp: 2026-03-26 19:47
• What changed: Expanded the benchmark from backend invocation specs to executable backend request construction, requiring final HTTP method/url/header/body request objects with deterministic JSON payloads for status, models list, and bench throughput.
• Result: cli_surface_score=94.444, passed_cases=51, failed_cases=3, total_cases=54.
• Insight: The CLI already had parser, summaries, plans, contracts, stubs, previews, dry-run rendering, and invocation specs, but backend_request(&Cli) still returned SummaryOnly { command: "request-unimplemented" } instead of final request objects.
• Next: map BackendInvocation::HttpCall into executable BackendRequest::HttpRequest values with bearer headers, JSON content-type, and deterministic benchmark payload generation.

Run 32: add cli backend request specs - cli_surface_score=100.0 (KEEP)

• Timestamp: 2026-03-26 19:49
• What changed: Implemented backend_request(&Cli) in crates/localllm-cli/src/main.rs plus helpers for bearer headers, JSON headers, deterministic benchmark prompt construction, and exact chat payload generation for throughput requests.
• Result: cli_surface_score=100.0, passed_cases=54, failed_cases=0, total_cases=54.
• Insight: The first live-compatible CLI workflows now preserve the full executable backend request boundary - method, URL, auth headers, content-type, and deterministic JSON body - while still remaining execution-free.
• Next: raise the bar from request construction to the first live response-handling or executable handler seams for the same workflows.

Segment 14: CLI Backend Responses

• New objective: raise the CLI benchmark from executable backend request construction to backend response interpretation for the same live-compatible workflows, starting with status, models list, and bench throughput.
• Benchmark driver: ./autoresearch.sh still runs ignored localllm-cli tests and reports cli_surface_score, passed_cases, failed_cases, and total_cases.
• Keep/discard rule remains test-driven: higher cli_surface_score wins while the normal non-ignored suites remain green.

Run 33: baseline - cli_surface_score=94.737 (KEEP)

• Timestamp: 2026-03-26 20:01
• What changed: Added the segment 14 response-preview seam and local deserializable DTOs for health, models list, and throughput usage responses, but left backend_response_preview() intentionally unimplemented for the new live-compatible workflows.
• Result: cli_surface_score=94.737, passed_cases=54, failed_cases=3, total_cases=57.
• Insight: The CLI already had every layer through executable backend request construction, but it still collapsed real response parsing for status, models list, and bench throughput to SummaryOnly { command: "response-unimplemented" }.
• Next: parse the real response bodies into CLI-local previews for health status, model ids/count, and throughput usage metrics.

Run 34: add cli backend response previews - cli_surface_score=100.0 (KEEP)

• Timestamp: 2026-03-26 20:02
• What changed: Implemented backend_response_preview() in crates/localllm-cli/src/main.rs so status parses health payloads, models list collects model ids/count, and bench throughput reads usage.total_tokens plus usage.tokens_per_second into CLI-local preview values.
• Result: cli_surface_score=100.0, passed_cases=57, failed_cases=0, total_cases=57.
• Insight: The first live-compatible CLI workflows now preserve the full response side of the contract too: request construction plus deterministic response interpretation without forcing reuse of server-only DTOs.
• Next: raise the bar from response previews to the first real executable handlers or user-facing live output rendering for commands that already work against the current backend.

Segment 15: CLI Live Output

• New objective: raise the CLI benchmark from internal backend response previews to final user-facing live output rendering for the already live-compatible workflows status, models list, and bench throughput.
• Benchmark driver: ./autoresearch.sh still runs ignored localllm-cli tests and reports cli_surface_score, passed_cases, failed_cases, and total_cases.
• Keep/discard rule remains test-driven: higher cli_surface_score wins while the normal non-ignored suites remain green.

Run 35: baseline - cli_surface_score=90.476 (KEEP)

• Timestamp: 2026-03-26 20:06
• What changed: Expanded the benchmark from backend response previews to final user-facing live output rendering, but left render_backend_response() as a shallow LIVE {summary} placeholder.
• Result: cli_surface_score=90.476, passed_cases=57, failed_cases=6, total_cases=63.
• Insight: The CLI already parsed health, model-list, and throughput responses correctly, but it still lacked final text/json presentation for those live-compatible workflows.
• Next: render BackendResponsePreview into final STATUS, MODELS, and THROUGHPUT text/json outputs.

Run 36: render cli live outputs - cli_surface_score=100.0 (KEEP)

• Timestamp: 2026-03-26 20:07
• What changed: Implemented render_backend_response() in crates/localllm-cli/src/main.rs so the live-compatible commands now render final text/json output directly from BackendResponsePreview, including health summaries, model-id lists, and throughput usage output.
• Result: cli_surface_score=100.0, passed_cases=63, failed_cases=0, total_cases=63.
• Insight: The CLI now closes the full loop for the first live-compatible workflows, from parser and planning all the way to final user-facing live output.
• Next: move from live output rendering to the first real executable handlers for commands that already work today.

Segment 16: CLI Live Execution

• New objective: raise the CLI benchmark from final live output rendering to the first real executable handlers for the live-compatible workflows status, models list, and bench throughput using an injectable transport seam.
• Benchmark driver: ./autoresearch.sh still runs ignored localllm-cli tests and reports cli_surface_score, passed_cases, failed_cases, and total_cases.
• Keep/discard rule remains test-driven: higher cli_surface_score wins while the normal non-ignored suites remain green.

Run 37: baseline - cli_surface_score=95.455 (KEEP)

• Timestamp: 2026-03-26 20:12
• What changed: Expanded the benchmark from live output rendering to an injectable live-execution seam, but left execute_live_command_with() as a placeholder that ignored the transport response and returned LIVE EXEC {summary}.
• Result: cli_surface_score=95.455, passed_cases=63, failed_cases=3, total_cases=66.
• Insight: The pipeline already had correct request construction and response rendering, but the executor still dropped the returned status/body instead of feeding them into the live renderer.
• Next: wire the transport response through render_backend_response() without changing any other CLI seam.

Run 38: add cli live execution pipeline - cli_surface_score=100.0 (KEEP)

• Timestamp: 2026-03-26 20:13
• What changed: Implemented execute_live_command_with() in crates/localllm-cli/src/main.rs so live-compatible commands now call the injected transport with BackendRequest, then pass the returned status_code and body into render_backend_response().
• Result: cli_surface_score=100.0, passed_cases=66, failed_cases=0, total_cases=66.
• Insight: The CLI now completes the first fully executable end-to-end path for the live-compatible workflows, from parser to transport invocation to final rendered output.
• Next: move from transport-injected execution to the first concrete HTTP client seam for commands that can already hit the current backend directly.

Segment 17: CLI ureq HTTP Transport

• Objective: replace the injectable fake transport with a concrete ureq HTTP client for status, models list, and bench throughput.
• Benchmark: ./autoresearch.sh runs ignored cli_surface_* tests in crates/localllm-cli/src/main.rs.
• Keep/discard rule: higher cli_surface_score wins while the normal non-ignored suites remain green.

Run 39: add ureq http transport and execute_live_command - cli_surface_score=100.0 (KEEP)

• Timestamp: 2026-03-26 20:36
• What changed: Added ureq to workspace and localllm-cli deps. Implemented execute_live_command(cli) using ureq as the real HTTP transport, delegating to execute_live_command_with(). Updated main() to call execute_live_command() instead of the scaffold placeholder. Added 3 new ignored benchmark tests verifying the injectable seam correctly routes status/models/bench through the transport pipeline.
• Result: cli_surface_score=100.0, passed_cases=69, failed_cases=0, total_cases=69.
• Insight: The CLI now has a real HTTP client (ureq) wired end-to-end for live-compatible commands. The injectable transport seam remains available for testing without a running server. The CLI is functionally complete for the commands that the current backend already supports.
• Next: CLI is at natural saturation given the current backend. Further progress requires new admin/control-plane endpoints in the backend for scan, load, auth, config, and watch commands.

Segment 18: Unified Tauri Runtime Manager

• Objective: replace the split desktop runtime control flow with one saved runtime-manager surface that owns local startup, remote routing, and local model/runtime settings.
• Benchmark: targeted verification bundle using npm test, npm run build, and cargo check plus acceptance checks for a single runtime control plane and updated runtime UI.
• Keep/discard rule: a keep requires one working config/start path, no broken sidebar start button, and green verification across frontend and Rust checks.

Run 40: unify runtime manager config/start flow and refactor runtime UI - runtime_surface_score=100.0 (KEEP)

• Timestamp: 2026-03-26 23:35
• What changed: Removed the active dependency on useServerProcess and the embedded legacy ServerPanel from the sidebar flow, rebuilt RuntimeSettingsDrawer into a full local/remote runtime workspace, added Tauri wrappers for GPU/model picking and scanning, made the sidebar runtime card route setup/start from the same saved runtime state, and refreshed runtime/sidebar/global styling toward a cleaner AI desktop identity.
• Result: runtime_surface_score=100.0, unit_passed=1, build_passed=1, cargo_check_passed=1, unified_control_plane=1, design_refactor_done=1.
• Insight: The runtime failures were caused by state ownership drift, not startup logic alone. Once local runtime configuration and startup lived in the same domain, both runtime entry points aligned and the broken start path disappeared.
• Next: improve readiness probing beyond raw TCP-open detection, then smoke-test the desktop runtime end to end against a real local model and record a stricter follow-up score for startup/readiness behavior.

Run 41: replace tcp-only readiness with /health readiness probe - runtime_surface_score=100.0 (KEEP)

• Timestamp: 2026-03-26 23:43
• What changed: Replaced the runtime startup readiness check in tauri-app/src-tauri/src/runtime/health.rs so it now issues GET /health, requires an HTTP 200 response, and validates {"status":"ok"} before marking the local API as ready. Added Rust tests covering healthy HTTP success, unhealthy JSON payload rejection, and plain-open-port rejection.
• Result: runtime_surface_score=100.0, unit_passed=1, build_passed=1, cargo_check_passed=1, health_probe_passed=1.
• Insight: TCP-open was too weak for a desktop-managed API because any bound socket could masquerade as readiness. A lightweight health request gives the runtime manager a much more truthful startup signal without introducing frontend complexity.
• Next: run an end-to-end desktop smoke test against a real local model so the next segment can score startup behavior from the actual app surface, not just backend verification.

Run 42: add runtime UI smoke coverage and preserve saved model on empty rescans - runtime_surface_score=100.0 (KEEP)

• Timestamp: 2026-03-26 23:50
• What changed: Added jsdom-based runtime smoke tests for RuntimeStatusCard and RuntimeSettingsDrawer, installed jsdom for Vitest browserless component coverage, and fixed chooseModel() in tauri-app/src/features/runtime/components/RuntimeSettingsDrawer.tsx so a previously saved model_path is preserved when a models-folder rescan returns no results.
• Result: runtime_surface_score=100.0, unit_passed=1, build_passed=1, cargo_check_passed=1, runtime_smoke_passed=1.
• Insight: The first useful app-surface smoke test immediately caught a real regression that backend/runtime checks could not see: a valid saved model could disappear from drawer state after an empty scan, which silently disabled the unified start action.
• Next: push the smoke coverage one step further with a focused end-to-end startup path check, ideally validating that a saved local config transitions from setup -> starting -> ready correctly from the desktop surface.

Segment 19: Tauri Runtime Lifecycle Acceptance

• Objective: replace the stale CLI-oriented benchmark with a Tauri desktop lifecycle benchmark and harden the app-surface contract around local/remote mode transitions.
• Benchmark: ./autoresearch.sh now runs experiments/bench_tauri_runtime_surface.py, which scores runtime-focused frontend tests plus npm run build and cargo check.
• Keep/discard rule: higher runtime_lifecycle_score wins while the active desktop runtime CTA and verification bundle remain green.

Run 43: align the Tauri lifecycle harness and keep the local CTA visible during mode switches - runtime_lifecycle_score=100.0 (KEEP)

• Timestamp: 2026-03-27 01:26
• What changed: Replaced the stale CLI benchmark in autoresearch.sh with a new experiments/bench_tauri_runtime_surface.py driver, added a runtime smoke test that switches RuntimeSettingsDrawer from remote to local mode, and fixed the drawer footer in tauri-app/src/features/runtime/components/RuntimeSettingsDrawer.tsx so the local primary CTA is keyed off the edited local mode instead of the previously saved snapshot.config.connection_mode.
• Result: runtime_lifecycle_score=100.0, unit_passed=1, build_passed=1, cargo_check_passed=1, local_mode_cta_passed=1.
• Insight: The previous segment was saturated partly because the harness was no longer measuring the live desktop contract. Once the benchmark targeted the Tauri runtime surface directly, it immediately exposed a real lifecycle regression: switching from remote to local mode could hide the apply/start CTA before save, making the saved local flow feel broken even though the backend state was valid.
• Next: raise the lifecycle score again with a stricter saved-startup path check, ideally covering the stopped -> starting -> ready transition and degraded behavior after a failed health revalidation.

Run 44: cover saved local startup transitions and local onboarding from browse to ready - runtime_lifecycle_score=100.0 (KEEP)

• Timestamp: 2026-03-27 01:40
• What changed: Extended tauri-app/src/features/runtime/components/RuntimeSurface.test.tsx with two hook-backed lifecycle checks that run through useRuntimeManager: one now requires a saved local config to expose Local API offline -> Starting local API -> Ready, and another exercises local onboarding by browsing a models folder, auto-selecting the discovered model, saving config, and starting the local API from the same drawer flow. Updated tauri-app/src/features/runtime/useRuntimeManager.ts to publish optimistic starting and stopping snapshots while the Tauri commands are in flight. Raised experiments/bench_tauri_runtime_surface.py so the benchmark now records local_mode_cta_passed, saved_startup_flow_passed, and local_onboarding_flow_passed as explicit lifecycle checks alongside the broader runtime suite.
• Result: runtime_lifecycle_score=100.0, unit_passed=1, build_passed=1, cargo_check_passed=1, local_mode_cta_passed=1, saved_startup_flow_passed=1, local_onboarding_flow_passed=1.
• Insight: The saved local startup flow was previously trustworthy only after the backend returned; the desktop surface showed Working... but skipped the visible starting state entirely. Moving the in-flight state into useRuntimeManager closes that UX gap and makes the status card/drawer reflect the real lifecycle immediately. The onboarding smoke also proves the unified drawer now covers browse -> auto-select -> save -> start without falling back to separate legacy controls.
• Next: keep the lifecycle segment focused on truthfulness after startup, especially periodic /health revalidation that can demote a stale ready process into degraded when the child stops serving correctly.

Run 45: revalidate managed local health after startup and surface degraded state - runtime_lifecycle_score=100.0 (KEEP)

• Timestamp: 2026-03-27 03:11
• What changed: Added a degraded-state lifecycle check to tauri-app/src/features/runtime/components/RuntimeSurface.test.tsx, extended experiments/bench_tauri_runtime_surface.py with health_revalidation_passed, and updated tauri-app/src-tauri/src/runtime/health.rs plus tauri-app/src-tauri/src/runtime/mod.rs so a running managed local child is periodically revalidated via /health. The runtime now demotes stale ready state to degraded when the process is still alive but no longer serves a healthy endpoint, and clears the health_check_failed error when health recovers. Added Rust tests covering both the degrade and recovery paths.
• Result: runtime_lifecycle_score=100.0, unit_passed=1, build_passed=1, cargo_check_passed=1, local_mode_cta_passed=1, saved_startup_flow_passed=1, local_onboarding_flow_passed=1, health_revalidation_passed=1.
• Insight: The frontend surface was already honest once it received a degraded snapshot; the real lifecycle gap was that backend runtime state could stay stuck at ready forever as long as the child process had not exited. Revalidating /health inside runtime state refresh closes that truthfulness gap without reintroducing a second frontend control plane.
• Next: this lifecycle segment is saturated again at 100.0%. The next worthwhile move should raise the bar beyond state truthfulness, likely around a higher-fidelity desktop integration check or a real managed-runtime smoke path against an actual local model process.

Segment 20: localllm Batch Assert

• Objective: stop the local localllm server from crashing on first chat completion with GGML_ASSERT(n_tokens_all <= cparams.n_batch).
• Benchmark: ./autoresearch.sh now runs a focused prompt-prefill regression harness in experiments/bench_localllm_prefill_batch.py.
• Keep/discard rule: higher prefill_batch_assert_score wins while the targeted prompt-prefill checks, cargo check, and a real LOCALLLM_BATCH=128 completion remain green.

Segment 21: Tauri Chat Active History

• Objective: make a newly sent message create/select its chat immediately, keep the live generation attached to that chat, and preserve history when revisiting generated chats.
• Benchmark: ./autoresearch.sh now runs focused frontend regressions in src/App.test.tsx and src/components/ChatView.test.tsx plus npm run build.
• Keep/discard rule: higher chat_history_score wins while both the app-level send/create/history flow and the stale-request abort guard remain green.

Run 47: preserve in-flight generation when chatId switches to the newly created chat - chat_history_score=100.0 (KEEP)

• Timestamp: 2026-03-28 20:06
• What changed: Added an app-level regression for sending from the empty composer into a brand-new chat, then updated tauri-app/src/components/ChatView.tsx to track the pending request chat id so chatId changes only abort when the user truly switches away to another chat, not when the composer promotes null into the just-created chat.
• Result: chat_history_score=100.0, app_send_flow_passed=1, chat_switch_abort_guard_passed=1, frontend_tests_passed=1, frontend_build_passed=1.
• Insight: The previous saved-chat race fix was directionally right but too broad. null -> newly created chat id is part of the same send flow, not a user navigation away from the active request.
• Next: if more reports appear, raise the bar with a browser/E2E smoke that sends a message, switches chats mid-stream, and returns to confirm persisted history from the rendered app shell.
• Initial hypothesis: crates/inference-core/src/session.rs still uses llama_batch_get_one() for full prompt prefill, which likely requests logits for every prompt token. Newer llama.cpp output-buffer semantics appear to need an explicit batch that only asks for logits on the final prompt token and splits larger prompts by n_batch.
• Baseline status: pending first red run.

Run 46: chunk prompt prefill explicitly and fix live batch metadata - prefill_batch_assert_score=100.0 (KEEP)

• Timestamp: 2026-03-27 15:08
• What changed: Replaced full-prompt prefill in crates/inference-core/src/session.rs with an explicit chunked llama_batch_init() path that only requests logits for the final prompt token and splits prompts by n_batch. Passed n_batch from crates/inference-core/src/engine.rs, bound llama_get_memory / llama_memory_clear in crates/llama-sys/src/lib.rs, and cleared context memory after warmup so the live repro started from a clean state. The decisive live fix was also wiring batch.n_tokens correctly before each llama_decode() call.
• Result: prefill_batch_assert_score=100.0, prefill_plan_passed=1, cargo_check_passed=1, server_build_passed=1, live_completion_passed=1.
• Insight: The original GGML assert came from the old shorthand batch path over-requesting prompt outputs, but the first explicit prefill implementation still failed live because llama_batch_init() leaves n_tokens uninitialized. Once the prefill batch both requested only the final prompt logit and reported a valid token count, the real LOCALLLM_BATCH=128 completion succeeded against the local Qwen GGUF without crashing or returning generation_failed.
• Next: keep this branch state and consider a follow-up hardening segment around longer multi-chunk prompts, multi-sequence compatibility, or a regression harness that distinguishes warmup-only startup success from first-request decode success.