465 lines
53 KiB
Markdown
465 lines
53 KiB
Markdown
# Autoresearch Worklog: latency-p95-chat-completions
|
|
|
|
## Session Summary
|
|
- Objective: reduce end-to-end p95 latency for local `POST /v1/chat/completions`
|
|
- Primary metric: `p95_ms` (lower is better)
|
|
- Secondary metrics: `p50_ms`, `mean_ms`, `completion_tps_mean`, `success_count`
|
|
- Benchmark harness: `./autoresearch.sh`
|
|
|
|
## Key Insights
|
|
- The first stable end-to-end benchmark is around 0.94 s p95 for a short non-streaming coding request on the local 4B GGUF model.
|
|
- The benchmark harness must use the canonical model id derived from the GGUF filename; the current server build does not actually honor the intended `local-model` alias path yet.
|
|
- Prompt prefill is only about 25 ms for this workload, so almost all latency sits in token generation rather than request parsing or prompt construction.
|
|
- A micro-optimization in per-token detokenization made both latency and throughput worse, which suggests the dominant cost is deeper in the generation/runtime path rather than heap allocation in Rust glue code.
|
|
- Shrinking the `Turbo4B` context and batch sizes also hurt throughput, so the current 4B profile appears closer to the local optimum for this hardware/workload than expected.
|
|
- Disabling internal llama.cpp perf counters produced the first real p95 improvement over baseline, which suggests a small but measurable amount of hot-path CPU overhead was coming from instrumentation rather than model math.
|
|
- Better stop behavior improved average and median latency, but it still failed to beat baseline p95, so the worst outliers are not explained solely by turn-boundary overruns.
|
|
- Switching to a Qwen-style chat template produced a severe p95 tail regression while median latency stayed near baseline, which strongly suggests worse stop behavior or occasional runaway generations rather than uniformly slower decoding.
|
|
|
|
## Next Ideas
|
|
- Measure baseline first, then inspect the hottest latency contributors in prompt preparation, request handling, and session creation.
|
|
- Investigate why `ModelCatalog` rejects `local-model` despite intending to expose aliases.
|
|
|
|
### Run 1: baseline - p95_ms=941.252 (KEEP)
|
|
- Timestamp: 2026-03-26 17:32
|
|
- What changed: Established the autoresearch harness, built the release server, started it against the local Qwen GGUF, and measured a fixed non-streaming coding request workload.
|
|
- Result: p95_ms=941.252, p50_ms=908.457, mean_ms=917.123, completion_tps_mean=52.427, success_count=7.
|
|
- Insight: The baseline is stable enough to compare future server/runtime changes, and the benchmark immediately exposed a compatibility bug in alias resolution that should be kept separate from true latency experiments.
|
|
- Next: fix low-risk benchmark-adjacent correctness issues only if they distort measurement; otherwise inspect hot latency contributors in the request/session path.
|
|
|
|
### Run 2: stack buffer detokenization - p95_ms=1032.315 (DISCARD)
|
|
- Timestamp: 2026-03-26 17:38
|
|
- What changed: Replaced the per-token heap allocation in `decode_token_piece_with()` with a 256-byte stack buffer and heap fallback in `crates/inference-core/src/session.rs`.
|
|
- Result: p95_ms=1032.315, p50_ms=1020.402, mean_ms=1011.466, completion_tps_mean=47.671, success_count=7.
|
|
- Insight: This path is not the practical bottleneck for end-to-end latency here; generation got slower overall despite the reduced allocation churn.
|
|
- Next: discard this branch state and focus on runtime/profile-level experiments that can improve generation throughput directly.
|
|
|
|
### Run 3: slim turbo4b runtime profile - p95_ms=1004.435 (DISCARD)
|
|
- Timestamp: 2026-03-26 17:40
|
|
- What changed: Reduced `Turbo4B` defaults in `crates/inference-core/src/profile.rs` from `n_ctx=4096`, `n_batch=512`, `n_ubatch=128` to `n_ctx=2048`, `n_batch=256`, `n_ubatch=64`.
|
|
- Result: p95_ms=1004.435, p50_ms=998.919, mean_ms=990.771, completion_tps_mean=48.528, success_count=7.
|
|
- Insight: The leaner profile still reduced effective generation speed on this RTX 3070 + Q6_K workload, so the current batch/context sizing is likely not the source of the baseline latency.
|
|
- Next: revert to the best-known commit and target generation semantics or model-facing prompt/template strategy rather than detokenization or profile shrinkage.
|
|
|
|
### Run 6: disable llama perf counters in hot path - p95_ms=925.587 (KEEP)
|
|
- Timestamp: 2026-03-26 17:57
|
|
- What changed: Set `llama_context_params.no_perf = true` in `crates/inference-core/src/engine.rs` and `sampler_chain_default_params.no_perf = true` in `crates/inference-core/src/session.rs`.
|
|
- Result: p95_ms=925.587, p50_ms=893.913, mean_ms=899.895, completion_tps_mean=53.618, success_count=7.
|
|
- Insight: Disabling internal perf bookkeeping shaved about 1.7% off baseline p95 with slightly better overall throughput, making this the first experiment worth keeping.
|
|
- Next: continue from this commit and look for adjacent hot-path flags or runtime settings that reduce generation overhead without changing model behavior.
|
|
|
|
### Run 5: default turn-boundary stops - p95_ms=965.663 (DISCARD)
|
|
- Timestamp: 2026-03-26 17:54
|
|
- What changed: Added rolling multi-token stop detection in `crates/inference-core/src/session.rs` and injected default turn-boundary stop sequences in `crates/inference-server/src/compat.rs` when the request did not specify `stop`.
|
|
- Result: p95_ms=965.663, p50_ms=877.030, mean_ms=897.251, completion_tps_mean=53.686, success_count=7.
|
|
- Insight: This reduced median and mean latency and slightly improved throughput, but the p95 still stayed above baseline, so tail latency likely comes from a smaller number of deeper generation stalls rather than just missed stop boundaries.
|
|
- Next: inspect per-sample latency spread and target the remaining long-tail outliers directly, likely in warmup/first-request behavior or occasional decode stalls.
|
|
|
|
### Run 4: qwen-style chat template - p95_ms=3027.145 (DISCARD)
|
|
- Timestamp: 2026-03-26 17:50
|
|
- What changed: Replaced the plain `System/User/Assistant:` prompt format in `crates/inference-server/src/compat.rs` with a Qwen-style `<|im_start|>...<|im_end|>` chat template.
|
|
- Result: p95_ms=3027.145, p50_ms=974.937, mean_ms=1467.026, completion_tps_mean=41.360, success_count=7.
|
|
- Insight: The template did not make typical requests much faster and introduced severe tail latency, likely because some generations stopped much later or followed a worse decoding path for this server/model combination.
|
|
- Next: revert this template change and focus on experiments that directly bound generation length or improve stop behavior without overhauling the prompt format.
|
|
|
|
## Segment 1: Dynamic Chat Templates
|
|
- New objective: add dynamic prompt templates by model family with built-in coverage for Qwen, Gemma, Llama, Kimi, and a generic fallback.
|
|
- Benchmark driver: `./autoresearch.sh` now runs ignored template-specific tests and reports `template_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
|
- Keep/discard rule for this segment is test-driven: higher `template_score` wins, while the regular non-ignored test suite must remain green.
|
|
- A minimal family detector plus embedded template registry in `compat.rs` was enough to move coverage from 1/5 to 5/5 without touching the runtime.
|
|
|
|
### Run 7: baseline - template_score=20.0 (KEEP)
|
|
- Timestamp: 2026-03-26 18:14
|
|
- What changed: Re-initialized autoresearch to a new segment focused on dynamic chat templates, added a dedicated ignored test suite for Qwen, Gemma, Llama, Kimi, and generic fallback behavior, and switched the benchmark harness to score that suite.
|
|
- Result: template_score=20.0, passed_cases=1, failed_cases=4, total_cases=5.
|
|
- Insight: Only the generic fallback works today; all named family-template expectations fail because prompt rendering is still generic and model-agnostic.
|
|
- Next: add a minimal model-family detector and template registry in `crates/inference-server/src/compat.rs`, then route `prepare_prompt()` through it.
|
|
|
|
### Run 8: add family-based chat templates - template_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 18:16
|
|
- What changed: Added model-family detection and template-aware prompt rendering in `crates/inference-server/src/compat.rs` for Qwen, Gemma, Llama, Kimi, and generic fallback, then routed system/tool/JSON-mode instructions and assistant prefixes through the selected template.
|
|
- Result: template_score=100.0, passed_cases=5, failed_cases=0, total_cases=5.
|
|
- Insight: The current seam in `prepare_prompt()` is sufficient for broad family-based template support; no runtime or routing changes were needed for the first successful implementation.
|
|
- Next: harden the design by adding alias-driven family detection, safer fallback behavior for mixed-case/variant model names, and more coverage for tool-call and response-format rendering under non-generic templates.
|
|
|
|
## Segment 2: Dynamic Chat Template Hardening
|
|
- New objective: keep family-based template selection, but harden structure-sensitive formatting for instruct-style models, starting with Llama.
|
|
- Benchmark driver: `./autoresearch.sh` still runs the ignored template suite, now with stricter Llama assertions.
|
|
- Keep/discard rule remains test-driven: higher `template_score` wins while the regular non-ignored suite stays green.
|
|
|
|
### Run 9: baseline - template_score=71.429 (KEEP)
|
|
- Timestamp: 2026-03-26 18:21
|
|
- What changed: Expanded the ignored benchmark suite with two stronger Llama assertions that require system+user to be merged into the first instruction block and forbid leaving an open `[INST]` prefix at generation time.
|
|
- Result: template_score=71.429, passed_cases=5, failed_cases=2, total_cases=7.
|
|
- Insight: The current family detector is good enough, but the Llama renderer is still structurally naive; it over-produces instruction blocks and leaves the prompt in an invalid generation state.
|
|
- Next: refactor Llama rendering into a dedicated formatter that folds system context into the first user turn and emits a proper assistant generation prefix.
|
|
|
|
### Run 10: harden llama chat template rendering - template_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 18:23
|
|
- What changed: Reworked the Llama path in `crates/inference-server/src/compat.rs` into a dedicated renderer that accumulates pending system/developer content, folds it into the next user/tool instruction block, and stops emitting a dangling open assistant `[INST]` prefix.
|
|
- Result: template_score=100.0, passed_cases=7, failed_cases=0, total_cases=7.
|
|
- Insight: Llama needed a specialized formatter instead of the generic per-message helper; once the prompt was rendered as proper closed instruction blocks with folded system context, the hardening suite passed cleanly.
|
|
- Next: raise the benchmark again around multi-turn continuity, tool-role placement, or alias-driven family detection so the next experiments target real edge cases instead of already-solved happy paths.
|
|
|
|
## Segment 3: Alias-Driven Template Selection
|
|
- New objective: harden family detection so template selection can still infer the right family from an alias even when the canonical/root model id is generic.
|
|
- Benchmark driver: `./autoresearch.sh` still runs the ignored template suite, now expanded with alias-sensitive cases.
|
|
- Keep/discard rule remains test-driven: higher `template_score` wins while the regular non-ignored suite stays green.
|
|
|
|
### Run 11: baseline - template_score=88.889 (KEEP)
|
|
- Timestamp: 2026-03-26 18:27
|
|
- What changed: Expanded the ignored benchmark suite with alias-sensitive family-selection coverage, especially a case where the canonical model id is generic but the requested alias carries the `qwen` family hint.
|
|
- Result: template_score=88.889, passed_cases=8, failed_cases=1, total_cases=9.
|
|
- Insight: The family templates and fallback behavior are otherwise stable; the remaining gap is specifically that `select_chat_template()` loses the alias hint when it resolves through a generic canonical/root id.
|
|
- Next: make template selection consider both the requested model string and the resolved catalog/root metadata, preferring whichever one contains a recognized family marker.
|
|
|
|
### Run 12: detect template family from aliases - template_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 18:28
|
|
- What changed: Updated `select_chat_template()` in `crates/inference-server/src/compat.rs` to evaluate the requested model name first, then resolved alias/canonical/root ids, so family hints present only in aliases still drive template selection.
|
|
- Result: template_score=100.0, passed_cases=9, failed_cases=0, total_cases=9.
|
|
- Insight: Alias-aware selection closes the last known family-detection gap without needing extra catalog metadata; simple ordered candidate matching is enough here.
|
|
- Next: raise the benchmark again around multi-turn continuity, tool-call/tool-role rendering, or family-specific response-format instructions to target subtler correctness gaps.
|
|
|
|
## Segment 4: Variant Family Detection
|
|
- New objective: harden family detection for common model-name variants, starting with Moonshot-branded Kimi aliases.
|
|
- Benchmark driver: `./autoresearch.sh` still runs the ignored template suite, now expanded with variant-sensitive cases.
|
|
- Keep/discard rule remains test-driven: higher `template_score` wins while the regular non-ignored suite stays green.
|
|
|
|
### Run 13: baseline - template_score=90.909 (KEEP)
|
|
- Timestamp: 2026-03-26 18:32
|
|
- What changed: Expanded the ignored template suite with Moonshot/Kimi variant coverage, including one positive Moonshot alias case and one near-name generic fallback case.
|
|
- Result: template_score=90.909, passed_cases=10, failed_cases=1, total_cases=11.
|
|
- Insight: The only remaining gap is that `select_chat_template()` recognizes `kimi` but not the common `moonshot` naming variant; nearby names like `moonshine` already fall back safely.
|
|
- Next: extend family detection so Moonshot-branded aliases map to the Kimi template without broadening the match so far that unrelated names stop using the generic fallback.
|
|
|
|
### Run 14: detect moonshot as kimi family - template_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 18:33
|
|
- What changed: Extended `select_chat_template()` in `crates/inference-server/src/compat.rs` so candidate names containing `moonshot` are treated as the Kimi family alongside direct `kimi` matches.
|
|
- Result: template_score=100.0, passed_cases=11, failed_cases=0, total_cases=11.
|
|
- Insight: A narrow Moonshot variant match closes the Kimi-family naming gap cleanly without breaking safe generic fallback for unrelated names such as `moonshine-local`.
|
|
- Next: raise the benchmark around multi-turn continuity, tool-role rendering, or family-specific response-format instructions.
|
|
|
|
## Segment 5: CLI Control Surface
|
|
- New objective: build a complete management CLI first, ahead of the Tauri app, on top of the current backend version.
|
|
- Benchmark driver: `./autoresearch.sh` now runs ignored CLI-surface tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
|
- Keep/discard rule is test-driven: higher `cli_surface_score` wins while the regular non-ignored suite remains green.
|
|
- This segment starts by benchmarking the command surface itself before deeper execution and backend-admin semantics are implemented.
|
|
|
|
### Run 15: baseline - cli_surface_score=18.182 (KEEP)
|
|
- Timestamp: 2026-03-26 18:53
|
|
- What changed: Initialized a new `crates/localllm-cli` crate with a minimal Clap-based CLI scaffold, switched the autoresearch harness from template tests to CLI surface tests, and defined the first ignored benchmark suite for professional command coverage.
|
|
- Result: cli_surface_score=18.182, passed_cases=2, failed_cases=9, total_cases=11.
|
|
- Insight: The scaffold already covers global flags plus `status` and `models list`, but almost the entire promised management surface is still missing.
|
|
- Next: add the remaining top-level command groups and model/auth/config/bench/logs/doctor subcommands so the CLI surface matches the intended product contract before wiring execution internals.
|
|
|
|
### Run 16: expand cli command surface - cli_surface_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 18:55
|
|
- What changed: Expanded `crates/localllm-cli/src/main.rs` into a broad Clap parser with top-level `status`, `models`, `auth`, `config`, `bench`, `logs`, and `doctor` commands plus the expected model/auth/config/throughput subcommands and global flags.
|
|
- Result: cli_surface_score=100.0, passed_cases=11, failed_cases=0, total_cases=11.
|
|
- Insight: The professional CLI contract can be locked in quickly at the parser layer before execution internals exist; Clap gives a maintainable foundation for the full management surface.
|
|
- Next: raise the benchmark from command presence to semantic correctness, starting with command-specific arguments, defaults, and output-contract expectations for the highest-value workflows.
|
|
|
|
## Segment 6: CLI Semantic Contracts
|
|
- New objective: keep the CLI surface stable, but raise the benchmark so key commands preserve useful semantic detail such as defaults, high-value arguments, and summary/intention contracts.
|
|
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
|
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
|
|
|
## Segment 7: CLI Execution Plans
|
|
- New objective: raise the CLI benchmark from human-readable semantic summaries to machine-usable execution plans for key workflows.
|
|
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
|
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
|
|
|
### Run 19: baseline - cli_surface_score=69.565 (KEEP)
|
|
- Timestamp: 2026-03-26 19:04
|
|
- What changed: Expanded the CLI benchmark with plan-oriented ignored tests that require key commands to map into machine-usable `CommandPlan` variants instead of only semantic summaries.
|
|
- Result: cli_surface_score=69.565, passed_cases=16, failed_cases=7, total_cases=23.
|
|
- Insight: The semantic summary layer is already solid, but `plan_command()` is still a stub that collapses everything to `SummaryOnly`, so execution planning has not been wired yet.
|
|
- Next: implement concrete `CommandPlan` mapping for status, models scan/load, auth enable, config set, bench throughput, and logs.
|
|
|
|
### Run 20: add cli execution plans - cli_surface_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 19:05
|
|
- What changed: Implemented concrete `CommandPlan` mapping in `crates/localllm-cli/src/main.rs` for status, model scan/load, auth enable, config set, throughput bench, and logs while leaving lower-priority commands on `SummaryOnly`.
|
|
- Result: cli_surface_score=100.0, passed_cases=23, failed_cases=0, total_cases=23.
|
|
- Insight: The CLI now has a machine-usable planning layer for the highest-value workflows, which is a stronger base for later real handlers and Tauri reuse than summary strings alone.
|
|
- Next: raise the benchmark from structured plans to execution-ready request payloads or output contracts at the backend/control-plane seam.
|
|
|
|
## Segment 8: CLI Execution Contracts
|
|
- New objective: turn the highest-value `CommandPlan` variants into concrete execution contracts that describe the exact HTTP or local operation each workflow intends to perform.
|
|
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
|
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
|
|
|
### Run 21: baseline - cli_surface_score=76.667 (KEEP)
|
|
- Timestamp: 2026-03-26 19:11
|
|
- What changed: Expanded the benchmark from `CommandPlan` semantics to `ExecutionContract` semantics, requiring the CLI to describe the concrete HTTP or local operation each high-value workflow intends to perform.
|
|
- Result: cli_surface_score=76.667, passed_cases=23, failed_cases=7, total_cases=30.
|
|
- Insight: The planning layer is now solid, but `execution_contract()` is still stubbed and returns `SummaryOnly { command: "unplanned" }` for every concrete plan, so the CLI cannot yet describe its backend/control-plane seam accurately.
|
|
- Next: map the high-value `CommandPlan` variants into concrete HTTP/local execution contracts for status, models scan/load, auth enable, config set, throughput bench, and logs.
|
|
|
|
### Run 22: add cli execution contracts - cli_surface_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 19:14
|
|
- What changed: Implemented concrete `execution_contract()` mapping in `crates/localllm-cli/src/main.rs`, turning the highest-value `CommandPlan` variants into explicit HTTP or local contracts for health probing, model scan/load, auth/config mutation, throughput requests, and log reads.
|
|
- Result: cli_surface_score=100.0, passed_cases=30, failed_cases=0, total_cases=30.
|
|
- Insight: The CLI now has a clean progression from parser -> semantic summary -> machine-usable plan -> concrete execution contract for the highest-value workflows, which is a strong base for wiring real handlers next.
|
|
- Next: raise the benchmark from contract shape to actual execution stubs or backend request payload semantics, especially for status, throughput bench, and model operations.
|
|
|
|
### Run 17: baseline - cli_surface_score=68.75 (KEEP)
|
|
- Timestamp: 2026-03-26 18:59
|
|
- What changed: Raised the benchmark with summary-focused ignored tests that require `render_command_summary()` to preserve output format, models directory, command-specific arguments, and key defaults for status, models load, auth enable, bench throughput, and logs.
|
|
- Result: cli_surface_score=68.75, passed_cases=11, failed_cases=5, total_cases=16.
|
|
- Insight: The parser surface is solid, but the current summary layer is too shallow; it only reports the top-level command and server, losing the semantic detail needed for later execution planning.
|
|
- Next: upgrade `render_command_summary()` so it emits nested command paths and the highest-value arguments/defaults for each management workflow.
|
|
|
|
### Run 18: enrich cli semantic summaries - cli_surface_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 19:00
|
|
- What changed: Reworked `render_command_summary()` in `crates/localllm-cli/src/main.rs` to emit nested command paths plus key defaults and high-value arguments for status, models, auth, config, bench, and logs workflows.
|
|
- Result: cli_surface_score=100.0, passed_cases=16, failed_cases=0, total_cases=16.
|
|
- Insight: A lightweight semantic/intention layer is enough to make the CLI contract meaningfully richer before wiring real handlers; the CLI now preserves actionable detail instead of collapsing everything to top-level groups.
|
|
- Next: raise the benchmark again toward execution planning or output contracts, e.g. structured action plans for backend/admin operations rather than just human-readable summaries.
|
|
|
|
## Segment 9: CLI Execution Stubs
|
|
- New objective: raise the CLI benchmark from abstract execution contracts to concrete execution stubs that preserve HTTP method/path/body semantics or named local actions for the highest-value workflows.
|
|
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
|
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
|
|
|
### Run 23: baseline - cli_surface_score=83.333 (KEEP)
|
|
- Timestamp: 2026-03-26 19:20
|
|
- What changed: Expanded the benchmark from execution contracts to concrete execution stubs, adding ignored tests for HTTP method/path/body semantics and named local actions for status, model scan/load, auth enable, throughput bench, and logs.
|
|
- Result: cli_surface_score=83.333, passed_cases=30, failed_cases=6, total_cases=36.
|
|
- Insight: The CLI already knows how to parse commands and describe execution contracts, but `execution_stub()` is still just a placeholder returning `SummaryOnly { command: "unimplemented" }` for every high-value workflow.
|
|
- Next: implement concrete stub mapping so status and throughput produce execution-ready HTTP request shapes and local workflows emit named local actions with preserved detail.
|
|
|
|
### Run 24: add cli execution stubs - cli_surface_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 19:22
|
|
- What changed: Implemented concrete `execution_stub()` mapping in `crates/localllm-cli/src/main.rs`, turning the highest-value execution contracts into execution-ready HTTP requests and named local actions with preserved detail.
|
|
- Result: cli_surface_score=100.0, passed_cases=36, failed_cases=0, total_cases=36.
|
|
- Insight: The CLI now has a full design ladder for the key workflows - parser, semantic summary, machine-usable plan, execution contract, and finally execution stub - which is a strong handoff point for wiring real handlers next.
|
|
- Next: raise the benchmark from stub shape to actual handler behavior or request payload construction for the same high-value workflows.
|
|
|
|
## Segment 10: CLI Execution Previews
|
|
- New objective: raise the CLI benchmark from execution stubs to dry-run previews that expand the same high-value workflows into full previewable HTTP URLs or local-effect descriptions.
|
|
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
|
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
|
|
|
### Run 25: baseline - cli_surface_score=85.714 (KEEP)
|
|
- Timestamp: 2026-03-26 19:27
|
|
- What changed: Expanded the benchmark from execution stubs to dry-run previews, adding ignored tests for full preview URLs and human-readable local-effect descriptions across status, models scan/load, auth enable, throughput bench, and logs.
|
|
- Result: cli_surface_score=85.714, passed_cases=36, failed_cases=6, total_cases=42.
|
|
- Insight: The CLI already models execution stubs correctly, but `execution_preview()` still collapses every high-value workflow to `SummaryOnly { command: "preview-unimplemented" }`, so there is no user-facing dry-run layer yet.
|
|
- Next: map the highest-value execution stubs into concrete `HttpPreview` and `LocalPreview` values using the resolved `--server` base URL and readable local-effect text.
|
|
|
|
### Run 26: add cli execution previews - cli_surface_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 19:28
|
|
- What changed: Implemented concrete `execution_preview()` mapping in `crates/localllm-cli/src/main.rs`, expanding HTTP stubs into full URLs with preserved payloads and local actions into readable effect descriptions via `preview_effect()`.
|
|
- Result: cli_surface_score=100.0, passed_cases=42, failed_cases=0, total_cases=42.
|
|
- Insight: The CLI now has a complete dry-run preview layer for the highest-value workflows, which is a strong bridge from internal planning seams to future real handlers and `--dry-run` UX.
|
|
- Next: raise the benchmark from preview shape to actual handler behavior or request-body construction details for status, throughput, and model-management workflows.
|
|
|
|
## Segment 11: CLI Dry-Run Output
|
|
- New objective: raise the CLI benchmark from internal preview objects to final user-facing `--dry-run` output in both `text` and `json`, so the same high-value workflows are ready for a real preview UX at the command line.
|
|
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
|
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
|
|
|
### Run 27: baseline - cli_surface_score=87.5 (KEEP)
|
|
- Timestamp: 2026-03-26 19:32
|
|
- What changed: Expanded the benchmark from internal `ExecutionPreview` objects to final `--dry-run` UX in both text and JSON, while keeping `render_execution_preview()` intentionally shallow as `DRY RUN {summary}`.
|
|
- Result: cli_surface_score=87.5, passed_cases=42, failed_cases=6, total_cases=48.
|
|
- Insight: The CLI already knows how to build the preview object graph, but the final presentation layer still collapses to summaries instead of rendering health/chat HTTP previews and local-effect dry-run text.
|
|
- Next: render the existing preview layer into final text/json output using the current `ExecutionPreview` data rather than introducing a new execution seam.
|
|
|
|
### Run 28: render cli dry-run previews - cli_surface_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 19:34
|
|
- What changed: Implemented final `--dry-run` rendering in `crates/localllm-cli/src/main.rs`, mapping `ExecutionPreview` values to user-facing text and JSON output and reusing the existing preview pipeline (`plan_command` -> `execution_contract` -> `execution_stub` -> `execution_preview`).
|
|
- Result: cli_surface_score=100.0, passed_cases=48, failed_cases=0, total_cases=48.
|
|
- Insight: The CLI now has a complete dry-run stack for the highest-value workflows, from parser through final user-visible text/json previews, without yet executing real side effects.
|
|
- Next: raise the bar from dry-run rendering to real handler behavior or deterministic request/body construction for backend-facing commands.
|
|
|
|
## Segment 12: CLI Backend Invocations
|
|
- New objective: raise the CLI benchmark from dry-run rendering to the first real backend-call specifications for commands that can already work today, especially `status`, `models list`, and `bench throughput`.
|
|
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
|
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
|
|
|
## Segment 13: CLI Backend Requests
|
|
- New objective: raise the CLI benchmark from backend invocation specs to executable backend request construction for the commands that already align with live server endpoints today.
|
|
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
|
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
|
|
|
### Run 29: baseline - cli_surface_score=94.118 (KEEP)
|
|
- Timestamp: 2026-03-26 19:39
|
|
- What changed: Expanded the benchmark from final dry-run rendering to backend invocation specs for commands that can already work today, adding ignored tests for live `GET /health`, live `GET /v1/models`, and a throughput `POST /v1/chat/completions` request that preserves the selected model and token budget.
|
|
- Result: cli_surface_score=94.118, passed_cases=48, failed_cases=3, total_cases=51.
|
|
- Insight: The CLI already handled every previous parser/planning/dry-run layer, but `backend_invocation()` was still a placeholder, and `models list` had not yet been promoted into the real backend-facing seam.
|
|
- Next: implement concrete backend invocation specs for `status`, `models list`, and `bench throughput` using the resolved server base URL and auth presence.
|
|
|
|
### Run 30: add cli backend invocation specs - cli_surface_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 19:41
|
|
- What changed: Implemented `backend_invocation(&Cli)` in `crates/localllm-cli/src/main.rs` so `status` maps to live `GET /health`, `models list` maps to live `GET /v1/models`, and `bench throughput` maps to a live `POST /v1/chat/completions` invocation that preserves `--model`, samples, token budget, and auth presence.
|
|
- Result: cli_surface_score=100.0, passed_cases=51, failed_cases=0, total_cases=51.
|
|
- Insight: The CLI now crosses from dry-run-only UX into concrete backend-facing invocation specs for commands that can already work against the current server today, without needing the future control-plane endpoints yet.
|
|
- Next: raise the bar from backend invocation specs to actual request payload construction or the first executable handlers for these already-supported commands.
|
|
|
|
### Run 31: baseline - cli_surface_score=94.444 (KEEP)
|
|
- Timestamp: 2026-03-26 19:47
|
|
- What changed: Expanded the benchmark from backend invocation specs to executable backend request construction, requiring final HTTP method/url/header/body request objects with deterministic JSON payloads for `status`, `models list`, and `bench throughput`.
|
|
- Result: cli_surface_score=94.444, passed_cases=51, failed_cases=3, total_cases=54.
|
|
- Insight: The CLI already had parser, summaries, plans, contracts, stubs, previews, dry-run rendering, and invocation specs, but `backend_request(&Cli)` still returned `SummaryOnly { command: "request-unimplemented" }` instead of final request objects.
|
|
- Next: map `BackendInvocation::HttpCall` into executable `BackendRequest::HttpRequest` values with bearer headers, JSON content-type, and deterministic benchmark payload generation.
|
|
|
|
### Run 32: add cli backend request specs - cli_surface_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 19:49
|
|
- What changed: Implemented `backend_request(&Cli)` in `crates/localllm-cli/src/main.rs` plus helpers for bearer headers, JSON headers, deterministic benchmark prompt construction, and exact chat payload generation for throughput requests.
|
|
- Result: cli_surface_score=100.0, passed_cases=54, failed_cases=0, total_cases=54.
|
|
- Insight: The first live-compatible CLI workflows now preserve the full executable backend request boundary - method, URL, auth headers, content-type, and deterministic JSON body - while still remaining execution-free.
|
|
- Next: raise the bar from request construction to the first live response-handling or executable handler seams for the same workflows.
|
|
|
|
## Segment 14: CLI Backend Responses
|
|
- New objective: raise the CLI benchmark from executable backend request construction to backend response interpretation for the same live-compatible workflows, starting with `status`, `models list`, and `bench throughput`.
|
|
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
|
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
|
|
|
### Run 33: baseline - cli_surface_score=94.737 (KEEP)
|
|
- Timestamp: 2026-03-26 20:01
|
|
- What changed: Added the segment 14 response-preview seam and local deserializable DTOs for health, models list, and throughput usage responses, but left `backend_response_preview()` intentionally unimplemented for the new live-compatible workflows.
|
|
- Result: cli_surface_score=94.737, passed_cases=54, failed_cases=3, total_cases=57.
|
|
- Insight: The CLI already had every layer through executable backend request construction, but it still collapsed real response parsing for `status`, `models list`, and `bench throughput` to `SummaryOnly { command: "response-unimplemented" }`.
|
|
- Next: parse the real response bodies into CLI-local previews for health status, model ids/count, and throughput usage metrics.
|
|
|
|
### Run 34: add cli backend response previews - cli_surface_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 20:02
|
|
- What changed: Implemented `backend_response_preview()` in `crates/localllm-cli/src/main.rs` so `status` parses health payloads, `models list` collects model ids/count, and `bench throughput` reads `usage.total_tokens` plus `usage.tokens_per_second` into CLI-local preview values.
|
|
- Result: cli_surface_score=100.0, passed_cases=57, failed_cases=0, total_cases=57.
|
|
- Insight: The first live-compatible CLI workflows now preserve the full response side of the contract too: request construction plus deterministic response interpretation without forcing reuse of server-only DTOs.
|
|
- Next: raise the bar from response previews to the first real executable handlers or user-facing live output rendering for commands that already work against the current backend.
|
|
|
|
## Segment 15: CLI Live Output
|
|
- New objective: raise the CLI benchmark from internal backend response previews to final user-facing live output rendering for the already live-compatible workflows `status`, `models list`, and `bench throughput`.
|
|
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
|
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
|
|
|
### Run 35: baseline - cli_surface_score=90.476 (KEEP)
|
|
- Timestamp: 2026-03-26 20:06
|
|
- What changed: Expanded the benchmark from backend response previews to final user-facing live output rendering, but left `render_backend_response()` as a shallow `LIVE {summary}` placeholder.
|
|
- Result: cli_surface_score=90.476, passed_cases=57, failed_cases=6, total_cases=63.
|
|
- Insight: The CLI already parsed health, model-list, and throughput responses correctly, but it still lacked final text/json presentation for those live-compatible workflows.
|
|
- Next: render `BackendResponsePreview` into final `STATUS`, `MODELS`, and `THROUGHPUT` text/json outputs.
|
|
|
|
### Run 36: render cli live outputs - cli_surface_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 20:07
|
|
- What changed: Implemented `render_backend_response()` in `crates/localllm-cli/src/main.rs` so the live-compatible commands now render final text/json output directly from `BackendResponsePreview`, including health summaries, model-id lists, and throughput usage output.
|
|
- Result: cli_surface_score=100.0, passed_cases=63, failed_cases=0, total_cases=63.
|
|
- Insight: The CLI now closes the full loop for the first live-compatible workflows, from parser and planning all the way to final user-facing live output.
|
|
- Next: move from live output rendering to the first real executable handlers for commands that already work today.
|
|
|
|
## Segment 16: CLI Live Execution
|
|
- New objective: raise the CLI benchmark from final live output rendering to the first real executable handlers for the live-compatible workflows `status`, `models list`, and `bench throughput` using an injectable transport seam.
|
|
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
|
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
|
|
|
### Run 37: baseline - cli_surface_score=95.455 (KEEP)
|
|
- Timestamp: 2026-03-26 20:12
|
|
- What changed: Expanded the benchmark from live output rendering to an injectable live-execution seam, but left `execute_live_command_with()` as a placeholder that ignored the transport response and returned `LIVE EXEC {summary}`.
|
|
- Result: cli_surface_score=95.455, passed_cases=63, failed_cases=3, total_cases=66.
|
|
- Insight: The pipeline already had correct request construction and response rendering, but the executor still dropped the returned status/body instead of feeding them into the live renderer.
|
|
- Next: wire the transport response through `render_backend_response()` without changing any other CLI seam.
|
|
|
|
### Run 38: add cli live execution pipeline - cli_surface_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 20:13
|
|
- What changed: Implemented `execute_live_command_with()` in `crates/localllm-cli/src/main.rs` so live-compatible commands now call the injected transport with `BackendRequest`, then pass the returned `status_code` and `body` into `render_backend_response()`.
|
|
- Result: cli_surface_score=100.0, passed_cases=66, failed_cases=0, total_cases=66.
|
|
- Insight: The CLI now completes the first fully executable end-to-end path for the live-compatible workflows, from parser to transport invocation to final rendered output.
|
|
- Next: move from transport-injected execution to the first concrete HTTP client seam for commands that can already hit the current backend directly.
|
|
|
|
## Segment 17: CLI ureq HTTP Transport
|
|
|
|
- Objective: replace the injectable fake transport with a concrete `ureq` HTTP client for `status`, `models list`, and `bench throughput`.
|
|
- Benchmark: `./autoresearch.sh` runs ignored `cli_surface_*` tests in `crates/localllm-cli/src/main.rs`.
|
|
- Keep/discard rule: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
|
|
|
### Run 39: add ureq http transport and execute_live_command - cli_surface_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 20:36
|
|
- What changed: Added `ureq` to workspace and `localllm-cli` deps. Implemented `execute_live_command(cli)` using `ureq` as the real HTTP transport, delegating to `execute_live_command_with()`. Updated `main()` to call `execute_live_command()` instead of the scaffold placeholder. Added 3 new ignored benchmark tests verifying the injectable seam correctly routes status/models/bench through the transport pipeline.
|
|
- Result: cli_surface_score=100.0, passed_cases=69, failed_cases=0, total_cases=69.
|
|
- Insight: The CLI now has a real HTTP client (`ureq`) wired end-to-end for live-compatible commands. The injectable transport seam remains available for testing without a running server. The CLI is functionally complete for the commands that the current backend already supports.
|
|
- Next: CLI is at natural saturation given the current backend. Further progress requires new admin/control-plane endpoints in the backend for scan, load, auth, config, and watch commands.
|
|
|
|
## Segment 18: Unified Tauri Runtime Manager
|
|
|
|
- Objective: replace the split desktop runtime control flow with one saved runtime-manager surface that owns local startup, remote routing, and local model/runtime settings.
|
|
- Benchmark: targeted verification bundle using `npm test`, `npm run build`, and `cargo check` plus acceptance checks for a single runtime control plane and updated runtime UI.
|
|
- Keep/discard rule: a keep requires one working config/start path, no broken sidebar start button, and green verification across frontend and Rust checks.
|
|
|
|
### Run 40: unify runtime manager config/start flow and refactor runtime UI - runtime_surface_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 23:35
|
|
- What changed: Removed the active dependency on `useServerProcess` and the embedded legacy `ServerPanel` from the sidebar flow, rebuilt `RuntimeSettingsDrawer` into a full local/remote runtime workspace, added Tauri wrappers for GPU/model picking and scanning, made the sidebar runtime card route setup/start from the same saved runtime state, and refreshed runtime/sidebar/global styling toward a cleaner AI desktop identity.
|
|
- Result: runtime_surface_score=100.0, unit_passed=1, build_passed=1, cargo_check_passed=1, unified_control_plane=1, design_refactor_done=1.
|
|
- Insight: The runtime failures were caused by state ownership drift, not startup logic alone. Once local runtime configuration and startup lived in the same domain, both runtime entry points aligned and the broken start path disappeared.
|
|
- Next: improve readiness probing beyond raw TCP-open detection, then smoke-test the desktop runtime end to end against a real local model and record a stricter follow-up score for startup/readiness behavior.
|
|
|
|
### Run 41: replace tcp-only readiness with /health readiness probe - runtime_surface_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 23:43
|
|
- What changed: Replaced the runtime startup readiness check in `tauri-app/src-tauri/src/runtime/health.rs` so it now issues `GET /health`, requires an HTTP 200 response, and validates `{"status":"ok"}` before marking the local API as ready. Added Rust tests covering healthy HTTP success, unhealthy JSON payload rejection, and plain-open-port rejection.
|
|
- Result: runtime_surface_score=100.0, unit_passed=1, build_passed=1, cargo_check_passed=1, health_probe_passed=1.
|
|
- Insight: TCP-open was too weak for a desktop-managed API because any bound socket could masquerade as readiness. A lightweight health request gives the runtime manager a much more truthful startup signal without introducing frontend complexity.
|
|
- Next: run an end-to-end desktop smoke test against a real local model so the next segment can score startup behavior from the actual app surface, not just backend verification.
|
|
|
|
### Run 42: add runtime UI smoke coverage and preserve saved model on empty rescans - runtime_surface_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-26 23:50
|
|
- What changed: Added jsdom-based runtime smoke tests for `RuntimeStatusCard` and `RuntimeSettingsDrawer`, installed `jsdom` for Vitest browserless component coverage, and fixed `chooseModel()` in `tauri-app/src/features/runtime/components/RuntimeSettingsDrawer.tsx` so a previously saved `model_path` is preserved when a models-folder rescan returns no results.
|
|
- Result: runtime_surface_score=100.0, unit_passed=1, build_passed=1, cargo_check_passed=1, runtime_smoke_passed=1.
|
|
- Insight: The first useful app-surface smoke test immediately caught a real regression that backend/runtime checks could not see: a valid saved model could disappear from drawer state after an empty scan, which silently disabled the unified start action.
|
|
- Next: push the smoke coverage one step further with a focused end-to-end startup path check, ideally validating that a saved local config transitions from setup -> starting -> ready correctly from the desktop surface.
|
|
|
|
## Segment 19: Tauri Runtime Lifecycle Acceptance
|
|
|
|
- Objective: replace the stale CLI-oriented benchmark with a Tauri desktop lifecycle benchmark and harden the app-surface contract around local/remote mode transitions.
|
|
- Benchmark: `./autoresearch.sh` now runs `experiments/bench_tauri_runtime_surface.py`, which scores runtime-focused frontend tests plus `npm run build` and `cargo check`.
|
|
- Keep/discard rule: higher `runtime_lifecycle_score` wins while the active desktop runtime CTA and verification bundle remain green.
|
|
|
|
### Run 43: align the Tauri lifecycle harness and keep the local CTA visible during mode switches - runtime_lifecycle_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-27 01:26
|
|
- What changed: Replaced the stale CLI benchmark in `autoresearch.sh` with a new `experiments/bench_tauri_runtime_surface.py` driver, added a runtime smoke test that switches `RuntimeSettingsDrawer` from remote to local mode, and fixed the drawer footer in `tauri-app/src/features/runtime/components/RuntimeSettingsDrawer.tsx` so the local primary CTA is keyed off the edited local `mode` instead of the previously saved `snapshot.config.connection_mode`.
|
|
- Result: runtime_lifecycle_score=100.0, unit_passed=1, build_passed=1, cargo_check_passed=1, local_mode_cta_passed=1.
|
|
- Insight: The previous segment was saturated partly because the harness was no longer measuring the live desktop contract. Once the benchmark targeted the Tauri runtime surface directly, it immediately exposed a real lifecycle regression: switching from remote to local mode could hide the apply/start CTA before save, making the saved local flow feel broken even though the backend state was valid.
|
|
- Next: raise the lifecycle score again with a stricter saved-startup path check, ideally covering the `stopped -> starting -> ready` transition and degraded behavior after a failed health revalidation.
|
|
|
|
### Run 44: cover saved local startup transitions and local onboarding from browse to ready - runtime_lifecycle_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-27 01:40
|
|
- What changed: Extended `tauri-app/src/features/runtime/components/RuntimeSurface.test.tsx` with two hook-backed lifecycle checks that run through `useRuntimeManager`: one now requires a saved local config to expose `Local API offline -> Starting local API -> Ready`, and another exercises local onboarding by browsing a models folder, auto-selecting the discovered model, saving config, and starting the local API from the same drawer flow. Updated `tauri-app/src/features/runtime/useRuntimeManager.ts` to publish optimistic `starting` and `stopping` snapshots while the Tauri commands are in flight. Raised `experiments/bench_tauri_runtime_surface.py` so the benchmark now records `local_mode_cta_passed`, `saved_startup_flow_passed`, and `local_onboarding_flow_passed` as explicit lifecycle checks alongside the broader runtime suite.
|
|
- Result: runtime_lifecycle_score=100.0, unit_passed=1, build_passed=1, cargo_check_passed=1, local_mode_cta_passed=1, saved_startup_flow_passed=1, local_onboarding_flow_passed=1.
|
|
- Insight: The saved local startup flow was previously trustworthy only after the backend returned; the desktop surface showed `Working...` but skipped the visible `starting` state entirely. Moving the in-flight state into `useRuntimeManager` closes that UX gap and makes the status card/drawer reflect the real lifecycle immediately. The onboarding smoke also proves the unified drawer now covers browse -> auto-select -> save -> start without falling back to separate legacy controls.
|
|
- Next: keep the lifecycle segment focused on truthfulness after startup, especially periodic `/health` revalidation that can demote a stale `ready` process into `degraded` when the child stops serving correctly.
|
|
|
|
### Run 45: revalidate managed local health after startup and surface degraded state - runtime_lifecycle_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-27 03:11
|
|
- What changed: Added a degraded-state lifecycle check to `tauri-app/src/features/runtime/components/RuntimeSurface.test.tsx`, extended `experiments/bench_tauri_runtime_surface.py` with `health_revalidation_passed`, and updated `tauri-app/src-tauri/src/runtime/health.rs` plus `tauri-app/src-tauri/src/runtime/mod.rs` so a running managed local child is periodically revalidated via `/health`. The runtime now demotes stale `ready` state to `degraded` when the process is still alive but no longer serves a healthy endpoint, and clears the `health_check_failed` error when health recovers. Added Rust tests covering both the degrade and recovery paths.
|
|
- Result: runtime_lifecycle_score=100.0, unit_passed=1, build_passed=1, cargo_check_passed=1, local_mode_cta_passed=1, saved_startup_flow_passed=1, local_onboarding_flow_passed=1, health_revalidation_passed=1.
|
|
- Insight: The frontend surface was already honest once it received a degraded snapshot; the real lifecycle gap was that backend runtime state could stay stuck at `ready` forever as long as the child process had not exited. Revalidating `/health` inside runtime state refresh closes that truthfulness gap without reintroducing a second frontend control plane.
|
|
- Next: this lifecycle segment is saturated again at `100.0%`. The next worthwhile move should raise the bar beyond state truthfulness, likely around a higher-fidelity desktop integration check or a real managed-runtime smoke path against an actual local model process.
|
|
|
|
## Segment 20: localllm Batch Assert
|
|
|
|
- Objective: stop the local `localllm` server from crashing on first chat completion with `GGML_ASSERT(n_tokens_all <= cparams.n_batch)`.
|
|
- Benchmark: `./autoresearch.sh` now runs a focused prompt-prefill regression harness in `experiments/bench_localllm_prefill_batch.py`.
|
|
- Keep/discard rule: higher `prefill_batch_assert_score` wins while the targeted prompt-prefill checks, `cargo check`, and a real `LOCALLLM_BATCH=128` completion remain green.
|
|
|
|
## Segment 21: Tauri Chat Active History
|
|
|
|
- Objective: make a newly sent message create/select its chat immediately, keep the live generation attached to that chat, and preserve history when revisiting generated chats.
|
|
- Benchmark: `./autoresearch.sh` now runs focused frontend regressions in `src/App.test.tsx` and `src/components/ChatView.test.tsx` plus `npm run build`.
|
|
- Keep/discard rule: higher `chat_history_score` wins while both the app-level send/create/history flow and the stale-request abort guard remain green.
|
|
|
|
### Run 47: preserve in-flight generation when chatId switches to the newly created chat - chat_history_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-28 20:06
|
|
- What changed: Added an app-level regression for sending from the empty composer into a brand-new chat, then updated `tauri-app/src/components/ChatView.tsx` to track the pending request chat id so `chatId` changes only abort when the user truly switches away to another chat, not when the composer promotes `null` into the just-created chat.
|
|
- Result: chat_history_score=100.0, app_send_flow_passed=1, chat_switch_abort_guard_passed=1, frontend_tests_passed=1, frontend_build_passed=1.
|
|
- Insight: The previous saved-chat race fix was directionally right but too broad. `null -> newly created chat id` is part of the same send flow, not a user navigation away from the active request.
|
|
- Next: if more reports appear, raise the bar with a browser/E2E smoke that sends a message, switches chats mid-stream, and returns to confirm persisted history from the rendered app shell.
|
|
- Initial hypothesis: `crates/inference-core/src/session.rs` still uses `llama_batch_get_one()` for full prompt prefill, which likely requests logits for every prompt token. Newer llama.cpp output-buffer semantics appear to need an explicit batch that only asks for logits on the final prompt token and splits larger prompts by `n_batch`.
|
|
- Baseline status: pending first red run.
|
|
|
|
### Run 46: chunk prompt prefill explicitly and fix live batch metadata - prefill_batch_assert_score=100.0 (KEEP)
|
|
- Timestamp: 2026-03-27 15:08
|
|
- What changed: Replaced full-prompt prefill in `crates/inference-core/src/session.rs` with an explicit chunked `llama_batch_init()` path that only requests logits for the final prompt token and splits prompts by `n_batch`. Passed `n_batch` from `crates/inference-core/src/engine.rs`, bound `llama_get_memory` / `llama_memory_clear` in `crates/llama-sys/src/lib.rs`, and cleared context memory after warmup so the live repro started from a clean state. The decisive live fix was also wiring `batch.n_tokens` correctly before each `llama_decode()` call.
|
|
- Result: prefill_batch_assert_score=100.0, prefill_plan_passed=1, cargo_check_passed=1, server_build_passed=1, live_completion_passed=1.
|
|
- Insight: The original GGML assert came from the old shorthand batch path over-requesting prompt outputs, but the first explicit prefill implementation still failed live because `llama_batch_init()` leaves `n_tokens` uninitialized. Once the prefill batch both requested only the final prompt logit and reported a valid token count, the real `LOCALLLM_BATCH=128` completion succeeded against the local Qwen GGUF without crashing or returning `generation_failed`.
|
|
- Next: keep this branch state and consider a follow-up hardening segment around longer multi-chunk prompts, multi-sequence compatibility, or a regression harness that distinguishes warmup-only startup success from first-request decode success.
|