ForgeLM/autoresearch.md

Autoresearch: localllm-cli-control-surface

Objective

Implement a full management CLI on top of the current local backend version, starting with a professional command surface for status, model operations, auth, config, logs, health, and throughput benchmarking. This segment focuses first on the CLI surface and contracts, so later experiments can add real execution behavior without redesigning the UX shape.

Metrics

• Primary: cli_surface_score (percent, higher is better)
• Secondary: passed_cases, failed_cases, total_cases

How to Run

./autoresearch.sh - runs targeted ignored tests for CLI command-surface behavior and outputs METRIC name=number lines.

Files in Scope

• crates/localllm-cli/ - new CLI crate, parser, command UX, output modes, and CLI-focused tests
• crates/inference-server/src/main.rs - current backend env/config seam the CLI will eventually manage
• crates/inference-server/src/routes.rs - existing inference endpoints the CLI can interrogate today
• crates/inference-server/src/compat.rs - existing auth/config knobs the CLI must understand and expose coherently
• crates/api-contracts/ - shared contracts the CLI may eventually consume
• experiments/bench_cli_surface.py - benchmark harness for the autoresearch loop
• autoresearch.sh - loop benchmark entrypoint and fast preflight

Off Limits

• models/ model binaries
• vendor/ third-party sources and prebuilt llama binaries
• registry/ Python registry service unless a later CLI experiment explicitly needs to replace or wire it
• ui/ frontend files
• Tauri app implementation in this segment
• unrelated latency or template experiments

Constraints

• Preserve the currently working backend behavior.
• Regular test suite must stay green for keeps.
• Prefer a stable, professional CLI contract over premature execution internals.
• Commands should map cleanly to the product plan discussed with the user: status, models, auth, config, bench, logs, doctor.
• The benchmark is test-driven: keep/discard follows the CLI surface suite score first.
• A justified CLI-focused dependency is acceptable if it reduces maintenance cost.

What's Been Tried

• Segment 5 pivots autoresearch from template hardening to productization, specifically a complete CLI-first control surface on top of the current backend version.
• The current repo has no CLI crate, no Tauri app, and no control-plane crate; the backend is still configured almost entirely by LOCALLLM_* environment variables and exposes only /health, /v1/models, and /v1/chat/completions.
• The first benchmark for this segment is intentionally surface-oriented: ignored tests score whether the CLI presents the expected professional command set before deeper execution behavior is implemented.
• Baseline run 15 (fd773a9) scored only 18.182%: the new CLI scaffold currently exposes status and models list, but the broader management surface (models scan/load/watch/download, auth, config, bench, logs, doctor) is still absent.
• Run 16 (70cc59a) expanded the Clap parser to the full intended command surface and raised segment 5 from 18.182% to 100.0% (11/11 passing), giving the CLI a stable UX contract before execution handlers are wired.
• Segment 6 raises the bar from command presence to semantic usefulness: the benchmark now checks whether key workflows produce meaningful command summaries that preserve defaults and high-value arguments, as a stepping stone toward real execution planning.
• Baseline run 17 (53a7dbb) scored 68.75%: the parser already accepted the workflows, but render_command_summary() still threw away important semantic detail such as nested command paths, output format, alias selection, and throughput defaults.
• Run 18 (66b30de) enriched semantic summaries across the CLI and raised segment 6 from 68.75% to 100.0% (16/16 passing), so the next bar should move from summary correctness to execution-planning or output-contract semantics.
• Segment 7 raises the bar from semantic summaries to structured execution plans, checking whether key commands preserve actionable details in a machine-usable plan that can later drive real handlers.
• Baseline run 19 (9154a7b) scored 69.565%: the CLI still passes all parser and summary tests, but every new plan-oriented workflow fails because plan_command() currently returns SummaryOnly for all commands.
• Run 20 (bc584be) introduced concrete CommandPlan variants for status, model scan/load, auth enable, config set, throughput bench, and logs, raising segment 7 from 69.565% to 100.0% (23/23 passing).
• Segment 8 raises the bar from machine-usable plans to execution contracts, checking whether key workflows can already describe the concrete HTTP/local action they intend to perform at the backend/control-plane seam.
• Baseline run 21 (3d1460b) scored 76.667%: the CLI already preserves parser, summary, and plan semantics, but execution_contract() still collapses the new workflows to SummaryOnly { command: "unplanned" } instead of concrete HTTP/local contracts.
• Run 22 (38acb52) mapped the highest-value CommandPlan variants into concrete ExecutionContract values for /health, /v1/chat/completions, local model scan/load, local auth/config mutation, and local log reads, raising segment 8 from 76.667% to 100.0% (30/30 passing).
• Segment 9 raises the bar from contract shape to execution stubs, checking whether the same high-value workflows can already describe concrete HTTP method/path/body details or named local actions that a future real executor can run directly.
• Baseline run 23 (7030290) scored 83.333%: parser, summaries, plans, and execution contracts still pass, but execution_stub() is only a placeholder and returns SummaryOnly { command: "unimplemented" } instead of concrete HTTP/local stubs for the highest-value workflows.
• Run 24 (50d435a) mapped the highest-value ExecutionContract variants into concrete ExecutionStub values for GET /health, POST /v1/chat/completions, local model scan/load, local auth config, and local log tail actions, raising segment 9 from 83.333% to 100.0% (36/36 passing).
• Segment 10 raises the bar from execution stubs to dry-run previews, checking whether the same high-value workflows can expand into full previewable HTTP URLs or local-effect descriptions that are ready to show in a future --dry-run UX.
• Baseline run 25 (cd480c4) scored 85.714%: parser, summaries, plans, contracts, and stubs still pass, but execution_preview() remained a placeholder and returned SummaryOnly { command: "preview-unimplemented" } instead of final dry-run previews.
• Run 26 (c8ef7cf) mapped the highest-value execution stubs into concrete ExecutionPreview values with full health/chat URLs and readable local-effect descriptions, raising segment 10 from 85.714% to 100.0% (42/42 passing).
• Segment 11 raises the bar from preview objects to final --dry-run output, checking whether the CLI can now render those previews as actual text/json UX for the same high-value workflows.
• Baseline run 27 (7738acf) scored 87.5%: the CLI already had parser/summary/plan/contract/stub/preview layers, but render_execution_preview() still collapsed everything to DRY RUN {summary} instead of final text/json dry-run output.
• Run 28 (99e5041) rendered the highest-value ExecutionPreview values into final user-facing text/json dry-run output, raising segment 11 from 87.5% to 100.0% (48/48 passing).
• Segment 12 raises the bar from dry-run UX to the first real backend-call specifications for commands that can already work today, specifically status, models list, and bench throughput.
• Baseline run 29 (1ff5ba3) scored 94.118%: every earlier CLI layer still passed, but backend_invocation() was still a placeholder and the new backend-facing tests for live status/models/bench requests failed.
• Run 30 (bf40b2f) added concrete backend invocation specs for GET /health, GET /v1/models, and POST /v1/chat/completions with preserved --model, token budget, server base URL, and auth presence, raising segment 12 from 94.118% to 100.0% (51/51 passing).
• Segment 13 raises the bar again from backend invocation specs to executable backend request construction, checking whether the same live-compatible commands can now produce final HTTP method/url/header/body request objects with deterministic JSON payloads.
• Run 31 (f970c05) scored 94.444% because backend_request() was still a placeholder and the new backend-request tests failed for status, models list, and bench throughput.
• Run 32 (5de35c5) introduced concrete executable backend request specs with bearer/content-type headers and deterministic JSON benchmark payloads, raising segment 13 from 94.444% to 100.0% (54/54 passing).
• Segment 14 raises the bar from executable backend request specs to backend response interpretation, checking whether the same live-compatible commands can parse real response bodies into CLI-facing previews for health, model lists, and throughput usage metrics.
• Run 33 (8cf03e3) scored 94.737% because backend_response_preview() was still a placeholder and the new response-parsing tests failed for status, models list, and bench throughput.
• Run 34 (0506a4c) introduced CLI-local backend response previews for health payloads, model list ids/count, and throughput usage metrics, raising segment 14 from 94.737% to 100.0% (57/57 passing).
• Segment 15 raises the bar from internal backend response previews to final user-facing live output rendering, checking whether the same live-compatible commands can render text/json output directly from parsed backend responses.