ForgeLM/autoresearch.md

Autoresearch: localllm-cli-control-surface

Objective

Implement a full management CLI on top of the current local backend version, starting with a professional command surface for status, model operations, auth, config, logs, health, and throughput benchmarking. This segment focuses first on the CLI surface and contracts, so later experiments can add real execution behavior without redesigning the UX shape.

Metrics

• Primary: cli_surface_score (percent, higher is better)
• Secondary: passed_cases, failed_cases, total_cases

How to Run

./autoresearch.sh - runs targeted ignored tests for CLI command-surface behavior and outputs METRIC name=number lines.

Files in Scope

• crates/localllm-cli/ - new CLI crate, parser, command UX, output modes, and CLI-focused tests
• crates/inference-server/src/main.rs - current backend env/config seam the CLI will eventually manage
• crates/inference-server/src/routes.rs - existing inference endpoints the CLI can interrogate today
• crates/inference-server/src/compat.rs - existing auth/config knobs the CLI must understand and expose coherently
• crates/api-contracts/ - shared contracts the CLI may eventually consume
• experiments/bench_cli_surface.py - benchmark harness for the autoresearch loop
• autoresearch.sh - loop benchmark entrypoint and fast preflight

Off Limits

• models/ model binaries
• vendor/ third-party sources and prebuilt llama binaries
• registry/ Python registry service unless a later CLI experiment explicitly needs to replace or wire it
• ui/ frontend files
• Tauri app implementation in this segment
• unrelated latency or template experiments

Constraints

• Preserve the currently working backend behavior.
• Regular test suite must stay green for keeps.
• Prefer a stable, professional CLI contract over premature execution internals.
• Commands should map cleanly to the product plan discussed with the user: status, models, auth, config, bench, logs, doctor.
• The benchmark is test-driven: keep/discard follows the CLI surface suite score first.
• A justified CLI-focused dependency is acceptable if it reduces maintenance cost.

What's Been Tried

• Segment 5 pivots autoresearch from template hardening to productization, specifically a complete CLI-first control surface on top of the current backend version.
• The current repo has no CLI crate, no Tauri app, and no control-plane crate; the backend is still configured almost entirely by LOCALLLM_* environment variables and exposes only /health, /v1/models, and /v1/chat/completions.
• The first benchmark for this segment is intentionally surface-oriented: ignored tests score whether the CLI presents the expected professional command set before deeper execution behavior is implemented.
• Baseline run 15 (fd773a9) scored only 18.182%: the new CLI scaffold currently exposes status and models list, but the broader management surface (models scan/load/watch/download, auth, config, bench, logs, doctor) is still absent.
• Run 16 (70cc59a) expanded the Clap parser to the full intended command surface and raised segment 5 from 18.182% to 100.0% (11/11 passing), giving the CLI a stable UX contract before execution handlers are wired.
• Segment 6 raises the bar from command presence to semantic usefulness: the benchmark now checks whether key workflows produce meaningful command summaries that preserve defaults and high-value arguments, as a stepping stone toward real execution planning.
• Baseline run 17 (53a7dbb) scored 68.75%: the parser already accepted the workflows, but render_command_summary() still threw away important semantic detail such as nested command paths, output format, alias selection, and throughput defaults.
• Run 18 (66b30de) enriched semantic summaries across the CLI and raised segment 6 from 68.75% to 100.0% (16/16 passing), so the next bar should move from summary correctness to execution-planning or output-contract semantics.
• Segment 7 raises the bar from semantic summaries to structured execution plans, checking whether key commands preserve actionable details in a machine-usable plan that can later drive real handlers.
• Baseline run 19 (9154a7b) scored 69.565%: the CLI still passes all parser and summary tests, but every new plan-oriented workflow fails because plan_command() currently returns SummaryOnly for all commands.