# Autoresearch: localllm-cli-control-surface

## Objective
Implement a full management CLI on top of the current local backend version, starting with
a professional command surface for status, model operations, auth, config, logs, health,
and throughput benchmarking. This segment focuses first on the CLI surface and contracts,
so later experiments can add real execution behavior without redesigning the UX shape.

## Metrics
- **Primary**: cli_surface_score (percent, higher is better)
- **Secondary**: passed_cases, failed_cases, total_cases

## How to Run
`./autoresearch.sh` - runs targeted ignored tests for CLI command-surface behavior and outputs `METRIC name=number` lines.

## Files in Scope
- `crates/localllm-cli/` - new CLI crate, parser, command UX, output modes, and CLI-focused tests
- `crates/inference-server/src/main.rs` - current backend env/config seam the CLI will eventually manage
- `crates/inference-server/src/routes.rs` - existing inference endpoints the CLI can interrogate today
- `crates/inference-server/src/compat.rs` - existing auth/config knobs the CLI must understand and expose coherently
- `crates/api-contracts/` - shared contracts the CLI may eventually consume
- `experiments/bench_cli_surface.py` - benchmark harness for the autoresearch loop
- `autoresearch.sh` - loop benchmark entrypoint and fast preflight

## Off Limits
- `models/` model binaries
- `vendor/` third-party sources and prebuilt llama binaries
- `registry/` Python registry service unless a later CLI experiment explicitly needs to replace or wire it
- `ui/` frontend files
- Tauri app implementation in this segment
- unrelated latency or template experiments

## Constraints
- Preserve the currently working backend behavior.
- Regular test suite must stay green for keeps.
- Prefer a stable, professional CLI contract over premature execution internals.
- Commands should map cleanly to the product plan discussed with the user: status, models, auth, config, bench, logs, doctor.
- The benchmark is test-driven: keep/discard follows the CLI surface suite score first.
- A justified CLI-focused dependency is acceptable if it reduces maintenance cost.

## What's Been Tried
- Segment 5 pivots autoresearch from template hardening to productization, specifically a complete CLI-first control surface on top of the current backend version.
- The current repo has no CLI crate, no Tauri app, and no control-plane crate; the backend is still configured almost entirely by `LOCALLLM_*` environment variables and exposes only `/health`, `/v1/models`, and `/v1/chat/completions`.
- The first benchmark for this segment is intentionally surface-oriented: ignored tests score whether the CLI presents the expected professional command set before deeper execution behavior is implemented.
- Baseline run 15 (`fd773a9`) scored only `18.182%`: the new CLI scaffold currently exposes `status` and `models list`, but the broader management surface (`models scan/load/watch/download`, `auth`, `config`, `bench`, `logs`, `doctor`) is still absent.
- Run 16 (`70cc59a`) expanded the Clap parser to the full intended command surface and raised segment 5 from `18.182%` to `100.0%` (11/11 passing), giving the CLI a stable UX contract before execution handlers are wired.
- Segment 6 raises the bar from command presence to semantic usefulness: the benchmark now checks whether key workflows produce meaningful command summaries that preserve defaults and high-value arguments, as a stepping stone toward real execution planning.
- Baseline run 17 (`53a7dbb`) scored `68.75%`: the parser already accepted the workflows, but `render_command_summary()` still threw away important semantic detail such as nested command paths, output format, alias selection, and throughput defaults.
- Run 18 (`66b30de`) enriched semantic summaries across the CLI and raised segment 6 from `68.75%` to `100.0%` (16/16 passing), so the next bar should move from summary correctness to execution-planning or output-contract semantics.
- Segment 7 raises the bar from semantic summaries to structured execution plans, checking whether key commands preserve actionable details in a machine-usable plan that can later drive real handlers.
- Baseline run 19 (`9154a7b`) scored `69.565%`: the CLI still passes all parser and summary tests, but every new plan-oriented workflow fails because `plan_command()` currently returns `SummaryOnly` for all commands.
- Run 20 (`bc584be`) introduced concrete `CommandPlan` variants for status, model scan/load, auth enable, config set, throughput bench, and logs, raising segment 7 from `69.565%` to `100.0%` (23/23 passing).
- Segment 8 raises the bar from machine-usable plans to execution contracts, checking whether key workflows can already describe the concrete HTTP/local action they intend to perform at the backend/control-plane seam.
- Baseline run 21 (`3d1460b`) scored `76.667%`: the CLI already preserves parser, summary, and plan semantics, but `execution_contract()` still collapses the new workflows to `SummaryOnly { command: "unplanned" }` instead of concrete HTTP/local contracts.
- Run 22 (`38acb52`) mapped the highest-value `CommandPlan` variants into concrete `ExecutionContract` values for `/health`, `/v1/chat/completions`, local model scan/load, local auth/config mutation, and local log reads, raising segment 8 from `76.667%` to `100.0%` (30/30 passing).
- Segment 9 raises the bar from contract shape to execution stubs, checking whether the same high-value workflows can already describe concrete HTTP method/path/body details or named local actions that a future real executor can run directly.