ForgeLM/autoresearch.md

# Autoresearch: localllm-cli-control-surface

## Objective
Implement a full management CLI on top of the current local backend version, starting with
a professional command surface for status, model operations, auth, config, logs, health,
and throughput benchmarking. This segment focuses first on the CLI surface and contracts,
so later experiments can add real execution behavior without redesigning the UX shape.

## Metrics
- **Primary**: cli_surface_score (percent, higher is better)
- **Secondary**: passed_cases, failed_cases, total_cases

## How to Run
`./autoresearch.sh` - runs targeted ignored tests for CLI command-surface behavior and outputs `METRIC name=number` lines.

## Files in Scope
- `crates/localllm-cli/` - new CLI crate, parser, command UX, output modes, and CLI-focused tests
- `crates/inference-server/src/main.rs` - current backend env/config seam the CLI will eventually manage
- `crates/inference-server/src/routes.rs` - existing inference endpoints the CLI can interrogate today
- `crates/inference-server/src/compat.rs` - existing auth/config knobs the CLI must understand and expose coherently
- `crates/api-contracts/` - shared contracts the CLI may eventually consume
- `experiments/bench_cli_surface.py` - benchmark harness for the autoresearch loop
- `autoresearch.sh` - loop benchmark entrypoint and fast preflight

## Off Limits
- `models/` model binaries
- `vendor/` third-party sources and prebuilt llama binaries
- `registry/` Python registry service unless a later CLI experiment explicitly needs to replace or wire it
- `ui/` frontend files
- Tauri app implementation in this segment
- unrelated latency or template experiments

## Constraints
- Preserve the currently working backend behavior.
- Regular test suite must stay green for keeps.
- Prefer a stable, professional CLI contract over premature execution internals.
- Commands should map cleanly to the product plan discussed with the user: status, models, auth, config, bench, logs, doctor.
- The benchmark is test-driven: keep/discard follows the CLI surface suite score first.
- A justified CLI-focused dependency is acceptable if it reduces maintenance cost.

## What's Been Tried
- Segment 5 pivots autoresearch from template hardening to productization, specifically a complete CLI-first control surface on top of the current backend version.
- The current repo has no CLI crate, no Tauri app, and no control-plane crate; the backend is still configured almost entirely by `LOCALLLM_*` environment variables and exposes only `/health`, `/v1/models`, and `/v1/chat/completions`.
- The first benchmark for this segment is intentionally surface-oriented: ignored tests score whether the CLI presents the expected professional command set before deeper execution behavior is implemented.
- Baseline run 15 (`fd773a9`) scored only `18.182%`: the new CLI scaffold currently exposes `status` and `models list`, but the broader management surface (`models scan/load/watch/download`, `auth`, `config`, `bench`, `logs`, `doctor`) is still absent.
- Run 16 (`70cc59a`) expanded the Clap parser to the full intended command surface and raised segment 5 from `18.182%` to `100.0%` (11/11 passing), giving the CLI a stable UX contract before execution handlers are wired.
- Segment 6 raises the bar from command presence to semantic usefulness: the benchmark now checks whether key workflows produce meaningful command summaries that preserve defaults and high-value arguments, as a stepping stone toward real execution planning.
- Baseline run 17 (`53a7dbb`) scored `68.75%`: the parser already accepted the workflows, but `render_command_summary()` still threw away important semantic detail such as nested command paths, output format, alias selection, and throughput defaults.
- Run 18 (`66b30de`) enriched semantic summaries across the CLI and raised segment 6 from `68.75%` to `100.0%` (16/16 passing), so the next bar should move from summary correctness to execution-planning or output-contract semantics.
- Segment 7 raises the bar from semantic summaries to structured execution plans, checking whether key commands preserve actionable details in a machine-usable plan that can later drive real handlers.
- Baseline run 19 (`9154a7b`) scored `69.565%`: the CLI still passes all parser and summary tests, but every new plan-oriented workflow fails because `plan_command()` currently returns `SummaryOnly` for all commands.
- Run 20 (`bc584be`) introduced concrete `CommandPlan` variants for status, model scan/load, auth enable, config set, throughput bench, and logs, raising segment 7 from `69.565%` to `100.0%` (23/23 passing).
- Segment 8 raises the bar from machine-usable plans to execution contracts, checking whether key workflows can already describe the concrete HTTP/local action they intend to perform at the backend/control-plane seam.
- Baseline run 21 (`3d1460b`) scored `76.667%`: the CLI already preserves parser, summary, and plan semantics, but `execution_contract()` still collapses the new workflows to `SummaryOnly { command: "unplanned" }` instead of concrete HTTP/local contracts.
- Run 22 (`38acb52`) mapped the highest-value `CommandPlan` variants into concrete `ExecutionContract` values for `/health`, `/v1/chat/completions`, local model scan/load, local auth/config mutation, and local log reads, raising segment 8 from `76.667%` to `100.0%` (30/30 passing).
- Segment 9 raises the bar from contract shape to execution stubs, checking whether the same high-value workflows can already describe concrete HTTP method/path/body details or named local actions that a future real executor can run directly.
- Baseline run 23 (`7030290`) scored `83.333%`: parser, summaries, plans, and execution contracts still pass, but `execution_stub()` is only a placeholder and returns `SummaryOnly { command: "unimplemented" }` instead of concrete HTTP/local stubs for the highest-value workflows.
- Run 24 (`50d435a`) mapped the highest-value `ExecutionContract` variants into concrete `ExecutionStub` values for `GET /health`, `POST /v1/chat/completions`, local model scan/load, local auth config, and local log tail actions, raising segment 9 from `83.333%` to `100.0%` (36/36 passing).
- Segment 10 raises the bar from execution stubs to dry-run previews, checking whether the same high-value workflows can expand into full previewable HTTP URLs or local-effect descriptions that are ready to show in a future `--dry-run` UX.
- Baseline run 25 (`cd480c4`) scored `85.714%`: parser, summaries, plans, contracts, and stubs still pass, but `execution_preview()` remained a placeholder and returned `SummaryOnly { command: "preview-unimplemented" }` instead of final dry-run previews.
- Run 26 (`c8ef7cf`) mapped the highest-value execution stubs into concrete `ExecutionPreview` values with full health/chat URLs and readable local-effect descriptions, raising segment 10 from `85.714%` to `100.0%` (42/42 passing).
- Segment 11 raises the bar from preview objects to final `--dry-run` output, checking whether the CLI can now render those previews as actual text/json UX for the same high-value workflows.
- Baseline run 27 (`7738acf`) scored `87.5%`: the CLI already had parser/summary/plan/contract/stub/preview layers, but `render_execution_preview()` still collapsed everything to `DRY RUN {summary}` instead of final text/json dry-run output.
- Run 28 (`99e5041`) rendered the highest-value `ExecutionPreview` values into final user-facing text/json dry-run output, raising segment 11 from `87.5%` to `100.0%` (48/48 passing).
- Segment 12 raises the bar from dry-run UX to the first real backend-call specifications for commands that can already work today, specifically `status`, `models list`, and `bench throughput`.
- Baseline run 29 (`1ff5ba3`) scored `94.118%`: every earlier CLI layer still passed, but `backend_invocation()` was still a placeholder and the new backend-facing tests for live status/models/bench requests failed.
- Run 30 (`bf40b2f`) added concrete backend invocation specs for `GET /health`, `GET /v1/models`, and `POST /v1/chat/completions` with preserved `--model`, token budget, server base URL, and auth presence, raising segment 12 from `94.118%` to `100.0%` (51/51 passing).
- Segment 13 raises the bar again from backend invocation specs to executable backend request construction, checking whether the same live-compatible commands can now produce final HTTP method/url/header/body request objects with deterministic JSON payloads.
- Run 31 (`f970c05`) scored `94.444%` because `backend_request()` was still a placeholder and the new backend-request tests failed for `status`, `models list`, and `bench throughput`.
- Run 32 (`5de35c5`) introduced concrete executable backend request specs with bearer/content-type headers and deterministic JSON benchmark payloads, raising segment 13 from `94.444%` to `100.0%` (54/54 passing).
- Segment 14 raises the bar from executable backend request specs to backend response interpretation, checking whether the same live-compatible commands can parse real response bodies into CLI-facing previews for health, model lists, and throughput usage metrics.
- Run 33 (`8cf03e3`) scored `94.737%` because `backend_response_preview()` was still a placeholder and the new response-parsing tests failed for `status`, `models list`, and `bench throughput`.
- Run 34 (`0506a4c`) introduced CLI-local backend response previews for health payloads, model list ids/count, and throughput usage metrics, raising segment 14 from `94.737%` to `100.0%` (57/57 passing).
- Segment 15 raises the bar from internal backend response previews to final user-facing live output rendering, checking whether the same live-compatible commands can render text/json output directly from parsed backend responses.
- Run 35 (`e1ada2c`) scored `90.476%` because `render_backend_response()` still collapsed the live-compatible workflows to `LIVE {summary}` instead of final text/json output.
- Run 36 (`c54107d`) rendered the first live-compatible backend responses into final user-facing text/json output, raising segment 15 from `90.476%` to `100.0%` (63/63 passing).
- Segment 16 raises the bar from live output rendering to the first real executable handlers, checking whether the same live-compatible workflows can execute through an injectable transport seam and return the final rendered output end-to-end.