55 lines
5.0 KiB
Markdown
55 lines
5.0 KiB
Markdown
# Autoresearch: localllm-cli-control-surface
|
|
|
|
## Objective
|
|
Implement a full management CLI on top of the current local backend version, starting with
|
|
a professional command surface for status, model operations, auth, config, logs, health,
|
|
and throughput benchmarking. This segment focuses first on the CLI surface and contracts,
|
|
so later experiments can add real execution behavior without redesigning the UX shape.
|
|
|
|
## Metrics
|
|
- **Primary**: cli_surface_score (percent, higher is better)
|
|
- **Secondary**: passed_cases, failed_cases, total_cases
|
|
|
|
## How to Run
|
|
`./autoresearch.sh` - runs targeted ignored tests for CLI command-surface behavior and outputs `METRIC name=number` lines.
|
|
|
|
## Files in Scope
|
|
- `crates/localllm-cli/` - new CLI crate, parser, command UX, output modes, and CLI-focused tests
|
|
- `crates/inference-server/src/main.rs` - current backend env/config seam the CLI will eventually manage
|
|
- `crates/inference-server/src/routes.rs` - existing inference endpoints the CLI can interrogate today
|
|
- `crates/inference-server/src/compat.rs` - existing auth/config knobs the CLI must understand and expose coherently
|
|
- `crates/api-contracts/` - shared contracts the CLI may eventually consume
|
|
- `experiments/bench_cli_surface.py` - benchmark harness for the autoresearch loop
|
|
- `autoresearch.sh` - loop benchmark entrypoint and fast preflight
|
|
|
|
## Off Limits
|
|
- `models/` model binaries
|
|
- `vendor/` third-party sources and prebuilt llama binaries
|
|
- `registry/` Python registry service unless a later CLI experiment explicitly needs to replace or wire it
|
|
- `ui/` frontend files
|
|
- Tauri app implementation in this segment
|
|
- unrelated latency or template experiments
|
|
|
|
## Constraints
|
|
- Preserve the currently working backend behavior.
|
|
- Regular test suite must stay green for keeps.
|
|
- Prefer a stable, professional CLI contract over premature execution internals.
|
|
- Commands should map cleanly to the product plan discussed with the user: status, models, auth, config, bench, logs, doctor.
|
|
- The benchmark is test-driven: keep/discard follows the CLI surface suite score first.
|
|
- A justified CLI-focused dependency is acceptable if it reduces maintenance cost.
|
|
|
|
## What's Been Tried
|
|
- Segment 5 pivots autoresearch from template hardening to productization, specifically a complete CLI-first control surface on top of the current backend version.
|
|
- The current repo has no CLI crate, no Tauri app, and no control-plane crate; the backend is still configured almost entirely by `LOCALLLM_*` environment variables and exposes only `/health`, `/v1/models`, and `/v1/chat/completions`.
|
|
- The first benchmark for this segment is intentionally surface-oriented: ignored tests score whether the CLI presents the expected professional command set before deeper execution behavior is implemented.
|
|
- Baseline run 15 (`fd773a9`) scored only `18.182%`: the new CLI scaffold currently exposes `status` and `models list`, but the broader management surface (`models scan/load/watch/download`, `auth`, `config`, `bench`, `logs`, `doctor`) is still absent.
|
|
- Run 16 (`70cc59a`) expanded the Clap parser to the full intended command surface and raised segment 5 from `18.182%` to `100.0%` (11/11 passing), giving the CLI a stable UX contract before execution handlers are wired.
|
|
- Segment 6 raises the bar from command presence to semantic usefulness: the benchmark now checks whether key workflows produce meaningful command summaries that preserve defaults and high-value arguments, as a stepping stone toward real execution planning.
|
|
- Baseline run 17 (`53a7dbb`) scored `68.75%`: the parser already accepted the workflows, but `render_command_summary()` still threw away important semantic detail such as nested command paths, output format, alias selection, and throughput defaults.
|
|
- Run 18 (`66b30de`) enriched semantic summaries across the CLI and raised segment 6 from `68.75%` to `100.0%` (16/16 passing), so the next bar should move from summary correctness to execution-planning or output-contract semantics.
|
|
- Segment 7 raises the bar from semantic summaries to structured execution plans, checking whether key commands preserve actionable details in a machine-usable plan that can later drive real handlers.
|
|
- Baseline run 19 (`9154a7b`) scored `69.565%`: the CLI still passes all parser and summary tests, but every new plan-oriented workflow fails because `plan_command()` currently returns `SummaryOnly` for all commands.
|
|
- Run 20 (`bc584be`) introduced concrete `CommandPlan` variants for status, model scan/load, auth enable, config set, throughput bench, and logs, raising segment 7 from `69.565%` to `100.0%` (23/23 passing).
|
|
- Segment 8 raises the bar from machine-usable plans to execution contracts, checking whether key workflows can already describe the concrete HTTP/local action they intend to perform at the backend/control-plane seam.
|
|
- Baseline run 21 (`3d1460b`) scored `76.667%`: the CLI already preserves parser, summary, and plan semantics, but `execution_contract()` still collapses the new workflows to `SummaryOnly { command: "unplanned" }` instead of concrete HTTP/local contracts.
|