48 lines
3.4 KiB
Markdown
48 lines
3.4 KiB
Markdown
# Autoresearch: localllm-cli-control-surface
|
|
|
|
## Objective
|
|
Implement a full management CLI on top of the current local backend version, starting with
|
|
a professional command surface for status, model operations, auth, config, logs, health,
|
|
and throughput benchmarking. This segment focuses first on the CLI surface and contracts,
|
|
so later experiments can add real execution behavior without redesigning the UX shape.
|
|
|
|
## Metrics
|
|
- **Primary**: cli_surface_score (percent, higher is better)
|
|
- **Secondary**: passed_cases, failed_cases, total_cases
|
|
|
|
## How to Run
|
|
`./autoresearch.sh` - runs targeted ignored tests for CLI command-surface behavior and outputs `METRIC name=number` lines.
|
|
|
|
## Files in Scope
|
|
- `crates/localllm-cli/` - new CLI crate, parser, command UX, output modes, and CLI-focused tests
|
|
- `crates/inference-server/src/main.rs` - current backend env/config seam the CLI will eventually manage
|
|
- `crates/inference-server/src/routes.rs` - existing inference endpoints the CLI can interrogate today
|
|
- `crates/inference-server/src/compat.rs` - existing auth/config knobs the CLI must understand and expose coherently
|
|
- `crates/api-contracts/` - shared contracts the CLI may eventually consume
|
|
- `experiments/bench_cli_surface.py` - benchmark harness for the autoresearch loop
|
|
- `autoresearch.sh` - loop benchmark entrypoint and fast preflight
|
|
|
|
## Off Limits
|
|
- `models/` model binaries
|
|
- `vendor/` third-party sources and prebuilt llama binaries
|
|
- `registry/` Python registry service unless a later CLI experiment explicitly needs to replace or wire it
|
|
- `ui/` frontend files
|
|
- Tauri app implementation in this segment
|
|
- unrelated latency or template experiments
|
|
|
|
## Constraints
|
|
- Preserve the currently working backend behavior.
|
|
- Regular test suite must stay green for keeps.
|
|
- Prefer a stable, professional CLI contract over premature execution internals.
|
|
- Commands should map cleanly to the product plan discussed with the user: status, models, auth, config, bench, logs, doctor.
|
|
- The benchmark is test-driven: keep/discard follows the CLI surface suite score first.
|
|
- A justified CLI-focused dependency is acceptable if it reduces maintenance cost.
|
|
|
|
## What's Been Tried
|
|
- Segment 5 pivots autoresearch from template hardening to productization, specifically a complete CLI-first control surface on top of the current backend version.
|
|
- The current repo has no CLI crate, no Tauri app, and no control-plane crate; the backend is still configured almost entirely by `LOCALLLM_*` environment variables and exposes only `/health`, `/v1/models`, and `/v1/chat/completions`.
|
|
- The first benchmark for this segment is intentionally surface-oriented: ignored tests score whether the CLI presents the expected professional command set before deeper execution behavior is implemented.
|
|
- Baseline run 15 (`fd773a9`) scored only `18.182%`: the new CLI scaffold currently exposes `status` and `models list`, but the broader management surface (`models scan/load/watch/download`, `auth`, `config`, `bench`, `logs`, `doctor`) is still absent.
|
|
- Run 16 (`70cc59a`) expanded the Clap parser to the full intended command surface and raised segment 5 from `18.182%` to `100.0%` (11/11 passing), giving the CLI a stable UX contract before execution handlers are wired.
|
|
- Segment 6 raises the bar from command presence to semantic usefulness: the benchmark now checks whether key workflows produce meaningful command summaries that preserve defaults and high-value arguments, as a stepping stone toward real execution planning.
|