diff --git a/autoresearch-dashboard.md b/autoresearch-dashboard.md index 058fb9d..d9f85ea 100644 --- a/autoresearch-dashboard.md +++ b/autoresearch-dashboard.md @@ -1,10 +1,9 @@ -# Autoresearch Dashboard: dynamic-chat-templates-variants +# Autoresearch Dashboard: localllm-cli-execution-plans -**Runs:** 2 | **Kept:** 2 | **Discarded:** 0 | **Crashed:** 0 -**Baseline:** template_score: 90.909% (#13) -**Best:** template_score: 100.0% (#14, +10.0%) +**Runs:** 1 | **Kept:** 1 | **Discarded:** 0 | **Crashed:** 0 +**Baseline:** cli_surface_score: 69.565% (#19) +**Best:** cli_surface_score: 69.565% (#19, +0.0%) -| # | commit | template_score | status | description | -|---|--------|----------------|--------|-------------| -| 13 | ffab766 | 90.909% (+0.0%) | keep | baseline | -| 14 | f200733 | 100.0% (+10.0%) | keep | detect moonshot as kimi family | +| # | commit | cli_surface_score | status | description | +|---|--------|-------------------|--------|-------------| +| 19 | 9154a7b | 69.565% (+0.0%) | keep | baseline | diff --git a/autoresearch.jsonl b/autoresearch.jsonl index 09386d0..f9397cc 100644 --- a/autoresearch.jsonl +++ b/autoresearch.jsonl @@ -21,3 +21,7 @@ {"run":15,"commit":"fd773a9","metric":18.182,"metrics":{"passed_cases":2,"failed_cases":9,"total_cases":11},"status":"keep","description":"baseline","timestamp":1774562027,"segment":5} {"run":16,"commit":"70cc59a","metric":100.0,"metrics":{"passed_cases":11,"failed_cases":0,"total_cases":11},"status":"keep","description":"expand cli command surface","timestamp":1774562138,"segment":5} {"type":"config","name":"localllm-cli-semantic-contracts","metricName":"cli_surface_score","metricUnit":"%","bestDirection":"higher"} +{"run":17,"commit":"53a7dbb","metric":68.75,"metrics":{"passed_cases":11,"failed_cases":5,"total_cases":16},"status":"keep","description":"baseline","timestamp":1774562358,"segment":6} +{"run":18,"commit":"66b30de","metric":100.0,"metrics":{"passed_cases":16,"failed_cases":0,"total_cases":16},"status":"keep","description":"enrich cli semantic summaries","timestamp":1774562434,"segment":6} +{"type":"config","name":"localllm-cli-execution-plans","metricName":"cli_surface_score","metricUnit":"%","bestDirection":"higher"} +{"run":19,"commit":"9154a7b","metric":69.565,"metrics":{"passed_cases":16,"failed_cases":7,"total_cases":23},"status":"keep","description":"baseline","timestamp":1774562688,"segment":7} diff --git a/autoresearch.md b/autoresearch.md index 60e884e..9909677 100644 --- a/autoresearch.md +++ b/autoresearch.md @@ -45,3 +45,7 @@ so later experiments can add real execution behavior without redesigning the UX - Baseline run 15 (`fd773a9`) scored only `18.182%`: the new CLI scaffold currently exposes `status` and `models list`, but the broader management surface (`models scan/load/watch/download`, `auth`, `config`, `bench`, `logs`, `doctor`) is still absent. - Run 16 (`70cc59a`) expanded the Clap parser to the full intended command surface and raised segment 5 from `18.182%` to `100.0%` (11/11 passing), giving the CLI a stable UX contract before execution handlers are wired. - Segment 6 raises the bar from command presence to semantic usefulness: the benchmark now checks whether key workflows produce meaningful command summaries that preserve defaults and high-value arguments, as a stepping stone toward real execution planning. +- Baseline run 17 (`53a7dbb`) scored `68.75%`: the parser already accepted the workflows, but `render_command_summary()` still threw away important semantic detail such as nested command paths, output format, alias selection, and throughput defaults. +- Run 18 (`66b30de`) enriched semantic summaries across the CLI and raised segment 6 from `68.75%` to `100.0%` (16/16 passing), so the next bar should move from summary correctness to execution-planning or output-contract semantics. +- Segment 7 raises the bar from semantic summaries to structured execution plans, checking whether key commands preserve actionable details in a machine-usable plan that can later drive real handlers. +- Baseline run 19 (`9154a7b`) scored `69.565%`: the CLI still passes all parser and summary tests, but every new plan-oriented workflow fails because `plan_command()` currently returns `SummaryOnly` for all commands. diff --git a/crates/localllm-cli/src/main.rs b/crates/localllm-cli/src/main.rs index cf453bb..e2186cf 100644 --- a/crates/localllm-cli/src/main.rs +++ b/crates/localllm-cli/src/main.rs @@ -233,8 +233,59 @@ fn render_command_summary(cli: &Cli) -> String { } fn plan_command(cli: &Cli) -> CommandPlan { - CommandPlan::SummaryOnly { - command: command_summary(&cli.command), + match &cli.command { + Commands::Status => CommandPlan::ProbeStatus { + server: cli.server.clone(), + api_key_present: cli.api_key.is_some(), + format: cli.format, + }, + Commands::Models(models) => match &models.command { + ModelsSubcommand::Scan => CommandPlan::ScanModels { + models_dir: cli.models_dir.clone(), + }, + ModelsSubcommand::Load(args) => CommandPlan::LoadModel { + models_dir: cli.models_dir.clone(), + model: args.model.clone(), + alias: args.alias.clone(), + }, + _ => CommandPlan::SummaryOnly { + command: command_summary(&cli.command), + }, + }, + Commands::Auth(auth) => match &auth.command { + AuthSubcommand::Enable(args) => CommandPlan::ConfigureAuth { + mode: args.mode, + api_key: args.api_key.clone(), + }, + _ => CommandPlan::SummaryOnly { + command: command_summary(&cli.command), + }, + }, + Commands::Config(config) => match &config.command { + ConfigSubcommand::Set(args) => CommandPlan::SetConfig { + key: args.key.clone(), + value: args.value.clone(), + }, + _ => CommandPlan::SummaryOnly { + command: command_summary(&cli.command), + }, + }, + Commands::Bench(bench) => match &bench.command { + BenchSubcommand::Throughput(args) => CommandPlan::RunThroughputBench { + server: cli.server.clone(), + api_key_present: cli.api_key.is_some(), + samples: args.samples, + prompt_tokens: args.prompt_tokens, + completion_tokens: args.completion_tokens, + }, + _ => CommandPlan::SummaryOnly { + command: command_summary(&cli.command), + }, + }, + Commands::Logs(args) => CommandPlan::ReadLogs { lines: args.lines }, + Commands::Doctor => CommandPlan::SummaryOnly { + command: command_summary(&cli.command), + }, } } diff --git a/experiments/worklog.md b/experiments/worklog.md index 572e00d..b154165 100644 --- a/experiments/worklog.md +++ b/experiments/worklog.md @@ -163,3 +163,29 @@ - New objective: keep the CLI surface stable, but raise the benchmark so key commands preserve useful semantic detail such as defaults, high-value arguments, and summary/intention contracts. - Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`. - Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green. + +## Segment 7: CLI Execution Plans +- New objective: raise the CLI benchmark from human-readable semantic summaries to machine-usable execution plans for key workflows. +- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`. +- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green. + +### Run 19: baseline - cli_surface_score=69.565 (KEEP) +- Timestamp: 2026-03-26 19:04 +- What changed: Expanded the CLI benchmark with plan-oriented ignored tests that require key commands to map into machine-usable `CommandPlan` variants instead of only semantic summaries. +- Result: cli_surface_score=69.565, passed_cases=16, failed_cases=7, total_cases=23. +- Insight: The semantic summary layer is already solid, but `plan_command()` is still a stub that collapses everything to `SummaryOnly`, so execution planning has not been wired yet. +- Next: implement concrete `CommandPlan` mapping for status, models scan/load, auth enable, config set, bench throughput, and logs. + +### Run 17: baseline - cli_surface_score=68.75 (KEEP) +- Timestamp: 2026-03-26 18:59 +- What changed: Raised the benchmark with summary-focused ignored tests that require `render_command_summary()` to preserve output format, models directory, command-specific arguments, and key defaults for status, models load, auth enable, bench throughput, and logs. +- Result: cli_surface_score=68.75, passed_cases=11, failed_cases=5, total_cases=16. +- Insight: The parser surface is solid, but the current summary layer is too shallow; it only reports the top-level command and server, losing the semantic detail needed for later execution planning. +- Next: upgrade `render_command_summary()` so it emits nested command paths and the highest-value arguments/defaults for each management workflow. + +### Run 18: enrich cli semantic summaries - cli_surface_score=100.0 (KEEP) +- Timestamp: 2026-03-26 19:00 +- What changed: Reworked `render_command_summary()` in `crates/localllm-cli/src/main.rs` to emit nested command paths plus key defaults and high-value arguments for status, models, auth, config, bench, and logs workflows. +- Result: cli_surface_score=100.0, passed_cases=16, failed_cases=0, total_cases=16. +- Insight: A lightweight semantic/intention layer is enough to make the CLI contract meaningfully richer before wiring real handlers; the CLI now preserves actionable detail instead of collapsing everything to top-level groups. +- Next: raise the benchmark again toward execution planning or output contracts, e.g. structured action plans for backend/admin operations rather than just human-readable summaries.