From 13ba05b02af0986866dab0df386f3417f452bbd7 Mon Sep 17 00:00:00 2001 From: Evandro Fonseca Junior Date: Thu, 26 Mar 2026 20:38:37 -0300 Subject: [PATCH] docs: finalize cli autoresearch segment 17 --- autoresearch-dashboard.md | 11 +++++------ autoresearch.jsonl | 4 ++++ autoresearch.md | 3 +++ experiments/worklog.md | 27 +++++++++++++++++++++++++++ 4 files changed, 39 insertions(+), 6 deletions(-) diff --git a/autoresearch-dashboard.md b/autoresearch-dashboard.md index 1138621..3e46e65 100644 --- a/autoresearch-dashboard.md +++ b/autoresearch-dashboard.md @@ -1,10 +1,9 @@ -# Autoresearch Dashboard: localllm-cli-backend-requests +# Autoresearch Dashboard: localllm-cli-ureq-transport -**Runs:** 2 | **Kept:** 2 | **Discarded:** 0 | **Crashed:** 0 -**Baseline:** cli_surface_score: 94.444% (#31) -**Best:** cli_surface_score: 100.0% (#32, +5.9%) +**Runs:** 1 | **Kept:** 1 | **Discarded:** 0 | **Crashed:** 0 +**Baseline:** cli_surface_score: 100.0% (#39) +**Best:** cli_surface_score: 100.0% (#39, +0.0%) | # | commit | cli_surface_score | status | description | |---|--------|-------------------|--------|-------------| -| 31 | f970c05 | 94.444% (+0.0%) | keep | baseline | -| 32 | 5de35c5 | 100.0% (+5.9%) | keep | add cli backend request specs | +| 39 | 9860d64 | 100.0% (+0.0%) | keep | add ureq http transport and execute_live_command | diff --git a/autoresearch.jsonl b/autoresearch.jsonl index 0dddbef..ec8b58b 100644 --- a/autoresearch.jsonl +++ b/autoresearch.jsonl @@ -51,3 +51,7 @@ {"run":35,"commit":"e1ada2c","metric":90.476,"metrics":{"passed_cases":57,"failed_cases":6,"total_cases":63},"status":"keep","description":"baseline","timestamp":1774566405,"segment":15} {"run":36,"commit":"c54107d","metric":100.0,"metrics":{"passed_cases":63,"failed_cases":0,"total_cases":63},"status":"keep","description":"render cli live outputs","timestamp":1774566456,"segment":15} {"type":"config","name":"localllm-cli-live-execution","metricName":"cli_surface_score","metricUnit":"%","bestDirection":"higher"} +{"run":37,"commit":"388733a","metric":95.455,"metrics":{"passed_cases":63,"failed_cases":3,"total_cases":66},"status":"keep","description":"baseline","timestamp":1774566763,"segment":16} +{"run":38,"commit":"6920e2a","metric":100.0,"metrics":{"passed_cases":66,"failed_cases":0,"total_cases":66},"status":"keep","description":"add cli live execution pipeline","timestamp":1774566821,"segment":16} +{"type":"config","name":"localllm-cli-ureq-transport","metricName":"cli_surface_score","metricUnit":"%","bestDirection":"higher"} +{"run":39,"commit":"9860d64","metric":100.0,"metrics":{"passed_cases":69,"failed_cases":0,"total_cases":69},"status":"keep","description":"add ureq http transport and execute_live_command","timestamp":1774568181,"segment":17} diff --git a/autoresearch.md b/autoresearch.md index c71a643..826b2d1 100644 --- a/autoresearch.md +++ b/autoresearch.md @@ -75,3 +75,6 @@ so later experiments can add real execution behavior without redesigning the UX - Run 35 (`e1ada2c`) scored `90.476%` because `render_backend_response()` still collapsed the live-compatible workflows to `LIVE {summary}` instead of final text/json output. - Run 36 (`c54107d`) rendered the first live-compatible backend responses into final user-facing text/json output, raising segment 15 from `90.476%` to `100.0%` (63/63 passing). - Segment 16 raises the bar from live output rendering to the first real executable handlers, checking whether the same live-compatible workflows can execute through an injectable transport seam and return the final rendered output end-to-end. +- Run 37 (`388733a`) scored `95.455%` because `execute_live_command_with()` still ignored the transport response and returned `LIVE EXEC {summary}` instead of final live output. +- Run 38 (`6920e2a`) wired the injected transport response into `render_backend_response()`, raising segment 16 from `95.455%` to `100.0%` (66/66 passing). +- Segment 17 adds `ureq` as the real HTTP transport and implements `execute_live_command(cli)` that delegates to `execute_live_command_with()` using `ureq`. Run 39 (`9860d64`) scored `100.0%` (69/69 passing) as a direct keep — the CLI is now functionally complete for the commands the current backend supports. diff --git a/experiments/worklog.md b/experiments/worklog.md index 71a41cd..4d4c5d6 100644 --- a/experiments/worklog.md +++ b/experiments/worklog.md @@ -353,3 +353,30 @@ - New objective: raise the CLI benchmark from final live output rendering to the first real executable handlers for the live-compatible workflows `status`, `models list`, and `bench throughput` using an injectable transport seam. - Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`. - Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green. + +### Run 37: baseline - cli_surface_score=95.455 (KEEP) +- Timestamp: 2026-03-26 20:12 +- What changed: Expanded the benchmark from live output rendering to an injectable live-execution seam, but left `execute_live_command_with()` as a placeholder that ignored the transport response and returned `LIVE EXEC {summary}`. +- Result: cli_surface_score=95.455, passed_cases=63, failed_cases=3, total_cases=66. +- Insight: The pipeline already had correct request construction and response rendering, but the executor still dropped the returned status/body instead of feeding them into the live renderer. +- Next: wire the transport response through `render_backend_response()` without changing any other CLI seam. + +### Run 38: add cli live execution pipeline - cli_surface_score=100.0 (KEEP) +- Timestamp: 2026-03-26 20:13 +- What changed: Implemented `execute_live_command_with()` in `crates/localllm-cli/src/main.rs` so live-compatible commands now call the injected transport with `BackendRequest`, then pass the returned `status_code` and `body` into `render_backend_response()`. +- Result: cli_surface_score=100.0, passed_cases=66, failed_cases=0, total_cases=66. +- Insight: The CLI now completes the first fully executable end-to-end path for the live-compatible workflows, from parser to transport invocation to final rendered output. +- Next: move from transport-injected execution to the first concrete HTTP client seam for commands that can already hit the current backend directly. + +## Segment 17: CLI ureq HTTP Transport + +- Objective: replace the injectable fake transport with a concrete `ureq` HTTP client for `status`, `models list`, and `bench throughput`. +- Benchmark: `./autoresearch.sh` runs ignored `cli_surface_*` tests in `crates/localllm-cli/src/main.rs`. +- Keep/discard rule: higher `cli_surface_score` wins while the normal non-ignored suites remain green. + +### Run 39: add ureq http transport and execute_live_command - cli_surface_score=100.0 (KEEP) +- Timestamp: 2026-03-26 20:36 +- What changed: Added `ureq` to workspace and `localllm-cli` deps. Implemented `execute_live_command(cli)` using `ureq` as the real HTTP transport, delegating to `execute_live_command_with()`. Updated `main()` to call `execute_live_command()` instead of the scaffold placeholder. Added 3 new ignored benchmark tests verifying the injectable seam correctly routes status/models/bench through the transport pipeline. +- Result: cli_surface_score=100.0, passed_cases=69, failed_cases=0, total_cases=69. +- Insight: The CLI now has a real HTTP client (`ureq`) wired end-to-end for live-compatible commands. The injectable transport seam remains available for testing without a running server. The CLI is functionally complete for the commands that the current backend already supports. +- Next: CLI is at natural saturation given the current backend. Further progress requires new admin/control-plane endpoints in the backend for scan, load, auth, config, and watch commands.