docs: finalize cli autoresearch segment 17

This commit is contained in:
Evandro Fonseca Junior committed 2026-03-26 20:38:37 -03:00
1 parent 257aed4cc7
commit 13ba05b02a
4 files changed
+39 -6

No files matched your search

+5 -6
View File
@@ -1,10 +1,9 @@
# Autoresearch Dashboard: localllm-cli-backend-requests
# Autoresearch Dashboard: localllm-cli-ureq-transport
**Runs:** 2 | **Kept:** 2 | **Discarded:** 0 | **Crashed:** 0
**Baseline:** cli_surface_score: 94.444% (#31)
**Best:** cli_surface_score: 100.0% (#32, +5.9%)
**Runs:** 1 | **Kept:** 1 | **Discarded:** 0 | **Crashed:** 0
**Baseline:** cli_surface_score: 100.0% (#39)
**Best:** cli_surface_score: 100.0% (#39, +0.0%)
| # | commit | cli_surface_score | status | description |
|---|--------|-------------------|--------|-------------|
| 31 | f970c05 | 94.444% (+0.0%) | keep | baseline |
| 32 | 5de35c5 | 100.0% (+5.9%) | keep | add cli backend request specs |
| 39 | 9860d64 | 100.0% (+0.0%) | keep | add ureq http transport and execute_live_command |
+4
View File
@@ -51,3 +51,7 @@
{"run":35,"commit":"e1ada2c","metric":90.476,"metrics":{"passed_cases":57,"failed_cases":6,"total_cases":63},"status":"keep","description":"baseline","timestamp":1774566405,"segment":15}
{"run":36,"commit":"c54107d","metric":100.0,"metrics":{"passed_cases":63,"failed_cases":0,"total_cases":63},"status":"keep","description":"render cli live outputs","timestamp":1774566456,"segment":15}
{"type":"config","name":"localllm-cli-live-execution","metricName":"cli_surface_score","metricUnit":"%","bestDirection":"higher"}
{"run":37,"commit":"388733a","metric":95.455,"metrics":{"passed_cases":63,"failed_cases":3,"total_cases":66},"status":"keep","description":"baseline","timestamp":1774566763,"segment":16}
{"run":38,"commit":"6920e2a","metric":100.0,"metrics":{"passed_cases":66,"failed_cases":0,"total_cases":66},"status":"keep","description":"add cli live execution pipeline","timestamp":1774566821,"segment":16}
{"type":"config","name":"localllm-cli-ureq-transport","metricName":"cli_surface_score","metricUnit":"%","bestDirection":"higher"}
{"run":39,"commit":"9860d64","metric":100.0,"metrics":{"passed_cases":69,"failed_cases":0,"total_cases":69},"status":"keep","description":"add ureq http transport and execute_live_command","timestamp":1774568181,"segment":17}
+3
View File
@@ -75,3 +75,6 @@ so later experiments can add real execution behavior without redesigning the UX
- Run 35 (`e1ada2c`) scored `90.476%` because `render_backend_response()` still collapsed the live-compatible workflows to `LIVE {summary}` instead of final text/json output.
- Run 36 (`c54107d`) rendered the first live-compatible backend responses into final user-facing text/json output, raising segment 15 from `90.476%` to `100.0%` (63/63 passing).
- Segment 16 raises the bar from live output rendering to the first real executable handlers, checking whether the same live-compatible workflows can execute through an injectable transport seam and return the final rendered output end-to-end.
- Run 37 (`388733a`) scored `95.455%` because `execute_live_command_with()` still ignored the transport response and returned `LIVE EXEC {summary}` instead of final live output.
- Run 38 (`6920e2a`) wired the injected transport response into `render_backend_response()`, raising segment 16 from `95.455%` to `100.0%` (66/66 passing).
- Segment 17 adds `ureq` as the real HTTP transport and implements `execute_live_command(cli)` that delegates to `execute_live_command_with()` using `ureq`. Run 39 (`9860d64`) scored `100.0%` (69/69 passing) as a direct keep — the CLI is now functionally complete for the commands the current backend supports.
+27
View File
@@ -353,3 +353,30 @@
- New objective: raise the CLI benchmark from final live output rendering to the first real executable handlers for the live-compatible workflows `status`, `models list`, and `bench throughput` using an injectable transport seam.
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
### Run 37: baseline - cli_surface_score=95.455 (KEEP)
- Timestamp: 2026-03-26 20:12
- What changed: Expanded the benchmark from live output rendering to an injectable live-execution seam, but left `execute_live_command_with()` as a placeholder that ignored the transport response and returned `LIVE EXEC {summary}`.
- Result: cli_surface_score=95.455, passed_cases=63, failed_cases=3, total_cases=66.
- Insight: The pipeline already had correct request construction and response rendering, but the executor still dropped the returned status/body instead of feeding them into the live renderer.
- Next: wire the transport response through `render_backend_response()` without changing any other CLI seam.
### Run 38: add cli live execution pipeline - cli_surface_score=100.0 (KEEP)
- Timestamp: 2026-03-26 20:13
- What changed: Implemented `execute_live_command_with()` in `crates/localllm-cli/src/main.rs` so live-compatible commands now call the injected transport with `BackendRequest`, then pass the returned `status_code` and `body` into `render_backend_response()`.
- Result: cli_surface_score=100.0, passed_cases=66, failed_cases=0, total_cases=66.
- Insight: The CLI now completes the first fully executable end-to-end path for the live-compatible workflows, from parser to transport invocation to final rendered output.
- Next: move from transport-injected execution to the first concrete HTTP client seam for commands that can already hit the current backend directly.
## Segment 17: CLI ureq HTTP Transport
- Objective: replace the injectable fake transport with a concrete `ureq` HTTP client for `status`, `models list`, and `bench throughput`.
- Benchmark: `./autoresearch.sh` runs ignored `cli_surface_*` tests in `crates/localllm-cli/src/main.rs`.
- Keep/discard rule: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
### Run 39: add ureq http transport and execute_live_command - cli_surface_score=100.0 (KEEP)
- Timestamp: 2026-03-26 20:36
- What changed: Added `ureq` to workspace and `localllm-cli` deps. Implemented `execute_live_command(cli)` using `ureq` as the real HTTP transport, delegating to `execute_live_command_with()`. Updated `main()` to call `execute_live_command()` instead of the scaffold placeholder. Added 3 new ignored benchmark tests verifying the injectable seam correctly routes status/models/bench through the transport pipeline.
- Result: cli_surface_score=100.0, passed_cases=69, failed_cases=0, total_cases=69.
- Insight: The CLI now has a real HTTP client (`ureq`) wired end-to-end for live-compatible commands. The injectable transport seam remains available for testing without a running server. The CLI is functionally complete for the commands that the current backend already supports.
- Next: CLI is at natural saturation given the current backend. Further progress requires new admin/control-plane endpoints in the backend for scan, load, auth, config, and watch commands.