docs: finalize cli autoresearch segment 17
This commit is contained in:
1 parent
257aed4cc7
commit
13ba05b02a
4 files changed
+39
-6
No files matched your search
@@ -1,10 +1,9 @@
|
|||||||
# Autoresearch Dashboard: localllm-cli-backend-requests
|
# Autoresearch Dashboard: localllm-cli-ureq-transport
|
||||||
|
|
||||||
**Runs:** 2 | **Kept:** 2 | **Discarded:** 0 | **Crashed:** 0
|
**Runs:** 1 | **Kept:** 1 | **Discarded:** 0 | **Crashed:** 0
|
||||||
**Baseline:** cli_surface_score: 94.444% (#31)
|
**Baseline:** cli_surface_score: 100.0% (#39)
|
||||||
**Best:** cli_surface_score: 100.0% (#32, +5.9%)
|
**Best:** cli_surface_score: 100.0% (#39, +0.0%)
|
||||||
|
|
||||||
| # | commit | cli_surface_score | status | description |
|
| # | commit | cli_surface_score | status | description |
|
||||||
|---|--------|-------------------|--------|-------------|
|
|---|--------|-------------------|--------|-------------|
|
||||||
| 31 | f970c05 | 94.444% (+0.0%) | keep | baseline |
|
| 39 | 9860d64 | 100.0% (+0.0%) | keep | add ureq http transport and execute_live_command |
|
||||||
| 32 | 5de35c5 | 100.0% (+5.9%) | keep | add cli backend request specs |
|
|
||||||
@@ -51,3 +51,7 @@
|
|||||||
{"run":35,"commit":"e1ada2c","metric":90.476,"metrics":{"passed_cases":57,"failed_cases":6,"total_cases":63},"status":"keep","description":"baseline","timestamp":1774566405,"segment":15}
|
{"run":35,"commit":"e1ada2c","metric":90.476,"metrics":{"passed_cases":57,"failed_cases":6,"total_cases":63},"status":"keep","description":"baseline","timestamp":1774566405,"segment":15}
|
||||||
{"run":36,"commit":"c54107d","metric":100.0,"metrics":{"passed_cases":63,"failed_cases":0,"total_cases":63},"status":"keep","description":"render cli live outputs","timestamp":1774566456,"segment":15}
|
{"run":36,"commit":"c54107d","metric":100.0,"metrics":{"passed_cases":63,"failed_cases":0,"total_cases":63},"status":"keep","description":"render cli live outputs","timestamp":1774566456,"segment":15}
|
||||||
{"type":"config","name":"localllm-cli-live-execution","metricName":"cli_surface_score","metricUnit":"%","bestDirection":"higher"}
|
{"type":"config","name":"localllm-cli-live-execution","metricName":"cli_surface_score","metricUnit":"%","bestDirection":"higher"}
|
||||||
|
{"run":37,"commit":"388733a","metric":95.455,"metrics":{"passed_cases":63,"failed_cases":3,"total_cases":66},"status":"keep","description":"baseline","timestamp":1774566763,"segment":16}
|
||||||
|
{"run":38,"commit":"6920e2a","metric":100.0,"metrics":{"passed_cases":66,"failed_cases":0,"total_cases":66},"status":"keep","description":"add cli live execution pipeline","timestamp":1774566821,"segment":16}
|
||||||
|
{"type":"config","name":"localllm-cli-ureq-transport","metricName":"cli_surface_score","metricUnit":"%","bestDirection":"higher"}
|
||||||
|
{"run":39,"commit":"9860d64","metric":100.0,"metrics":{"passed_cases":69,"failed_cases":0,"total_cases":69},"status":"keep","description":"add ureq http transport and execute_live_command","timestamp":1774568181,"segment":17}
|
||||||
@@ -75,3 +75,6 @@ so later experiments can add real execution behavior without redesigning the UX
|
|||||||
- Run 35 (`e1ada2c`) scored `90.476%` because `render_backend_response()` still collapsed the live-compatible workflows to `LIVE {summary}` instead of final text/json output.
|
- Run 35 (`e1ada2c`) scored `90.476%` because `render_backend_response()` still collapsed the live-compatible workflows to `LIVE {summary}` instead of final text/json output.
|
||||||
- Run 36 (`c54107d`) rendered the first live-compatible backend responses into final user-facing text/json output, raising segment 15 from `90.476%` to `100.0%` (63/63 passing).
|
- Run 36 (`c54107d`) rendered the first live-compatible backend responses into final user-facing text/json output, raising segment 15 from `90.476%` to `100.0%` (63/63 passing).
|
||||||
- Segment 16 raises the bar from live output rendering to the first real executable handlers, checking whether the same live-compatible workflows can execute through an injectable transport seam and return the final rendered output end-to-end.
|
- Segment 16 raises the bar from live output rendering to the first real executable handlers, checking whether the same live-compatible workflows can execute through an injectable transport seam and return the final rendered output end-to-end.
|
||||||
|
- Run 37 (`388733a`) scored `95.455%` because `execute_live_command_with()` still ignored the transport response and returned `LIVE EXEC {summary}` instead of final live output.
|
||||||
|
- Run 38 (`6920e2a`) wired the injected transport response into `render_backend_response()`, raising segment 16 from `95.455%` to `100.0%` (66/66 passing).
|
||||||
|
- Segment 17 adds `ureq` as the real HTTP transport and implements `execute_live_command(cli)` that delegates to `execute_live_command_with()` using `ureq`. Run 39 (`9860d64`) scored `100.0%` (69/69 passing) as a direct keep — the CLI is now functionally complete for the commands the current backend supports.
|
||||||
@@ -353,3 +353,30 @@
|
|||||||
- New objective: raise the CLI benchmark from final live output rendering to the first real executable handlers for the live-compatible workflows `status`, `models list`, and `bench throughput` using an injectable transport seam.
|
- New objective: raise the CLI benchmark from final live output rendering to the first real executable handlers for the live-compatible workflows `status`, `models list`, and `bench throughput` using an injectable transport seam.
|
||||||
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
||||||
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
||||||
|
|
||||||
|
### Run 37: baseline - cli_surface_score=95.455 (KEEP)
|
||||||
|
- Timestamp: 2026-03-26 20:12
|
||||||
|
- What changed: Expanded the benchmark from live output rendering to an injectable live-execution seam, but left `execute_live_command_with()` as a placeholder that ignored the transport response and returned `LIVE EXEC {summary}`.
|
||||||
|
- Result: cli_surface_score=95.455, passed_cases=63, failed_cases=3, total_cases=66.
|
||||||
|
- Insight: The pipeline already had correct request construction and response rendering, but the executor still dropped the returned status/body instead of feeding them into the live renderer.
|
||||||
|
- Next: wire the transport response through `render_backend_response()` without changing any other CLI seam.
|
||||||
|
|
||||||
|
### Run 38: add cli live execution pipeline - cli_surface_score=100.0 (KEEP)
|
||||||
|
- Timestamp: 2026-03-26 20:13
|
||||||
|
- What changed: Implemented `execute_live_command_with()` in `crates/localllm-cli/src/main.rs` so live-compatible commands now call the injected transport with `BackendRequest`, then pass the returned `status_code` and `body` into `render_backend_response()`.
|
||||||
|
- Result: cli_surface_score=100.0, passed_cases=66, failed_cases=0, total_cases=66.
|
||||||
|
- Insight: The CLI now completes the first fully executable end-to-end path for the live-compatible workflows, from parser to transport invocation to final rendered output.
|
||||||
|
- Next: move from transport-injected execution to the first concrete HTTP client seam for commands that can already hit the current backend directly.
|
||||||
|
|
||||||
|
## Segment 17: CLI ureq HTTP Transport
|
||||||
|
|
||||||
|
- Objective: replace the injectable fake transport with a concrete `ureq` HTTP client for `status`, `models list`, and `bench throughput`.
|
||||||
|
- Benchmark: `./autoresearch.sh` runs ignored `cli_surface_*` tests in `crates/localllm-cli/src/main.rs`.
|
||||||
|
- Keep/discard rule: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
||||||
|
|
||||||
|
### Run 39: add ureq http transport and execute_live_command - cli_surface_score=100.0 (KEEP)
|
||||||
|
- Timestamp: 2026-03-26 20:36
|
||||||
|
- What changed: Added `ureq` to workspace and `localllm-cli` deps. Implemented `execute_live_command(cli)` using `ureq` as the real HTTP transport, delegating to `execute_live_command_with()`. Updated `main()` to call `execute_live_command()` instead of the scaffold placeholder. Added 3 new ignored benchmark tests verifying the injectable seam correctly routes status/models/bench through the transport pipeline.
|
||||||
|
- Result: cli_surface_score=100.0, passed_cases=69, failed_cases=0, total_cases=69.
|
||||||
|
- Insight: The CLI now has a real HTTP client (`ureq`) wired end-to-end for live-compatible commands. The injectable transport seam remains available for testing without a running server. The CLI is functionally complete for the commands that the current backend already supports.
|
||||||
|
- Next: CLI is at natural saturation given the current backend. Further progress requires new admin/control-plane endpoints in the backend for scan, load, auth, config, and watch commands.
|
||||||
Reference in new issue
Block a user