chore: expand cli backend response coverage

This commit is contained in:
Evandro Fonseca Junior committed 2026-03-26 20:01:09 -03:00
1 parent e313c88bdb
commit bf985ea9ea
7 files changed
+341 -9

No files matched your search

+38
View File
@@ -277,3 +277,41 @@
- New objective: raise the CLI benchmark from dry-run rendering to the first real backend-call specifications for commands that can already work today, especially `status`, `models list`, and `bench throughput`.
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
## Segment 13: CLI Backend Requests
- New objective: raise the CLI benchmark from backend invocation specs to executable backend request construction for the commands that already align with live server endpoints today.
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
### Run 29: baseline - cli_surface_score=94.118 (KEEP)
- Timestamp: 2026-03-26 19:39
- What changed: Expanded the benchmark from final dry-run rendering to backend invocation specs for commands that can already work today, adding ignored tests for live `GET /health`, live `GET /v1/models`, and a throughput `POST /v1/chat/completions` request that preserves the selected model and token budget.
- Result: cli_surface_score=94.118, passed_cases=48, failed_cases=3, total_cases=51.
- Insight: The CLI already handled every previous parser/planning/dry-run layer, but `backend_invocation()` was still a placeholder, and `models list` had not yet been promoted into the real backend-facing seam.
- Next: implement concrete backend invocation specs for `status`, `models list`, and `bench throughput` using the resolved server base URL and auth presence.
### Run 30: add cli backend invocation specs - cli_surface_score=100.0 (KEEP)
- Timestamp: 2026-03-26 19:41
- What changed: Implemented `backend_invocation(&Cli)` in `crates/localllm-cli/src/main.rs` so `status` maps to live `GET /health`, `models list` maps to live `GET /v1/models`, and `bench throughput` maps to a live `POST /v1/chat/completions` invocation that preserves `--model`, samples, token budget, and auth presence.
- Result: cli_surface_score=100.0, passed_cases=51, failed_cases=0, total_cases=51.
- Insight: The CLI now crosses from dry-run-only UX into concrete backend-facing invocation specs for commands that can already work against the current server today, without needing the future control-plane endpoints yet.
- Next: raise the bar from backend invocation specs to actual request payload construction or the first executable handlers for these already-supported commands.
### Run 31: baseline - cli_surface_score=94.444 (KEEP)
- Timestamp: 2026-03-26 19:47
- What changed: Expanded the benchmark from backend invocation specs to executable backend request construction, requiring final HTTP method/url/header/body request objects with deterministic JSON payloads for `status`, `models list`, and `bench throughput`.
- Result: cli_surface_score=94.444, passed_cases=51, failed_cases=3, total_cases=54.
- Insight: The CLI already had parser, summaries, plans, contracts, stubs, previews, dry-run rendering, and invocation specs, but `backend_request(&Cli)` still returned `SummaryOnly { command: "request-unimplemented" }` instead of final request objects.
- Next: map `BackendInvocation::HttpCall` into executable `BackendRequest::HttpRequest` values with bearer headers, JSON content-type, and deterministic benchmark payload generation.
### Run 32: add cli backend request specs - cli_surface_score=100.0 (KEEP)
- Timestamp: 2026-03-26 19:49
- What changed: Implemented `backend_request(&Cli)` in `crates/localllm-cli/src/main.rs` plus helpers for bearer headers, JSON headers, deterministic benchmark prompt construction, and exact chat payload generation for throughput requests.
- Result: cli_surface_score=100.0, passed_cases=54, failed_cases=0, total_cases=54.
- Insight: The first live-compatible CLI workflows now preserve the full executable backend request boundary - method, URL, auth headers, content-type, and deterministic JSON body - while still remaining execution-free.
- Next: raise the bar from request construction to the first live response-handling or executable handler seams for the same workflows.
## Segment 14: CLI Backend Responses
- New objective: raise the CLI benchmark from executable backend request construction to backend response interpretation for the same live-compatible workflows, starting with `status`, `models list`, and `bench throughput`.
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.