feat: add cli backend invocation specs
This commit is contained in:
1 parent
3969f12f26
commit
f538fc60bb
4 files changed
+108
-8
No files matched your search
@@ -220,3 +220,60 @@
|
||||
- New objective: raise the CLI benchmark from abstract execution contracts to concrete execution stubs that preserve HTTP method/path/body semantics or named local actions for the highest-value workflows.
|
||||
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
||||
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
||||
|
||||
### Run 23: baseline - cli_surface_score=83.333 (KEEP)
|
||||
- Timestamp: 2026-03-26 19:20
|
||||
- What changed: Expanded the benchmark from execution contracts to concrete execution stubs, adding ignored tests for HTTP method/path/body semantics and named local actions for status, model scan/load, auth enable, throughput bench, and logs.
|
||||
- Result: cli_surface_score=83.333, passed_cases=30, failed_cases=6, total_cases=36.
|
||||
- Insight: The CLI already knows how to parse commands and describe execution contracts, but `execution_stub()` is still just a placeholder returning `SummaryOnly { command: "unimplemented" }` for every high-value workflow.
|
||||
- Next: implement concrete stub mapping so status and throughput produce execution-ready HTTP request shapes and local workflows emit named local actions with preserved detail.
|
||||
|
||||
### Run 24: add cli execution stubs - cli_surface_score=100.0 (KEEP)
|
||||
- Timestamp: 2026-03-26 19:22
|
||||
- What changed: Implemented concrete `execution_stub()` mapping in `crates/localllm-cli/src/main.rs`, turning the highest-value execution contracts into execution-ready HTTP requests and named local actions with preserved detail.
|
||||
- Result: cli_surface_score=100.0, passed_cases=36, failed_cases=0, total_cases=36.
|
||||
- Insight: The CLI now has a full design ladder for the key workflows - parser, semantic summary, machine-usable plan, execution contract, and finally execution stub - which is a strong handoff point for wiring real handlers next.
|
||||
- Next: raise the benchmark from stub shape to actual handler behavior or request payload construction for the same high-value workflows.
|
||||
|
||||
## Segment 10: CLI Execution Previews
|
||||
- New objective: raise the CLI benchmark from execution stubs to dry-run previews that expand the same high-value workflows into full previewable HTTP URLs or local-effect descriptions.
|
||||
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
||||
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
||||
|
||||
### Run 25: baseline - cli_surface_score=85.714 (KEEP)
|
||||
- Timestamp: 2026-03-26 19:27
|
||||
- What changed: Expanded the benchmark from execution stubs to dry-run previews, adding ignored tests for full preview URLs and human-readable local-effect descriptions across status, models scan/load, auth enable, throughput bench, and logs.
|
||||
- Result: cli_surface_score=85.714, passed_cases=36, failed_cases=6, total_cases=42.
|
||||
- Insight: The CLI already models execution stubs correctly, but `execution_preview()` still collapses every high-value workflow to `SummaryOnly { command: "preview-unimplemented" }`, so there is no user-facing dry-run layer yet.
|
||||
- Next: map the highest-value execution stubs into concrete `HttpPreview` and `LocalPreview` values using the resolved `--server` base URL and readable local-effect text.
|
||||
|
||||
### Run 26: add cli execution previews - cli_surface_score=100.0 (KEEP)
|
||||
- Timestamp: 2026-03-26 19:28
|
||||
- What changed: Implemented concrete `execution_preview()` mapping in `crates/localllm-cli/src/main.rs`, expanding HTTP stubs into full URLs with preserved payloads and local actions into readable effect descriptions via `preview_effect()`.
|
||||
- Result: cli_surface_score=100.0, passed_cases=42, failed_cases=0, total_cases=42.
|
||||
- Insight: The CLI now has a complete dry-run preview layer for the highest-value workflows, which is a strong bridge from internal planning seams to future real handlers and `--dry-run` UX.
|
||||
- Next: raise the benchmark from preview shape to actual handler behavior or request-body construction details for status, throughput, and model-management workflows.
|
||||
|
||||
## Segment 11: CLI Dry-Run Output
|
||||
- New objective: raise the CLI benchmark from internal preview objects to final user-facing `--dry-run` output in both `text` and `json`, so the same high-value workflows are ready for a real preview UX at the command line.
|
||||
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
||||
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
||||
|
||||
### Run 27: baseline - cli_surface_score=87.5 (KEEP)
|
||||
- Timestamp: 2026-03-26 19:32
|
||||
- What changed: Expanded the benchmark from internal `ExecutionPreview` objects to final `--dry-run` UX in both text and JSON, while keeping `render_execution_preview()` intentionally shallow as `DRY RUN {summary}`.
|
||||
- Result: cli_surface_score=87.5, passed_cases=42, failed_cases=6, total_cases=48.
|
||||
- Insight: The CLI already knows how to build the preview object graph, but the final presentation layer still collapses to summaries instead of rendering health/chat HTTP previews and local-effect dry-run text.
|
||||
- Next: render the existing preview layer into final text/json output using the current `ExecutionPreview` data rather than introducing a new execution seam.
|
||||
|
||||
### Run 28: render cli dry-run previews - cli_surface_score=100.0 (KEEP)
|
||||
- Timestamp: 2026-03-26 19:34
|
||||
- What changed: Implemented final `--dry-run` rendering in `crates/localllm-cli/src/main.rs`, mapping `ExecutionPreview` values to user-facing text and JSON output and reusing the existing preview pipeline (`plan_command` -> `execution_contract` -> `execution_stub` -> `execution_preview`).
|
||||
- Result: cli_surface_score=100.0, passed_cases=48, failed_cases=0, total_cases=48.
|
||||
- Insight: The CLI now has a complete dry-run stack for the highest-value workflows, from parser through final user-visible text/json previews, without yet executing real side effects.
|
||||
- Next: raise the bar from dry-run rendering to real handler behavior or deterministic request/body construction for backend-facing commands.
|
||||
|
||||
## Segment 12: CLI Backend Invocations
|
||||
- New objective: raise the CLI benchmark from dry-run rendering to the first real backend-call specifications for commands that can already work today, especially `status`, `models list`, and `bench throughput`.
|
||||
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
||||
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
||||
Reference in new issue
Block a user