chore: expand cli semantic benchmark coverage
This commit is contained in:
1 parent
d1dbc27031
commit
a3f5585700
4 files changed
+121
-5
No files matched your search
@@ -144,3 +144,22 @@
|
||||
- Benchmark driver: `./autoresearch.sh` now runs ignored CLI-surface tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
||||
- Keep/discard rule is test-driven: higher `cli_surface_score` wins while the regular non-ignored suite remains green.
|
||||
- This segment starts by benchmarking the command surface itself before deeper execution and backend-admin semantics are implemented.
|
||||
|
||||
### Run 15: baseline - cli_surface_score=18.182 (KEEP)
|
||||
- Timestamp: 2026-03-26 18:53
|
||||
- What changed: Initialized a new `crates/localllm-cli` crate with a minimal Clap-based CLI scaffold, switched the autoresearch harness from template tests to CLI surface tests, and defined the first ignored benchmark suite for professional command coverage.
|
||||
- Result: cli_surface_score=18.182, passed_cases=2, failed_cases=9, total_cases=11.
|
||||
- Insight: The scaffold already covers global flags plus `status` and `models list`, but almost the entire promised management surface is still missing.
|
||||
- Next: add the remaining top-level command groups and model/auth/config/bench/logs/doctor subcommands so the CLI surface matches the intended product contract before wiring execution internals.
|
||||
|
||||
### Run 16: expand cli command surface - cli_surface_score=100.0 (KEEP)
|
||||
- Timestamp: 2026-03-26 18:55
|
||||
- What changed: Expanded `crates/localllm-cli/src/main.rs` into a broad Clap parser with top-level `status`, `models`, `auth`, `config`, `bench`, `logs`, and `doctor` commands plus the expected model/auth/config/throughput subcommands and global flags.
|
||||
- Result: cli_surface_score=100.0, passed_cases=11, failed_cases=0, total_cases=11.
|
||||
- Insight: The professional CLI contract can be locked in quickly at the parser layer before execution internals exist; Clap gives a maintainable foundation for the full management surface.
|
||||
- Next: raise the benchmark from command presence to semantic correctness, starting with command-specific arguments, defaults, and output-contract expectations for the highest-value workflows.
|
||||
|
||||
## Segment 6: CLI Semantic Contracts
|
||||
- New objective: keep the CLI surface stable, but raise the benchmark so key commands preserve useful semantic detail such as defaults, high-value arguments, and summary/intention contracts.
|
||||
- Benchmark driver: `./autoresearch.sh` still runs ignored `localllm-cli` tests and reports `cli_surface_score`, `passed_cases`, `failed_cases`, and `total_cases`.
|
||||
- Keep/discard rule remains test-driven: higher `cli_surface_score` wins while the normal non-ignored suites remain green.
|
||||
Reference in new issue
Block a user