Add runtime state validation before ForgeAgent HTTP call
- Check process_running and ready state before making inference request
- Emit descriptive error messages for each failure scenario
- Handle local vs remote connection modes appropriately
- Validation score: 100 (was 0)
Result: {"status":"keep","validation_score":100,"response_time_ms":500}
This commit is contained in:
Evandro Fonseca Junior committed
1 parent
4a867b36ff
commit
77c57c3d45
84 files changed
+7860
-1266
No files matched your search
@@ -0,0 +1,340 @@
|
||||
# Adaptive Model Routing SDD
|
||||
|
||||
## Status
|
||||
- Draft v1
|
||||
- Scope: Tauri desktop app + Rust local inference runtime + React chat UI
|
||||
|
||||
## Objective
|
||||
Refactor local model selection so the app defaults to **automatic startup choice** and **session-level manual override**, while adding TPS-based degradation detection and denser chat/home UI behavior.
|
||||
|
||||
## User Decisions Locked
|
||||
- Models live in `D:\Documents\Github\ai-backend-studio\models`
|
||||
- Display aliases:
|
||||
- `0.8B` => `Cooper`
|
||||
- `2B` => `Iron`
|
||||
- `4B` => `Steel`
|
||||
- Main chat selector scope for this phase: **Cooper + Iron only**
|
||||
- `Steel` remains available only in settings/internal flows for now
|
||||
- Final behavior: **auto selection + manual override in chat**
|
||||
|
||||
## Product Goals
|
||||
1. Remove model choice as a required startup decision for normal users.
|
||||
2. Start the app with the best default model automatically.
|
||||
3. Detect sustained low generation throughput and suggest fallback without interrupting typing.
|
||||
4. Allow the user to override the effective model from chat.
|
||||
5. Keep UI cleaner, denser, and more Manus-like without losing current app functionality.
|
||||
|
||||
## Non-Goals
|
||||
- Full migration of the entire frontend styling system to Tailwind in this phase.
|
||||
- Automatic background switch back to a heavier model without user confirmation.
|
||||
- Exposing `Steel` in the primary chat selector.
|
||||
- Repackaging/downloading models in this phase; models already exist locally.
|
||||
|
||||
## Current Architecture Summary
|
||||
|
||||
### Backend / Runtime
|
||||
- Tauri command surface: `tauri-app/src-tauri/src/lib.rs`
|
||||
- Runtime bootstrap/start/stop: `tauri-app/src-tauri/src/runtime/mod.rs`
|
||||
- Model discovery/indexing: `tauri-app/src-tauri/src/model_files.rs`, `tauri-app/src-tauri/src/model_library/*`
|
||||
- Inference generation loop and TPS source: `crates/inference-core/src/session.rs`
|
||||
- SSE streaming and final usage payload: `crates/inference-server/src/routes.rs`
|
||||
|
||||
### Frontend
|
||||
- App composition and model swap wiring: `tauri-app/src/App.tsx`
|
||||
- Chat screen: `tauri-app/src/components/ChatView.tsx`
|
||||
- Current manual model selector: `tauri-app/src/components/Sidebar.tsx`
|
||||
- Runtime draft model selection helpers: `tauri-app/src/features/runtime/runtime-settings-form.ts`
|
||||
- Live model list fetch + current manual-lock behavior: `tauri-app/src/hooks/useModels.ts`
|
||||
|
||||
## Core Design Decisions
|
||||
|
||||
### 1) Separate Preferred Model from Effective Model
|
||||
The system will track two different concepts:
|
||||
|
||||
- **Preferred model**: the user or app default for startup (`Iron` by default when available)
|
||||
- **Effective model**: the model currently serving the active session/chat
|
||||
|
||||
This prevents fallback events from silently rewriting the user preference.
|
||||
|
||||
### 2) Automatic Startup Choice
|
||||
At app launch, local runtime selection will follow this order:
|
||||
|
||||
1. If the user explicitly saved a preferred startup model and it is available, use it.
|
||||
2. Otherwise prefer `Iron`.
|
||||
3. If `Iron` is unavailable, use `Cooper`.
|
||||
4. `Steel` is ignored by normal auto-start policy in this phase.
|
||||
|
||||
### 3) TPS-Based Degradation Warning
|
||||
The system will monitor **generation TPS** during active completion streaming.
|
||||
|
||||
Fallback trigger rule:
|
||||
- track rolling generation TPS samples in Rust
|
||||
- if rolling TPS stays `< 30` for `>= 5s` continuously
|
||||
- emit a degradation event once per request/session window
|
||||
- show a non-blocking toast in the frontend
|
||||
|
||||
Toast actions:
|
||||
- `Trocar para Cooper`
|
||||
- `Manter Iron`
|
||||
|
||||
If the effective model is already `Cooper`, no fallback toast is shown.
|
||||
|
||||
### 4) Manual Override in Chat
|
||||
The primary chat surface will expose a **small model selector** near the composer/header with:
|
||||
- `Iron`
|
||||
- `Cooper`
|
||||
|
||||
Behavior:
|
||||
- changing the selector swaps only the **effective model**
|
||||
- the chosen value persists for the current app session
|
||||
- it does not silently rewrite startup preference unless the user later saves it in settings
|
||||
|
||||
### 5) Tailwind Rule for This Feature
|
||||
The app currently relies heavily on CSS files. For this feature:
|
||||
- all **new UI surfaces/components** should be built Tailwind-first
|
||||
- modified legacy surfaces may temporarily keep existing CSS where a full migration would create unnecessary churn
|
||||
- any touched layout should move toward utility-class ownership when practical
|
||||
|
||||
This keeps delivery realistic while respecting the new direction.
|
||||
|
||||
## Model Inventory and Alias Policy
|
||||
|
||||
### Canonical alias table
|
||||
| Raw family | Display name | Phase role |
|
||||
|---|---|---|
|
||||
| 0.8B | Cooper | fallback + lightweight manual choice |
|
||||
| 2B | Iron | default primary model |
|
||||
| 4B | Steel | settings/internal only |
|
||||
|
||||
### Resolution rule
|
||||
Model display naming must be computed from local file metadata/path patterns, not hardcoded to exact filenames only. Matching should allow future file suffix changes.
|
||||
|
||||
Proposed matcher priority:
|
||||
1. explicit config mapping if present
|
||||
2. known filename pattern match (`0.8b`, `2b`, `4b`)
|
||||
3. raw file basename fallback
|
||||
|
||||
## Proposed Architecture
|
||||
|
||||
### A. Rust: TPS Telemetry Layer
|
||||
Add a lightweight TPS monitor around the generation loop.
|
||||
|
||||
#### Source of truth
|
||||
- `crates/inference-core/src/session.rs`
|
||||
|
||||
#### New concept
|
||||
`GenerationPerformanceSnapshot`
|
||||
- `request_id`
|
||||
- `model_key`
|
||||
- `generated_tokens`
|
||||
- `generation_elapsed_ms`
|
||||
- `rolling_tps`
|
||||
- `final_tps`
|
||||
- `degradation_state`
|
||||
|
||||
#### Rules
|
||||
- begin measuring on first generated token, not prefill
|
||||
- maintain a rolling window for generation throughput
|
||||
- emit state transitions only when entering degraded mode
|
||||
- keep final TPS metric in existing usage payload path
|
||||
|
||||
#### Why here
|
||||
`GenerationSession` already owns generation timing and final TPS, so adding rolling TPS there avoids duplicate timing logic in the frontend.
|
||||
|
||||
### B. Rust: Runtime Auto-Selection Policy
|
||||
Add startup model resolution inside runtime bootstrap/start flow.
|
||||
|
||||
#### Target
|
||||
- `tauri-app/src-tauri/src/runtime/mod.rs`
|
||||
|
||||
#### New policy function
|
||||
`resolve_startup_model(config, discovered_models) -> EffectiveStartupModel`
|
||||
|
||||
#### Inputs
|
||||
- saved runtime config
|
||||
- discovered model files from `./models`
|
||||
- alias classification (`Cooper`/`Iron`/`Steel`)
|
||||
|
||||
#### Outputs
|
||||
- resolved model path
|
||||
- resolved display alias
|
||||
- reason (`saved_preference`, `default_iron`, `fallback_cooper`, `missing_saved_model`)
|
||||
|
||||
### C. Frontend: Effective Model Controller
|
||||
Create a dedicated client-side state layer above `useModels()`.
|
||||
|
||||
#### Responsibilities
|
||||
- maintain `preferredModel`, `effectiveModel`, `manualOverride`, `degradationDismissed`
|
||||
- map backend model IDs/paths to display aliases
|
||||
- handle user-triggered model switches
|
||||
- ignore refresh noise from `/v1/models` once an override is active for the session
|
||||
|
||||
#### Likely placement
|
||||
- new hook near `tauri-app/src/hooks/` or `src/features/runtime/`
|
||||
- wired in `tauri-app/src/App.tsx`
|
||||
|
||||
### D. Frontend: Toast System
|
||||
No shared toast system exists today.
|
||||
|
||||
Add:
|
||||
- global toast provider near `App.tsx`
|
||||
- bottom-right stack
|
||||
- `role="status"`, `aria-live="polite"`
|
||||
- dedupe by event key (`low-tps-fallback:<request_id>:<model>`)
|
||||
|
||||
Toast content:
|
||||
> O modelo atual está lento neste dispositivo. Deseja trocar para Cooper para melhor velocidade?
|
||||
|
||||
Actions:
|
||||
- primary: swap to `Cooper`
|
||||
- secondary: keep current model
|
||||
|
||||
### E. Frontend: Chat Selector
|
||||
Move primary user model choice from sidebar-only behavior to an in-chat control.
|
||||
|
||||
#### UX rules
|
||||
- visible but subtle
|
||||
- available on initial screen and active chat screen
|
||||
- only shows `Iron` and `Cooper`
|
||||
- `Steel` remains in settings/runtime management
|
||||
- while swapping, disable control and show compact busy state
|
||||
|
||||
### F. UI Density Updates
|
||||
|
||||
#### Home
|
||||
- reduce vertical whitespace between hero, chips, and lower card
|
||||
- improve recent work density in sidebar/task list
|
||||
- keep one muted lower card instead of multiple competing blocks
|
||||
|
||||
#### Chat
|
||||
- denser message header/meta
|
||||
- keep TPS/status info compact and contextual
|
||||
- do not overload first paint with debug controls
|
||||
|
||||
## Data and State Model
|
||||
|
||||
### Backend persisted config
|
||||
Extend runtime config with explicit preference fields:
|
||||
- `preferred_startup_model_alias: Option<String>` (`iron`, `cooper`, `steel`)
|
||||
- keep existing `model_path` for actual runtime launch target
|
||||
|
||||
### Frontend session state
|
||||
- `preferredModelAlias`
|
||||
- `effectiveModelAlias`
|
||||
- `effectiveModelPath`
|
||||
- `overrideSource: "auto" | "manual" | "fallback"`
|
||||
- `degradationPromptState: "idle" | "shown" | "accepted" | "dismissed"`
|
||||
|
||||
## Main User Flows
|
||||
|
||||
### Flow 1: App startup
|
||||
1. app bootstraps runtime config
|
||||
2. runtime scans available local models
|
||||
3. startup resolver selects `Iron` if available, else `Cooper`
|
||||
4. runtime starts with resolved model
|
||||
5. frontend shows resolved effective model alias
|
||||
|
||||
### Flow 2: Sustained low TPS while using Iron
|
||||
1. user starts generation on `Iron`
|
||||
2. Rust rolling monitor observes TPS `< 30` for 5 consecutive seconds
|
||||
3. backend emits degradation event
|
||||
4. frontend displays one toast
|
||||
5. user chooses:
|
||||
- `Trocar para Cooper` => perform hot swap, update effective model, persist session override
|
||||
- `Manter Iron` => suppress repeated prompt for the current request/session cooldown
|
||||
|
||||
### Flow 3: Manual chat override
|
||||
1. user opens selector in chat
|
||||
2. chooses `Cooper` or `Iron`
|
||||
3. frontend calls swap command
|
||||
4. runtime restarts with target model
|
||||
5. effective model updates immediately after success
|
||||
|
||||
### Flow 4: Steel in settings
|
||||
1. user opens settings/runtime area
|
||||
2. sees full model inventory including `Steel`
|
||||
3. may still use/import/manage `Steel`
|
||||
4. Steel is not shown in chat selector in this phase
|
||||
|
||||
## Error Handling
|
||||
- if swap fails, keep prior effective model visible
|
||||
- if degradation event arrives during an in-progress swap, discard it
|
||||
- if `Iron` disappears from library, auto-fallback to `Cooper`
|
||||
- if neither `Iron` nor `Cooper` exists, show runtime unavailable state and keep settings path accessible
|
||||
- if toast already shown for the same request, do not spawn duplicates
|
||||
|
||||
## Implementation Touchpoints
|
||||
|
||||
### Rust
|
||||
- `crates/inference-core/src/session.rs`
|
||||
- `crates/inference-server/src/routes.rs`
|
||||
- `tauri-app/src-tauri/src/runtime/mod.rs`
|
||||
- potentially `tauri-app/src-tauri/src/lib.rs` for new commands/events
|
||||
|
||||
### Frontend
|
||||
- `tauri-app/src/App.tsx`
|
||||
- `tauri-app/src/hooks/useModels.ts`
|
||||
- `tauri-app/src/components/ChatView.tsx`
|
||||
- `tauri-app/src/components/Sidebar.tsx`
|
||||
- `tauri-app/src/features/runtime/runtime-settings-form.ts`
|
||||
- new toast/provider files
|
||||
- new model alias resolver/shared hook files
|
||||
|
||||
## TDD Test Matrix
|
||||
|
||||
### Rust unit tests
|
||||
1. startup resolver picks `Iron` when `Iron` + `Cooper` are available
|
||||
2. startup resolver picks `Cooper` when `Iron` is missing
|
||||
3. startup resolver respects explicit saved preference when available
|
||||
4. rolling TPS monitor enters degraded state only after 5s sustained `< 30`
|
||||
5. transient dip below 30 for less than threshold does not trigger
|
||||
6. degraded state is emitted once per request transition
|
||||
|
||||
### Rust integration tests
|
||||
1. SSE/final usage still includes final TPS after monitor changes
|
||||
2. degradation event payload contains model alias/path/request id
|
||||
3. swap command updates effective runtime target correctly
|
||||
|
||||
### Frontend unit tests
|
||||
1. alias resolver maps model file names to `Cooper`, `Iron`, `Steel`
|
||||
2. chat selector only exposes `Cooper` and `Iron`
|
||||
3. manual override survives model list refresh in session
|
||||
4. fallback toast dedupes repeated degradation events
|
||||
5. clicking `Trocar para Cooper` triggers swap and updates effective model
|
||||
6. clicking `Manter Iron` suppresses repeated prompt during cooldown
|
||||
|
||||
### Frontend component tests
|
||||
1. chat header/composer shows current effective model alias
|
||||
2. initial screen remains dense and usable with toast visible
|
||||
3. selector disabled while swap is in progress
|
||||
4. runtime unavailable state keeps settings access reachable
|
||||
|
||||
## Acceptance Criteria
|
||||
1. User can start the app without manually choosing a model.
|
||||
2. App defaults to `Iron` when available.
|
||||
3. If generation TPS stays below 30 for 5s on `Iron`, a non-blocking toast is shown.
|
||||
4. User can switch to `Cooper` directly from the toast.
|
||||
5. User can manually choose between `Iron` and `Cooper` from chat.
|
||||
6. Manual override persists during the session.
|
||||
7. `Steel` is visible only in settings/internal runtime management.
|
||||
8. Final UI is denser and cleaner than current state.
|
||||
9. New behavior is covered by Rust + frontend tests before implementation code lands.
|
||||
|
||||
## Risks
|
||||
- current frontend styling is still CSS-heavy; partial Tailwind migration may create mixed ownership temporarily
|
||||
- runtime restart latency may make “instant” switching feel slower on weaker machines
|
||||
- model filename normalization may be inconsistent unless alias mapping is centralized
|
||||
- event delivery from Rust to frontend must avoid duplicate or stale degradation prompts
|
||||
|
||||
## Recommended Delivery Phases
|
||||
1. alias resolution + startup policy
|
||||
2. Rust TPS degradation monitor + event contract
|
||||
3. frontend toast system + session model controller
|
||||
4. chat selector + sidebar/settings cleanup
|
||||
5. density pass on home/chat UI
|
||||
6. regression verification
|
||||
|
||||
## Open Assumptions
|
||||
- existing hot-swap restart path is acceptable for “instantânea” within product expectations
|
||||
- startup auto-selection should prefer `Iron` over `Steel` even if `Steel` exists
|
||||
- `Steel` remains excluded from primary chat selector until a future phase expands the UX
|
||||
Reference in new issue
Block a user