ForgeLM/docs/specs/adaptive-model-routing-sdd.md

# Adaptive Model Routing SDD

## Status
- Draft v1
- Scope: Tauri desktop app + Rust local inference runtime + React chat UI

## Objective
Refactor local model selection so the app defaults to **automatic startup choice** and **session-level manual override**, while adding TPS-based degradation detection and denser chat/home UI behavior.

## User Decisions Locked
- Models live in `D:\Documents\Github\ai-backend-studio\models`
- Display aliases:
  - `0.8B` => `Cooper`
  - `2B` => `Iron`
  - `4B` => `Steel`
- Main chat selector scope for this phase: **Cooper + Iron only**
- `Steel` remains available only in settings/internal flows for now
- Final behavior: **auto selection + manual override in chat**

## Product Goals
1. Remove model choice as a required startup decision for normal users.
2. Start the app with the best default model automatically.
3. Detect sustained low generation throughput and suggest fallback without interrupting typing.
4. Allow the user to override the effective model from chat.
5. Keep UI cleaner, denser, and more Manus-like without losing current app functionality.

## Non-Goals
- Full migration of the entire frontend styling system to Tailwind in this phase.
- Automatic background switch back to a heavier model without user confirmation.
- Exposing `Steel` in the primary chat selector.
- Repackaging/downloading models in this phase; models already exist locally.

## Current Architecture Summary

### Backend / Runtime
- Tauri command surface: `tauri-app/src-tauri/src/lib.rs`
- Runtime bootstrap/start/stop: `tauri-app/src-tauri/src/runtime/mod.rs`
- Model discovery/indexing: `tauri-app/src-tauri/src/model_files.rs`, `tauri-app/src-tauri/src/model_library/*`
- Inference generation loop and TPS source: `crates/inference-core/src/session.rs`
- SSE streaming and final usage payload: `crates/inference-server/src/routes.rs`

### Frontend
- App composition and model swap wiring: `tauri-app/src/App.tsx`
- Chat screen: `tauri-app/src/components/ChatView.tsx`
- Current manual model selector: `tauri-app/src/components/Sidebar.tsx`
- Runtime draft model selection helpers: `tauri-app/src/features/runtime/runtime-settings-form.ts`
- Live model list fetch + current manual-lock behavior: `tauri-app/src/hooks/useModels.ts`

## Core Design Decisions

### 1) Separate Preferred Model from Effective Model
The system will track two different concepts:

- **Preferred model**: the user or app default for startup (`Iron` by default when available)
- **Effective model**: the model currently serving the active session/chat

This prevents fallback events from silently rewriting the user preference.

### 2) Automatic Startup Choice
At app launch, local runtime selection will follow this order:

1. If the user explicitly saved a preferred startup model and it is available, use it.
2. Otherwise prefer `Iron`.
3. If `Iron` is unavailable, use `Cooper`.
4. `Steel` is ignored by normal auto-start policy in this phase.

### 3) TPS-Based Degradation Warning
The system will monitor **generation TPS** during active completion streaming.

Fallback trigger rule:
- track rolling generation TPS samples in Rust
- if rolling TPS stays `< 30` for `>= 5s` continuously
- emit a degradation event once per request/session window
- show a non-blocking toast in the frontend

Toast actions:
- `Trocar para Cooper`
- `Manter Iron`

If the effective model is already `Cooper`, no fallback toast is shown.

### 4) Manual Override in Chat
The primary chat surface will expose a **small model selector** near the composer/header with:
- `Iron`
- `Cooper`

Behavior:
- changing the selector swaps only the **effective model**
- the chosen value persists for the current app session
- it does not silently rewrite startup preference unless the user later saves it in settings

### 5) Tailwind Rule for This Feature
The app currently relies heavily on CSS files. For this feature:
- all **new UI surfaces/components** should be built Tailwind-first
- modified legacy surfaces may temporarily keep existing CSS where a full migration would create unnecessary churn
- any touched layout should move toward utility-class ownership when practical

This keeps delivery realistic while respecting the new direction.

## Model Inventory and Alias Policy

### Canonical alias table
| Raw family | Display name | Phase role |
|---|---|---|
| 0.8B | Cooper | fallback + lightweight manual choice |
| 2B | Iron | default primary model |
| 4B | Steel | settings/internal only |

### Resolution rule
Model display naming must be computed from local file metadata/path patterns, not hardcoded to exact filenames only. Matching should allow future file suffix changes.

Proposed matcher priority:
1. explicit config mapping if present
2. known filename pattern match (`0.8b`, `2b`, `4b`)
3. raw file basename fallback

## Proposed Architecture

### A. Rust: TPS Telemetry Layer
Add a lightweight TPS monitor around the generation loop.

#### Source of truth
- `crates/inference-core/src/session.rs`

#### New concept
`GenerationPerformanceSnapshot`
- `request_id`
- `model_key`
- `generated_tokens`
- `generation_elapsed_ms`
- `rolling_tps`
- `final_tps`
- `degradation_state`

#### Rules
- begin measuring on first generated token, not prefill
- maintain a rolling window for generation throughput
- emit state transitions only when entering degraded mode
- keep final TPS metric in existing usage payload path

#### Why here
`GenerationSession` already owns generation timing and final TPS, so adding rolling TPS there avoids duplicate timing logic in the frontend.

### B. Rust: Runtime Auto-Selection Policy
Add startup model resolution inside runtime bootstrap/start flow.

#### Target
- `tauri-app/src-tauri/src/runtime/mod.rs`

#### New policy function
`resolve_startup_model(config, discovered_models) -> EffectiveStartupModel`

#### Inputs
- saved runtime config
- discovered model files from `./models`
- alias classification (`Cooper`/`Iron`/`Steel`)

#### Outputs
- resolved model path
- resolved display alias
- reason (`saved_preference`, `default_iron`, `fallback_cooper`, `missing_saved_model`)

### C. Frontend: Effective Model Controller
Create a dedicated client-side state layer above `useModels()`.

#### Responsibilities
- maintain `preferredModel`, `effectiveModel`, `manualOverride`, `degradationDismissed`
- map backend model IDs/paths to display aliases
- handle user-triggered model switches
- ignore refresh noise from `/v1/models` once an override is active for the session

#### Likely placement
- new hook near `tauri-app/src/hooks/` or `src/features/runtime/`
- wired in `tauri-app/src/App.tsx`

### D. Frontend: Toast System
No shared toast system exists today.

Add:
- global toast provider near `App.tsx`
- bottom-right stack
- `role="status"`, `aria-live="polite"`
- dedupe by event key (`low-tps-fallback:<request_id>:<model>`)

Toast content:
> O modelo atual está lento neste dispositivo. Deseja trocar para Cooper para melhor velocidade?

Actions:
- primary: swap to `Cooper`
- secondary: keep current model

### E. Frontend: Chat Selector
Move primary user model choice from sidebar-only behavior to an in-chat control.

#### UX rules
- visible but subtle
- available on initial screen and active chat screen
- only shows `Iron` and `Cooper`
- `Steel` remains in settings/runtime management
- while swapping, disable control and show compact busy state

### F. UI Density Updates

#### Home
- reduce vertical whitespace between hero, chips, and lower card
- improve recent work density in sidebar/task list
- keep one muted lower card instead of multiple competing blocks

#### Chat
- denser message header/meta
- keep TPS/status info compact and contextual
- do not overload first paint with debug controls

## Data and State Model

### Backend persisted config
Extend runtime config with explicit preference fields:
- `preferred_startup_model_alias: Option<String>` (`iron`, `cooper`, `steel`)
- keep existing `model_path` for actual runtime launch target

### Frontend session state
- `preferredModelAlias`
- `effectiveModelAlias`
- `effectiveModelPath`
- `overrideSource: "auto" | "manual" | "fallback"`
- `degradationPromptState: "idle" | "shown" | "accepted" | "dismissed"`

## Main User Flows

### Flow 1: App startup
1. app bootstraps runtime config
2. runtime scans available local models
3. startup resolver selects `Iron` if available, else `Cooper`
4. runtime starts with resolved model
5. frontend shows resolved effective model alias

### Flow 2: Sustained low TPS while using Iron
1. user starts generation on `Iron`
2. Rust rolling monitor observes TPS `< 30` for 5 consecutive seconds
3. backend emits degradation event
4. frontend displays one toast
5. user chooses:
   - `Trocar para Cooper` => perform hot swap, update effective model, persist session override
   - `Manter Iron` => suppress repeated prompt for the current request/session cooldown

### Flow 3: Manual chat override
1. user opens selector in chat
2. chooses `Cooper` or `Iron`
3. frontend calls swap command
4. runtime restarts with target model
5. effective model updates immediately after success

### Flow 4: Steel in settings
1. user opens settings/runtime area
2. sees full model inventory including `Steel`
3. may still use/import/manage `Steel`
4. Steel is not shown in chat selector in this phase

## Error Handling
- if swap fails, keep prior effective model visible
- if degradation event arrives during an in-progress swap, discard it
- if `Iron` disappears from library, auto-fallback to `Cooper`
- if neither `Iron` nor `Cooper` exists, show runtime unavailable state and keep settings path accessible
- if toast already shown for the same request, do not spawn duplicates

## Implementation Touchpoints

### Rust
- `crates/inference-core/src/session.rs`
- `crates/inference-server/src/routes.rs`
- `tauri-app/src-tauri/src/runtime/mod.rs`
- potentially `tauri-app/src-tauri/src/lib.rs` for new commands/events

### Frontend
- `tauri-app/src/App.tsx`
- `tauri-app/src/hooks/useModels.ts`
- `tauri-app/src/components/ChatView.tsx`
- `tauri-app/src/components/Sidebar.tsx`
- `tauri-app/src/features/runtime/runtime-settings-form.ts`
- new toast/provider files
- new model alias resolver/shared hook files

## TDD Test Matrix

### Rust unit tests
1. startup resolver picks `Iron` when `Iron` + `Cooper` are available
2. startup resolver picks `Cooper` when `Iron` is missing
3. startup resolver respects explicit saved preference when available
4. rolling TPS monitor enters degraded state only after 5s sustained `< 30`
5. transient dip below 30 for less than threshold does not trigger
6. degraded state is emitted once per request transition

### Rust integration tests
1. SSE/final usage still includes final TPS after monitor changes
2. degradation event payload contains model alias/path/request id
3. swap command updates effective runtime target correctly

### Frontend unit tests
1. alias resolver maps model file names to `Cooper`, `Iron`, `Steel`
2. chat selector only exposes `Cooper` and `Iron`
3. manual override survives model list refresh in session
4. fallback toast dedupes repeated degradation events
5. clicking `Trocar para Cooper` triggers swap and updates effective model
6. clicking `Manter Iron` suppresses repeated prompt during cooldown

### Frontend component tests
1. chat header/composer shows current effective model alias
2. initial screen remains dense and usable with toast visible
3. selector disabled while swap is in progress
4. runtime unavailable state keeps settings access reachable

## Acceptance Criteria
1. User can start the app without manually choosing a model.
2. App defaults to `Iron` when available.
3. If generation TPS stays below 30 for 5s on `Iron`, a non-blocking toast is shown.
4. User can switch to `Cooper` directly from the toast.
5. User can manually choose between `Iron` and `Cooper` from chat.
6. Manual override persists during the session.
7. `Steel` is visible only in settings/internal runtime management.
8. Final UI is denser and cleaner than current state.
9. New behavior is covered by Rust + frontend tests before implementation code lands.

## Risks
- current frontend styling is still CSS-heavy; partial Tailwind migration may create mixed ownership temporarily
- runtime restart latency may make “instant” switching feel slower on weaker machines
- model filename normalization may be inconsistent unless alias mapping is centralized
- event delivery from Rust to frontend must avoid duplicate or stale degradation prompts

## Recommended Delivery Phases
1. alias resolution + startup policy
2. Rust TPS degradation monitor + event contract
3. frontend toast system + session model controller
4. chat selector + sidebar/settings cleanup
5. density pass on home/chat UI
6. regression verification

## Open Assumptions
- existing hot-swap restart path is acceptable for “instantânea” within product expectations
- startup auto-selection should prefer `Iron` over `Steel` even if `Steel` exists
- `Steel` remains excluded from primary chat selector until a future phase expands the UX