ForgeLM/docs/specs/adaptive-model-routing-sdd.md

Adaptive Model Routing SDD

Status

• Draft v1
• Scope: Tauri desktop app + Rust local inference runtime + React chat UI

Objective

Refactor local model selection so the app defaults to automatic startup choice and session-level manual override, while adding TPS-based degradation detection and denser chat/home UI behavior.

User Decisions Locked

• Models live in D:\Documents\Github\ai-backend-studio\models
• Display aliases:
  • 0.8B => Cooper
  • 2B => Iron
  • 4B => Steel
• Main chat selector scope for this phase: Cooper + Iron only
• Steel remains available only in settings/internal flows for now
• Final behavior: auto selection + manual override in chat

Product Goals

1. Remove model choice as a required startup decision for normal users.
2. Start the app with the best default model automatically.
3. Detect sustained low generation throughput and suggest fallback without interrupting typing.
4. Allow the user to override the effective model from chat.
5. Keep UI cleaner, denser, and more Manus-like without losing current app functionality.

Non-Goals

• Full migration of the entire frontend styling system to Tailwind in this phase.
• Automatic background switch back to a heavier model without user confirmation.
• Exposing Steel in the primary chat selector.
• Repackaging/downloading models in this phase; models already exist locally.

Current Architecture Summary

Backend / Runtime

• Tauri command surface: tauri-app/src-tauri/src/lib.rs
• Runtime bootstrap/start/stop: tauri-app/src-tauri/src/runtime/mod.rs
• Model discovery/indexing: tauri-app/src-tauri/src/model_files.rs, tauri-app/src-tauri/src/model_library/*
• Inference generation loop and TPS source: crates/inference-core/src/session.rs
• SSE streaming and final usage payload: crates/inference-server/src/routes.rs

Frontend

• App composition and model swap wiring: tauri-app/src/App.tsx
• Chat screen: tauri-app/src/components/ChatView.tsx
• Current manual model selector: tauri-app/src/components/Sidebar.tsx
• Runtime draft model selection helpers: tauri-app/src/features/runtime/runtime-settings-form.ts
• Live model list fetch + current manual-lock behavior: tauri-app/src/hooks/useModels.ts

Core Design Decisions

1) Separate Preferred Model from Effective Model

The system will track two different concepts:

• Preferred model: the user or app default for startup (Iron by default when available)
• Effective model: the model currently serving the active session/chat

This prevents fallback events from silently rewriting the user preference.

2) Automatic Startup Choice

At app launch, local runtime selection will follow this order:

1. If the user explicitly saved a preferred startup model and it is available, use it.
2. Otherwise prefer Iron.
3. If Iron is unavailable, use Cooper.
4. Steel is ignored by normal auto-start policy in this phase.

3) TPS-Based Degradation Warning

The system will monitor generation TPS during active completion streaming.

Fallback trigger rule:

• track rolling generation TPS samples in Rust
• if rolling TPS stays < 30 for >= 5s continuously
• emit a degradation event once per request/session window
• show a non-blocking toast in the frontend

Toast actions:

• Trocar para Cooper
• Manter Iron

If the effective model is already Cooper, no fallback toast is shown.

4) Manual Override in Chat

The primary chat surface will expose a small model selector near the composer/header with:

• Iron
• Cooper

Behavior:

• changing the selector swaps only the effective model
• the chosen value persists for the current app session
• it does not silently rewrite startup preference unless the user later saves it in settings

5) Tailwind Rule for This Feature

The app currently relies heavily on CSS files. For this feature:

• all new UI surfaces/components should be built Tailwind-first
• modified legacy surfaces may temporarily keep existing CSS where a full migration would create unnecessary churn
• any touched layout should move toward utility-class ownership when practical

This keeps delivery realistic while respecting the new direction.

Model Inventory and Alias Policy

Canonical alias table

Raw family	Display name	Phase role
0.8B	Cooper	fallback + lightweight manual choice
2B	Iron	default primary model
4B	Steel	settings/internal only

Resolution rule

Model display naming must be computed from local file metadata/path patterns, not hardcoded to exact filenames only. Matching should allow future file suffix changes.

Proposed matcher priority:

1. explicit config mapping if present
2. known filename pattern match (0.8b, 2b, 4b)
3. raw file basename fallback

Proposed Architecture

A. Rust: TPS Telemetry Layer

Add a lightweight TPS monitor around the generation loop.

Source of truth

• crates/inference-core/src/session.rs

New concept

GenerationPerformanceSnapshot

• request_id
• model_key
• generated_tokens
• generation_elapsed_ms
• rolling_tps
• final_tps
• degradation_state

Rules

• begin measuring on first generated token, not prefill
• maintain a rolling window for generation throughput
• emit state transitions only when entering degraded mode
• keep final TPS metric in existing usage payload path

Why here

GenerationSession already owns generation timing and final TPS, so adding rolling TPS there avoids duplicate timing logic in the frontend.

B. Rust: Runtime Auto-Selection Policy

Add startup model resolution inside runtime bootstrap/start flow.

Target

• tauri-app/src-tauri/src/runtime/mod.rs

New policy function

resolve_startup_model(config, discovered_models) -> EffectiveStartupModel

Inputs

• saved runtime config
• discovered model files from ./models
• alias classification (Cooper/Iron/Steel)

Outputs

• resolved model path
• resolved display alias
• reason (saved_preference, default_iron, fallback_cooper, missing_saved_model)

C. Frontend: Effective Model Controller

Create a dedicated client-side state layer above useModels().

Responsibilities

• maintain preferredModel, effectiveModel, manualOverride, degradationDismissed
• map backend model IDs/paths to display aliases
• handle user-triggered model switches
• ignore refresh noise from /v1/models once an override is active for the session

Likely placement

• new hook near tauri-app/src/hooks/ or src/features/runtime/
• wired in tauri-app/src/App.tsx

D. Frontend: Toast System

No shared toast system exists today.

Add:

• global toast provider near App.tsx
• bottom-right stack
• role="status", aria-live="polite"
• dedupe by event key (low-tps-fallback:<request_id>:<model>)

Toast content:

O modelo atual está lento neste dispositivo. Deseja trocar para Cooper para melhor velocidade?

Actions:

• primary: swap to Cooper
• secondary: keep current model

E. Frontend: Chat Selector

Move primary user model choice from sidebar-only behavior to an in-chat control.

UX rules

• visible but subtle
• available on initial screen and active chat screen
• only shows Iron and Cooper
• Steel remains in settings/runtime management
• while swapping, disable control and show compact busy state

F. UI Density Updates

Home

• reduce vertical whitespace between hero, chips, and lower card
• improve recent work density in sidebar/task list
• keep one muted lower card instead of multiple competing blocks

Chat

• denser message header/meta
• keep TPS/status info compact and contextual
• do not overload first paint with debug controls

Data and State Model

Backend persisted config

Extend runtime config with explicit preference fields:

• preferred_startup_model_alias: Option<String> (iron, cooper, steel)
• keep existing model_path for actual runtime launch target

Frontend session state

• preferredModelAlias
• effectiveModelAlias
• effectiveModelPath
• overrideSource: "auto" | "manual" | "fallback"
• degradationPromptState: "idle" | "shown" | "accepted" | "dismissed"

Main User Flows

Flow 1: App startup

1. app bootstraps runtime config
2. runtime scans available local models
3. startup resolver selects Iron if available, else Cooper
4. runtime starts with resolved model
5. frontend shows resolved effective model alias

Flow 2: Sustained low TPS while using Iron

1. user starts generation on Iron
2. Rust rolling monitor observes TPS < 30 for 5 consecutive seconds
3. backend emits degradation event
4. frontend displays one toast
5. user chooses:
   • Trocar para Cooper => perform hot swap, update effective model, persist session override
   • Manter Iron => suppress repeated prompt for the current request/session cooldown

Flow 3: Manual chat override

1. user opens selector in chat
2. chooses Cooper or Iron
3. frontend calls swap command
4. runtime restarts with target model
5. effective model updates immediately after success

Flow 4: Steel in settings

1. user opens settings/runtime area
2. sees full model inventory including Steel
3. may still use/import/manage Steel
4. Steel is not shown in chat selector in this phase

Error Handling

• if swap fails, keep prior effective model visible
• if degradation event arrives during an in-progress swap, discard it
• if Iron disappears from library, auto-fallback to Cooper
• if neither Iron nor Cooper exists, show runtime unavailable state and keep settings path accessible
• if toast already shown for the same request, do not spawn duplicates

Implementation Touchpoints

Rust

• crates/inference-core/src/session.rs
• crates/inference-server/src/routes.rs
• tauri-app/src-tauri/src/runtime/mod.rs
• potentially tauri-app/src-tauri/src/lib.rs for new commands/events

Frontend

• tauri-app/src/App.tsx
• tauri-app/src/hooks/useModels.ts
• tauri-app/src/components/ChatView.tsx
• tauri-app/src/components/Sidebar.tsx
• tauri-app/src/features/runtime/runtime-settings-form.ts
• new toast/provider files
• new model alias resolver/shared hook files

TDD Test Matrix

Rust unit tests

1. startup resolver picks Iron when Iron + Cooper are available
2. startup resolver picks Cooper when Iron is missing
3. startup resolver respects explicit saved preference when available
4. rolling TPS monitor enters degraded state only after 5s sustained < 30
5. transient dip below 30 for less than threshold does not trigger
6. degraded state is emitted once per request transition

Rust integration tests

1. SSE/final usage still includes final TPS after monitor changes
2. degradation event payload contains model alias/path/request id
3. swap command updates effective runtime target correctly

Frontend unit tests

1. alias resolver maps model file names to Cooper, Iron, Steel
2. chat selector only exposes Cooper and Iron
3. manual override survives model list refresh in session
4. fallback toast dedupes repeated degradation events
5. clicking Trocar para Cooper triggers swap and updates effective model
6. clicking Manter Iron suppresses repeated prompt during cooldown

Frontend component tests

1. chat header/composer shows current effective model alias
2. initial screen remains dense and usable with toast visible
3. selector disabled while swap is in progress
4. runtime unavailable state keeps settings access reachable

Acceptance Criteria

1. User can start the app without manually choosing a model.
2. App defaults to Iron when available.
3. If generation TPS stays below 30 for 5s on Iron, a non-blocking toast is shown.
4. User can switch to Cooper directly from the toast.
5. User can manually choose between Iron and Cooper from chat.
6. Manual override persists during the session.
7. Steel is visible only in settings/internal runtime management.
8. Final UI is denser and cleaner than current state.
9. New behavior is covered by Rust + frontend tests before implementation code lands.

Risks

• current frontend styling is still CSS-heavy; partial Tailwind migration may create mixed ownership temporarily
• runtime restart latency may make “instant” switching feel slower on weaker machines
• model filename normalization may be inconsistent unless alias mapping is centralized
• event delivery from Rust to frontend must avoid duplicate or stale degradation prompts

Recommended Delivery Phases

1. alias resolution + startup policy
2. Rust TPS degradation monitor + event contract
3. frontend toast system + session model controller
4. chat selector + sidebar/settings cleanup
5. density pass on home/chat UI
6. regression verification

Open Assumptions

• existing hot-swap restart path is acceptable for “instantânea” within product expectations
• startup auto-selection should prefer Iron over Steel even if Steel exists
• Steel remains excluded from primary chat selector until a future phase expands the UX