151 lines
8.1 KiB
Markdown
151 lines
8.1 KiB
Markdown
# Architecture
|
|
|
|
**Analysis Date:** 2026-03-28
|
|
|
|
## Pattern Overview
|
|
|
|
**Overall:** Local-first polyglot workspace with shared contracts and three user surfaces
|
|
|
|
**Key Characteristics:**
|
|
- Keep API contracts in Rust shared types before exposing them to CLI and desktop consumers in `crates/api-contracts/src/lib.rs` and `crates/api-contracts/src/chat.rs`
|
|
- Keep inference logic behind a single engine/session boundary in `crates/inference-core/src/engine.rs` and `crates/inference-core/src/session.rs`
|
|
- Keep the desktop app as an orchestration shell that manages process lifecycle, model discovery, and chat persistence in `tauri-app/src/App.tsx` and `tauri-app/src-tauri/src/runtime/mod.rs`
|
|
|
|
## Layers
|
|
|
|
**Inference Core:**
|
|
- Purpose: Load GGUF models, own llama context/model pointers, and generate tokens
|
|
- Location: `crates/inference-core/src/`
|
|
- Contains: `EngineConfig`, `InferenceEngine`, runtime profiles, sampler config, generation session
|
|
- Depends on: `crates/llama-sys`, Rust stdlib, `anyhow`, `tracing`
|
|
- Used by: `crates/inference-server/src/main.rs`
|
|
|
|
**HTTP Compatibility Layer:**
|
|
- Purpose: Adapt OpenAI-like chat/model requests to the local engine
|
|
- Location: `crates/inference-server/src/`
|
|
- Contains: env-driven `ServerConfig`, auth/CORS rules, prompt preparation, routing, SSE streaming
|
|
- Depends on: `crates/inference-core`, `crates/api-contracts`, `crates/metrics`
|
|
- Used by: local HTTP consumers, CLI, desktop frontend
|
|
|
|
**Shared API Contract Layer:**
|
|
- Purpose: Keep request/response models and error shapes consistent
|
|
- Location: `crates/api-contracts/src/`
|
|
- Contains: chat completion structs, model list, health payload, error payloads
|
|
- Depends on: `serde`, `serde_json`
|
|
- Used by: server code directly; mirrored conceptually by frontend/CLI callers
|
|
|
|
**CLI Surface Layer:**
|
|
- Purpose: Expose operator workflows and a future control plane over HTTP/local actions
|
|
- Location: `crates/localllm-cli/src/main.rs`
|
|
- Contains: command parsing, planning, preview rendering, selective live transport
|
|
- Depends on: `clap`, `ureq`, `serde_json`
|
|
- Used by: developers/operators
|
|
|
|
**Desktop Backend Layer:**
|
|
- Purpose: Persist runtime config, discover model files, detect GPU, spawn/stop local API
|
|
- Location: `tauri-app/src-tauri/src/`
|
|
- Contains: Tauri commands, runtime state machine, health probing, file dialog integration
|
|
- Depends on: Tauri runtime, native process and filesystem APIs
|
|
- Used by: `tauri-app/src/features/runtime/api.ts`
|
|
|
|
**Desktop Frontend Layer:**
|
|
- Purpose: Present runtime controls, model selection, and persistent chat UI
|
|
- Location: `tauri-app/src/`
|
|
- Contains: shell layout, runtime manager hooks, chat view, sidebar, markdown rendering, local DB wrapper
|
|
- Depends on: React, fetch, Tauri invoke bridge, browser storage/CDN-loaded sql.js
|
|
- Used by: end users of the desktop app
|
|
|
|
## Data Flow
|
|
|
|
**Local chat request flow:**
|
|
|
|
1. `tauri-app/src/components/ChatView.tsx` builds message history from local sql.js storage and posts to `${server}/v1/chat/completions`
|
|
2. `crates/inference-server/src/routes.rs` validates auth/model/mode, prepares the prompt through `crates/inference-server/src/compat.rs`, and locks the shared `InferenceEngine`
|
|
3. `crates/inference-core/src/session.rs` tokenizes, pre-fills, samples, and returns streamed or buffered content back through Axum SSE/JSON
|
|
4. `tauri-app/src/components/ChatView.tsx` parses `<think>...</think>`, stores assistant content plus usage metrics in `tauri-app/src/lib/db.ts`, and updates the UI
|
|
|
|
**Runtime management flow:**
|
|
|
|
1. `tauri-app/src/features/runtime/useRuntimeManager.ts` bootstraps a `RuntimeSnapshot` from `tauri-app/src/features/runtime/api.ts`
|
|
2. `tauri-app/src-tauri/src/lib.rs` routes invoke calls into `tauri-app/src-tauri/src/runtime/mod.rs`
|
|
3. `tauri-app/src-tauri/src/runtime/process.rs` finds and spawns the `localllm` binary, then `tauri-app/src-tauri/src/runtime/health.rs` probes `/health`
|
|
4. `tauri-app/src/features/runtime/components/RuntimeStatusCard.tsx` and `tauri-app/src/features/runtime/components/RuntimeSettingsDrawer.tsx` render and mutate the runtime state
|
|
|
|
**State Management:**
|
|
- Backend API state is a single shared `AppState` with `Arc<Mutex<InferenceEngine>>` in `crates/inference-server/src/routes.rs`
|
|
- Desktop runtime state is a single `RuntimeState(pub Mutex<RuntimeInner>)` in `tauri-app/src-tauri/src/runtime/state.rs`
|
|
- Frontend state is local React hook state; no global store is used in `tauri-app/src/App.tsx`, `tauri-app/src/hooks/useModels.ts`, and `tauri-app/src/features/runtime/useRuntimeManager.ts`
|
|
- Chat persistence is local-first and browser-backed via `localStorage` + sql.js in `tauri-app/src/lib/db.ts`
|
|
|
|
## Key Abstractions
|
|
|
|
**InferenceEngine:**
|
|
- Purpose: Own one loaded model and one llama context
|
|
- Examples: `crates/inference-core/src/engine.rs`, `crates/inference-core/src/session.rs`
|
|
- Pattern: Single-owner, RAII-managed native resource wrapper with explicit non-concurrent use
|
|
|
|
**ServerConfig / ModelCatalog:**
|
|
- Purpose: Centralize runtime auth, prompt, CORS, and alias rules
|
|
- Examples: `crates/inference-server/src/compat.rs`
|
|
- Pattern: Env-derived compatibility adapter layered in front of the engine
|
|
|
|
**RuntimeSnapshot:**
|
|
- Purpose: Serialize the desktop runtime state into a frontend-safe shape
|
|
- Examples: `tauri-app/src/features/runtime/types.ts`, `tauri-app/src-tauri/src/runtime/state.rs`
|
|
- Pattern: Backend source of truth mirrored into a polling frontend hook
|
|
|
|
**DB wrapper:**
|
|
- Purpose: Hide sql.js statement handling and persistence details behind a tiny query/run API
|
|
- Examples: `tauri-app/src/lib/db.ts`
|
|
- Pattern: Thin synchronous local repository over browser storage
|
|
|
|
## Entry Points
|
|
|
|
**HTTP API binary:**
|
|
- Location: `crates/inference-server/src/main.rs`
|
|
- Triggers: `cargo run -p inference-server` or spawned local desktop runtime
|
|
- Responsibilities: read env, load/warm engine, build router, bind network listener
|
|
|
|
**CLI binary:**
|
|
- Location: `crates/localllm-cli/src/main.rs`
|
|
- Triggers: `cargo run -p localllm-cli -- ...`
|
|
- Responsibilities: parse commands, build execution plans, optionally perform live HTTP requests
|
|
|
|
**Desktop frontend:**
|
|
- Location: `tauri-app/src/main.tsx` and `tauri-app/src/App.tsx`
|
|
- Triggers: Vite dev server or Tauri desktop app startup
|
|
- Responsibilities: render app shell, bootstrap DB/runtime hooks, connect sidebar and chat surface
|
|
|
|
**Desktop backend:**
|
|
- Location: `tauri-app/src-tauri/src/lib.rs`
|
|
- Triggers: Tauri app startup and frontend `invoke()` calls
|
|
- Responsibilities: expose commands, manage runtime state, wire OS dialogs/plugins
|
|
|
|
## Error Handling
|
|
|
|
**Strategy:** Convert most backend failures into typed user-facing snapshots/responses rather than panic paths
|
|
|
|
**Patterns:**
|
|
- API uses `ApiError` + `ErrorResponse` with HTTP status mapping in `crates/inference-server/src/routes.rs`
|
|
- Desktop runtime stores recoverable errors in `last_error` on `RuntimeSnapshot` in `tauri-app/src-tauri/src/runtime/state.rs`
|
|
- Frontend fetch and invoke failures typically degrade to fallback state or inline assistant error messages in `tauri-app/src/components/ChatView.tsx` and `tauri-app/src/features/runtime/api.ts`
|
|
|
|
## Cross-Cutting Concerns
|
|
|
|
**Logging:** `tracing` on Rust services in `crates/inference-server/src/main.rs`; frontend logging is minimal and mostly absent
|
|
**Validation:** request and mode validation lives in `crates/inference-server/src/compat.rs`; runtime form validation is mostly helper-based in `tauri-app/src/features/runtime/runtime-settings-form.ts`
|
|
**Authentication:** optional API-key auth is enforced in `crates/inference-server/src/compat.rs`; desktop frontend forwards the configured API key from runtime config in `tauri-app/src/features/runtime/api.ts`
|
|
|
|
## Boundaries
|
|
|
|
**App boundaries:**
|
|
- `crates/inference-core/` should stay inference-only and not absorb HTTP/UI concerns
|
|
- `crates/inference-server/` should stay the public backend boundary and not own chat persistence or desktop state
|
|
- `crates/localllm-cli/` should stay an operator/client surface and not become the runtime itself
|
|
- `tauri-app/src-tauri/` should own process/config/filesystem concerns for desktop only
|
|
- `tauri-app/src/` should stay a consumer/orchestrator over backend contracts and local chat persistence
|
|
|
|
---
|
|
|
|
*Architecture analysis: 2026-03-28*
|