ForgeLM/.planning/codebase/ARCHITECTURE.md

Architecture

Analysis Date: 2026-03-28

Pattern Overview

Overall: Local-first polyglot workspace with shared contracts and three user surfaces

Key Characteristics:

• Keep API contracts in Rust shared types before exposing them to CLI and desktop consumers in crates/api-contracts/src/lib.rs and crates/api-contracts/src/chat.rs
• Keep inference logic behind a single engine/session boundary in crates/inference-core/src/engine.rs and crates/inference-core/src/session.rs
• Keep the desktop app as an orchestration shell that manages process lifecycle, model discovery, and chat persistence in tauri-app/src/App.tsx and tauri-app/src-tauri/src/runtime/mod.rs

Layers

Inference Core:

• Purpose: Load GGUF models, own llama context/model pointers, and generate tokens
• Location: crates/inference-core/src/
• Contains: EngineConfig, InferenceEngine, runtime profiles, sampler config, generation session
• Depends on: crates/llama-sys, Rust stdlib, anyhow, tracing
• Used by: crates/inference-server/src/main.rs

HTTP Compatibility Layer:

• Purpose: Adapt OpenAI-like chat/model requests to the local engine
• Location: crates/inference-server/src/
• Contains: env-driven ServerConfig, auth/CORS rules, prompt preparation, routing, SSE streaming
• Depends on: crates/inference-core, crates/api-contracts, crates/metrics
• Used by: local HTTP consumers, CLI, desktop frontend

Shared API Contract Layer:

• Purpose: Keep request/response models and error shapes consistent
• Location: crates/api-contracts/src/
• Contains: chat completion structs, model list, health payload, error payloads
• Depends on: serde, serde_json
• Used by: server code directly; mirrored conceptually by frontend/CLI callers

CLI Surface Layer:

• Purpose: Expose operator workflows and a future control plane over HTTP/local actions
• Location: crates/localllm-cli/src/main.rs
• Contains: command parsing, planning, preview rendering, selective live transport
• Depends on: clap, ureq, serde_json
• Used by: developers/operators

Desktop Backend Layer:

• Purpose: Persist runtime config, discover model files, detect GPU, spawn/stop local API
• Location: tauri-app/src-tauri/src/
• Contains: Tauri commands, runtime state machine, health probing, file dialog integration
• Depends on: Tauri runtime, native process and filesystem APIs
• Used by: tauri-app/src/features/runtime/api.ts

Desktop Frontend Layer:

• Purpose: Present runtime controls, model selection, and persistent chat UI
• Location: tauri-app/src/
• Contains: shell layout, runtime manager hooks, chat view, sidebar, markdown rendering, local DB wrapper
• Depends on: React, fetch, Tauri invoke bridge, browser storage/CDN-loaded sql.js
• Used by: end users of the desktop app

Data Flow

Local chat request flow:

1. tauri-app/src/components/ChatView.tsx builds message history from local sql.js storage and posts to ${server}/v1/chat/completions
2. crates/inference-server/src/routes.rs validates auth/model/mode, prepares the prompt through crates/inference-server/src/compat.rs, and locks the shared InferenceEngine
3. crates/inference-core/src/session.rs tokenizes, pre-fills, samples, and returns streamed or buffered content back through Axum SSE/JSON
4. tauri-app/src/components/ChatView.tsx parses <think>...</think>, stores assistant content plus usage metrics in tauri-app/src/lib/db.ts, and updates the UI

Runtime management flow:

1. tauri-app/src/features/runtime/useRuntimeManager.ts bootstraps a RuntimeSnapshot from tauri-app/src/features/runtime/api.ts
2. tauri-app/src-tauri/src/lib.rs routes invoke calls into tauri-app/src-tauri/src/runtime/mod.rs
3. tauri-app/src-tauri/src/runtime/process.rs finds and spawns the localllm binary, then tauri-app/src-tauri/src/runtime/health.rs probes /health
4. tauri-app/src/features/runtime/components/RuntimeStatusCard.tsx and tauri-app/src/features/runtime/components/RuntimeSettingsDrawer.tsx render and mutate the runtime state

State Management:

• Backend API state is a single shared AppState with Arc<Mutex<InferenceEngine>> in crates/inference-server/src/routes.rs
• Desktop runtime state is a single RuntimeState(pub Mutex<RuntimeInner>) in tauri-app/src-tauri/src/runtime/state.rs
• Frontend state is local React hook state; no global store is used in tauri-app/src/App.tsx, tauri-app/src/hooks/useModels.ts, and tauri-app/src/features/runtime/useRuntimeManager.ts
• Chat persistence is local-first and browser-backed via localStorage + sql.js in tauri-app/src/lib/db.ts

Key Abstractions

InferenceEngine:

• Purpose: Own one loaded model and one llama context
• Examples: crates/inference-core/src/engine.rs, crates/inference-core/src/session.rs
• Pattern: Single-owner, RAII-managed native resource wrapper with explicit non-concurrent use

ServerConfig / ModelCatalog:

• Purpose: Centralize runtime auth, prompt, CORS, and alias rules
• Examples: crates/inference-server/src/compat.rs
• Pattern: Env-derived compatibility adapter layered in front of the engine

RuntimeSnapshot:

• Purpose: Serialize the desktop runtime state into a frontend-safe shape
• Examples: tauri-app/src/features/runtime/types.ts, tauri-app/src-tauri/src/runtime/state.rs
• Pattern: Backend source of truth mirrored into a polling frontend hook

DB wrapper:

• Purpose: Hide sql.js statement handling and persistence details behind a tiny query/run API
• Examples: tauri-app/src/lib/db.ts
• Pattern: Thin synchronous local repository over browser storage

Entry Points

HTTP API binary:

• Location: crates/inference-server/src/main.rs
• Triggers: cargo run -p inference-server or spawned local desktop runtime
• Responsibilities: read env, load/warm engine, build router, bind network listener

CLI binary:

• Location: crates/localllm-cli/src/main.rs
• Triggers: cargo run -p localllm-cli -- ...
• Responsibilities: parse commands, build execution plans, optionally perform live HTTP requests

Desktop frontend:

• Location: tauri-app/src/main.tsx and tauri-app/src/App.tsx
• Triggers: Vite dev server or Tauri desktop app startup
• Responsibilities: render app shell, bootstrap DB/runtime hooks, connect sidebar and chat surface

Desktop backend:

• Location: tauri-app/src-tauri/src/lib.rs
• Triggers: Tauri app startup and frontend invoke() calls
• Responsibilities: expose commands, manage runtime state, wire OS dialogs/plugins

Error Handling

Strategy: Convert most backend failures into typed user-facing snapshots/responses rather than panic paths

Patterns:

• API uses ApiError + ErrorResponse with HTTP status mapping in crates/inference-server/src/routes.rs
• Desktop runtime stores recoverable errors in last_error on RuntimeSnapshot in tauri-app/src-tauri/src/runtime/state.rs
• Frontend fetch and invoke failures typically degrade to fallback state or inline assistant error messages in tauri-app/src/components/ChatView.tsx and tauri-app/src/features/runtime/api.ts

Cross-Cutting Concerns

Logging: tracing on Rust services in crates/inference-server/src/main.rs; frontend logging is minimal and mostly absent Validation: request and mode validation lives in crates/inference-server/src/compat.rs; runtime form validation is mostly helper-based in tauri-app/src/features/runtime/runtime-settings-form.ts Authentication: optional API-key auth is enforced in crates/inference-server/src/compat.rs; desktop frontend forwards the configured API key from runtime config in tauri-app/src/features/runtime/api.ts

Boundaries

App boundaries:

• crates/inference-core/ should stay inference-only and not absorb HTTP/UI concerns
• crates/inference-server/ should stay the public backend boundary and not own chat persistence or desktop state
• crates/localllm-cli/ should stay an operator/client surface and not become the runtime itself
• tauri-app/src-tauri/ should own process/config/filesystem concerns for desktop only
• tauri-app/src/ should stay a consumer/orchestrator over backend contracts and local chat persistence

---

Architecture analysis: 2026-03-28