## Project **ForgeLM** ForgeLM (formerly AI Backend Studio) is a local-first desktop workspace for running and managing local LLM runtimes, selecting models, and chatting against the active runtime from a Tauri app. This roadmap focuses on the existing runtime workspace experience: making runtime control and configuration easier to understand, safer to edit, and more reliable for operators using the desktop app. **Core Value:** A user can configure and operate the runtime from one trustworthy workspace without the UI fighting their in-progress edits. ### Constraints - **Tech stack**: Keep the existing Rust + Tauri + React/TypeScript architecture - the project is improving the current app, not rewriting it - **Brownfield compatibility**: Preserve working runtime lifecycle, model selection, and chat flows while reorganizing the UI - regressions in validated capabilities are not acceptable - **UX reliability**: Unsaved user edits must not be overwritten by polling or snapshot refresh - trust in the settings surface is the main product constraint - **Local-first**: Continue supporting local desktop runtime management patterns already established in the app - scope should not depend on external hosted infrastructure ## Technology Stack ## Languages - Rust 2021 - core backend, inference runtime, CLI, shared contracts, metrics, and Tauri desktop backend in `Cargo.toml`, `crates/inference-server/src/main.rs`, `crates/inference-core/src/engine.rs`, `crates/localllm-cli/src/main.rs`, and `tauri-app/src-tauri/src/lib.rs` - TypeScript 5.5 - desktop frontend, runtime manager UI, chat UI, and tests in `tauri-app/package.json`, `tauri-app/src/App.tsx`, and `tauri-app/src/features/runtime/components/RuntimeSettingsDrawer.tsx` - CSS - hand-authored styling with app-wide tokens and feature CSS modules in `tauri-app/src/index.css`, `tauri-app/src/components/Sidebar.css`, and `tauri-app/src/components/ChatView.css` - Markdown - product and architecture docs in `README.MD`, `docs/api/overview.md`, and `docs/app/overview.md` - Python - benchmark and experiment scripts in `experiments/bench_chat_latency.py`, `experiments/bench_cli_surface.py`, and related files under `experiments/` ## Runtime - Rust toolchain targeting native desktop/server binaries; no workspace-level rust-toolchain file detected - Node.js runtime for Vite/Tauri frontend workflows in `tauri-app/package.json` - Cargo for Rust workspace dependencies and binaries in `Cargo.toml` - npm is the documented frontend package manager via `README.MD:47`; lockfiles for npm and pnpm are both present in `tauri-app/package-lock.json` and `tauri-app/pnpm-lock.yaml` - Lockfile: present for Rust and frontend in `Cargo.lock` and `tauri-app/package-lock.json` ## Frameworks - Axum 0.8 - local HTTP API in `Cargo.toml` and `crates/inference-server/src/routes.rs` - React 18 - desktop UI shell and chat surface in `tauri-app/package.json` and `tauri-app/src/App.tsx` - Tauri 2 - desktop shell, local process control, file dialogs, and frontend-to-Rust command bridge in `tauri-app/src-tauri/Cargo.toml` and `tauri-app/src-tauri/src/lib.rs` - llama.cpp FFI via custom `llama-sys` - model loading and token generation in `crates/inference-core/src/engine.rs` and `crates/inference-core/src/session.rs` - Vitest 4 - frontend/unit/component tests in `tauri-app/package.json` and `tauri-app/src/features/runtime/components/RuntimeSurface.test.tsx` - Rust built-in test harness - backend/unit tests co-located in files such as `crates/api-contracts/src/chat.rs` and `tauri-app/src-tauri/src/runtime/state.rs` - Vite 5 - frontend dev/build tool in `tauri-app/vite.config.ts` - TypeScript compiler - typecheck step in `tauri-app/package.json:9` - Tauri CLI 2 - desktop dev/build entry in `tauri-app/package.json:11` ## Key Dependencies - `tokio` - async runtime for API server in `Cargo.toml` and `crates/inference-server/src/main.rs` - `tower-http` - CORS and trace middleware in `Cargo.toml` and `crates/inference-server/src/compat.rs` - `serde` / `serde_json` - shared request/response contracts across API, CLI, and Tauri in `crates/api-contracts/src/chat.rs` and `tauri-app/src-tauri/src/runtime/config.rs` - `clap` - CLI command surface in `crates/localllm-cli/src/main.rs` - `ureq` - synchronous HTTP transport for CLI live execution in `crates/localllm-cli/src/main.rs:404` - `marked`, `dompurify`, `highlight.js` - markdown rendering/sanitization/code highlighting in `tauri-app/src/lib/markdown.ts` - `@tauri-apps/api` and `@tauri-apps/plugin-shell` - frontend/Tauri bridge in `tauri-app/package.json` and `tauri-app/src/features/runtime/api.ts` - `uuid` - response IDs for chat completions in `crates/inference-server/src/routes.rs:126` - `tracing` and `tracing-subscriber` - backend logging and startup diagnostics in `crates/inference-server/src/main.rs` - `async-stream` - SSE streaming implementation in `crates/inference-server/Cargo.toml` and `crates/inference-server/src/routes.rs:137` ## Configuration - Backend runtime is environment-driven via `LOCALLLM_MODEL`, `LOCALLLM_PROFILE`, `LOCALLLM_ADDR`, `LOCALLLM_GPU_LAYERS`, `LOCALLLM_CTX`, `LOCALLLM_API_KEY`, `LOCALLLM_CORS_ORIGINS`, `LOCALLLM_AUTH_HEADER`, `LOCALLLM_AUTH_SCHEME`, `LOCALLLM_SYSTEM_PROMPT`, and `LOCALLLM_MODEL_ALIASES` in `crates/inference-server/src/main.rs` and `crates/inference-server/src/compat.rs` - Desktop runtime config is persisted as JSON in the app data directory through `tauri-app/src-tauri/src/runtime/config.rs` - Frontend fallback runtime snapshot is persisted in `localStorage` through `tauri-app/src/features/runtime/api.ts` - Rust workspace build in `Cargo.toml` - Tauri desktop build in `tauri-app/src-tauri/Cargo.toml` and `tauri-app/src-tauri/tauri.conf.json` - Frontend dev server/build in `tauri-app/vite.config.ts` and `tauri-app/package.json` ## Platform Requirements - Rust + Cargo for backend and desktop backend in `README.MD:56` - Node.js/npm for the Tauri frontend in `README.MD:57` - Local `.gguf` model for real inference in `README.MD:58` - Optional NVIDIA GPU for accelerated local runtime; CPU fallback profile exists in `README.MD:59` and `crates/inference-core/src/profile.rs` - Local-first desktop deployment via Tauri app in `tauri-app/` - Local HTTP API deployment as standalone binary `localllm` in `crates/inference-server/Cargo.toml:8` ## Conventions Conventions not yet established. Will populate as patterns emerge during development. ## Architecture ## Pattern Overview - Keep API contracts in Rust shared types before exposing them to CLI and desktop consumers in `crates/api-contracts/src/lib.rs` and `crates/api-contracts/src/chat.rs` - Keep inference logic behind a single engine/session boundary in `crates/inference-core/src/engine.rs` and `crates/inference-core/src/session.rs` - Keep the desktop app as an orchestration shell that manages process lifecycle, model discovery, and chat persistence in `tauri-app/src/App.tsx` and `tauri-app/src-tauri/src/runtime/mod.rs` ## Layers - Purpose: Load GGUF models, own llama context/model pointers, and generate tokens - Location: `crates/inference-core/src/` - Contains: `EngineConfig`, `InferenceEngine`, runtime profiles, sampler config, generation session - Depends on: `crates/llama-sys`, Rust stdlib, `anyhow`, `tracing` - Used by: `crates/inference-server/src/main.rs` - Purpose: Adapt OpenAI-like chat/model requests to the local engine - Location: `crates/inference-server/src/` - Contains: env-driven `ServerConfig`, auth/CORS rules, prompt preparation, routing, SSE streaming - Depends on: `crates/inference-core`, `crates/api-contracts`, `crates/metrics` - Used by: local HTTP consumers, CLI, desktop frontend - Purpose: Keep request/response models and error shapes consistent - Location: `crates/api-contracts/src/` - Contains: chat completion structs, model list, health payload, error payloads - Depends on: `serde`, `serde_json` - Used by: server code directly; mirrored conceptually by frontend/CLI callers - Purpose: Expose operator workflows and a future control plane over HTTP/local actions - Location: `crates/localllm-cli/src/main.rs` - Contains: command parsing, planning, preview rendering, selective live transport - Depends on: `clap`, `ureq`, `serde_json` - Used by: developers/operators - Purpose: Persist runtime config, discover model files, detect GPU, spawn/stop local API - Location: `tauri-app/src-tauri/src/` - Contains: Tauri commands, runtime state machine, health probing, file dialog integration - Depends on: Tauri runtime, native process and filesystem APIs - Used by: `tauri-app/src/features/runtime/api.ts` - Purpose: Present runtime controls, model selection, and persistent chat UI - Location: `tauri-app/src/` - Contains: shell layout, runtime manager hooks, chat view, sidebar, markdown rendering, local DB wrapper - Depends on: React, fetch, Tauri invoke bridge, browser storage/CDN-loaded sql.js - Used by: end users of the desktop app ## Data Flow - Backend API state is a single shared `AppState` with `Arc>` in `crates/inference-server/src/routes.rs` - Desktop runtime state is a single `RuntimeState(pub Mutex)` in `tauri-app/src-tauri/src/runtime/state.rs` - Frontend state is local React hook state; no global store is used in `tauri-app/src/App.tsx`, `tauri-app/src/hooks/useModels.ts`, and `tauri-app/src/features/runtime/useRuntimeManager.ts` - Chat persistence is local-first and browser-backed via `localStorage` + sql.js in `tauri-app/src/lib/db.ts` ## Key Abstractions - Purpose: Own one loaded model and one llama context - Examples: `crates/inference-core/src/engine.rs`, `crates/inference-core/src/session.rs` - Pattern: Single-owner, RAII-managed native resource wrapper with explicit non-concurrent use - Purpose: Centralize runtime auth, prompt, CORS, and alias rules - Examples: `crates/inference-server/src/compat.rs` - Pattern: Env-derived compatibility adapter layered in front of the engine - Purpose: Serialize the desktop runtime state into a frontend-safe shape - Examples: `tauri-app/src/features/runtime/types.ts`, `tauri-app/src-tauri/src/runtime/state.rs` - Pattern: Backend source of truth mirrored into a polling frontend hook - Purpose: Hide sql.js statement handling and persistence details behind a tiny query/run API - Examples: `tauri-app/src/lib/db.ts` - Pattern: Thin synchronous local repository over browser storage ## Entry Points - Location: `crates/inference-server/src/main.rs` - Triggers: `cargo run -p inference-server` or spawned local desktop runtime - Responsibilities: read env, load/warm engine, build router, bind network listener - Location: `crates/localllm-cli/src/main.rs` - Triggers: `cargo run -p localllm-cli -- ...` - Responsibilities: parse commands, build execution plans, optionally perform live HTTP requests - Location: `tauri-app/src/main.tsx` and `tauri-app/src/App.tsx` - Triggers: Vite dev server or Tauri desktop app startup - Responsibilities: render app shell, bootstrap DB/runtime hooks, connect sidebar and chat surface - Location: `tauri-app/src-tauri/src/lib.rs` - Triggers: Tauri app startup and frontend `invoke()` calls - Responsibilities: expose commands, manage runtime state, wire OS dialogs/plugins ## Error Handling - API uses `ApiError` + `ErrorResponse` with HTTP status mapping in `crates/inference-server/src/routes.rs` - Desktop runtime stores recoverable errors in `last_error` on `RuntimeSnapshot` in `tauri-app/src-tauri/src/runtime/state.rs` - Frontend fetch and invoke failures typically degrade to fallback state or inline assistant error messages in `tauri-app/src/components/ChatView.tsx` and `tauri-app/src/features/runtime/api.ts` ## Cross-Cutting Concerns ## Boundaries - `crates/inference-core/` should stay inference-only and not absorb HTTP/UI concerns - `crates/inference-server/` should stay the public backend boundary and not own chat persistence or desktop state - `crates/localllm-cli/` should stay an operator/client surface and not become the runtime itself - `tauri-app/src-tauri/` should own process/config/filesystem concerns for desktop only - `tauri-app/src/` should stay a consumer/orchestrator over backend contracts and local chat persistence ## GSD Workflow Enforcement Before using Edit, Write, or other file-changing tools, start work through a GSD command so planning artifacts and execution context stay in sync. Use these entry points: - `/gsd:quick` for small fixes, doc updates, and ad-hoc tasks - `/gsd:debug` for investigation and bug fixing - `/gsd:execute-phase` for planned phase work Do not make direct repo edits outside a GSD workflow unless the user explicitly asks to bypass it. ## Developer Profile > Profile not yet configured. Run `/gsd:profile-user` to generate your developer profile. > This section is managed by `generate-claude-profile` -- do not edit manually.