# ForgeLM

High-performance local inference runtime for small and mid-sized language models, designed for efficient execution on commodity hardware.

## Overview

ForgeLM is a local-first inference runtime built for developers who need a clean, controllable, and efficient backend for running quantized language models on consumer GPUs and constrained systems.

The project is centered on a practical engineering goal: extracting high throughput and low latency from compact and mid-sized models without turning the runtime into an opaque monolith or an overextended product shell.

The initial focus is on text-only inference for models in the range of Qwen 4B / 9B class deployments, with special attention to NVIDIA GPUs such as the RTX 3070 and to fallback paths for weaker hardware.

ForgeLM is not intended to be a desktop UI product. It is intended to be a modular inference backend that can be embedded, extended, benchmarked, and reasoned about as infrastructure.

---

## Design Goals

ForgeLM is built around a small number of non-negotiable goals:

- **High throughput**
  - Prioritize sustained token generation performance for small and mid-sized models.

- **Low latency**
  - Minimize time-to-first-token and reduce generation overhead in the hot path.

- **Efficient memory usage**
  - Make disciplined use of VRAM and RAM through careful model loading, batching strategy, and KV-cache policies.

- **Predictable architecture**
  - Keep the inference path understandable, inspectable, and modular.

- **Hardware realism**
  - Optimize for hardware that developers actually own, including 8 GB consumer GPUs and CPU fallback scenarios.

- **Separation of concerns**
  - Keep the inference core, API layer, model registry, and tooling clearly separated.

- **Open integration surface**
  - Provide a clean local API compatible with common inference client patterns.

---

## Project Scope

ForgeLM focuses on the runtime layer of local language model serving.

### In scope

- Local text inference
- GGUF-based model execution
- High-performance inference backends
- OpenAI-compatible HTTP API
- Streaming responses via SSE
- Runtime tuning for throughput and memory efficiency
- Model registry and download workflow
- Benchmarking and observability

### Out of scope for the initial versions

- Full desktop product UX
- Multimodal inference
- Enterprise orchestration features
- Training and fine-tuning workflows
- Plugin-heavy product abstractions
- Broad feature surface at the expense of runtime efficiency

---

## Core Principles

### 1. The hot path must remain small

The token generation path is where performance is won or lost. ForgeLM keeps this path narrow:

- model execution in a native backend
- Rust in the serving and orchestration layer
- no Python in the inference hot path
- controlled allocation and bounded runtime overhead

### 2. Performance comes before feature breadth

A runtime that supports everything but performs poorly is not useful for the target use case. ForgeLM deliberately starts with a constrained feature set in order to produce measurable runtime efficiency first.

### 3. Public architecture should remain understandable

The repository is intended to be inspectable by engineers. The design should be explainable without product-language shortcuts, hidden runtime behavior, or unnecessary framework complexity.

### 4. Consumer hardware is a first-class target

Many local inference projects are evaluated on unrealistic hardware assumptions. ForgeLM explicitly targets practical systems, including 8 GB GPUs and weaker fallback environments.

---

## Technical Direction

ForgeLM is structured as a layered system.

### Inference Core

The inference core is built around a native execution backend for quantized GGUF models. The initial backend strategy is centered on `llama.cpp`-class execution, selected for its strong hardware coverage, mature quantized model support, and practical performance characteristics.

Responsibilities:

- model loading
- tokenizer access
- context management
- decode loop execution
- GPU offloading control
- sampling configuration
- KV-cache handling

### Serving Layer

The HTTP serving layer is implemented in Rust and is responsible for:

- request validation
- session orchestration
- streaming token delivery
- concurrency control
- backpressure
- metrics and health endpoints

This layer exists to expose the runtime cleanly without polluting the core execution path.

### Model Registry

Model search, metadata resolution, and download management are intentionally isolated from the hot path. Registry operations may use a separate process and different tooling, but they do not participate in token generation.

Responsibilities:

- model discovery
- metadata normalization
- artifact download
- integrity verification
- local registry indexing

### Tooling and Benchmarks

Performance claims without reproducible measurement are not useful. Benchmarking and observability are part of the project rather than afterthoughts.

Responsibilities:

- throughput measurement
- latency measurement
- memory usage tracking
- benchmark replay
- runtime profiling

---

## Initial Optimization Target

The first optimization target is the practical deployment envelope of small and mid-sized text models on commodity NVIDIA hardware.

That means prioritizing:

- compact and mid-sized Qwen-class models
- strong throughput on RTX 3070-class systems
- viable degraded execution on weaker hardware
- text-only serving before broader capability expansion

In practical terms, the project is designed to explore the trade space between:

- model size
- quantization level
- context length
- GPU offloading depth
- batching strategy
- latency
- aggregate throughput
- output quality

---

## Architectural Priorities

The implementation roadmap is shaped by the following priorities:

### Text-only first

Text inference is the fastest path to a stable and high-performance serving core. Additional modalities, if ever added, must not distort the original runtime goals.

### Single-model efficiency before multi-model orchestration

Before adding model multiplexing or dynamic model switching, the runtime must demonstrate excellent single-model performance and stable serving behavior.

### Controlled concurrency before maximum concurrency

Concurrency is useful only when it improves real system throughput without collapsing latency or wasting memory. The project favors disciplined scheduling over superficial parallelism.

### Measured optimization over speculative abstraction

Every substantial optimization should be benchmarked. Every abstraction in the serving path should justify its cost.

---

## Proposed Stack

The current technical direction is:

### Native backend
- C/C++
- `llama.cpp`-class runtime backend
- CUDA path for NVIDIA acceleration

### Serving layer
- Rust
- Tokio
- Axum
- Hyper
- FFI bindings generated with `bindgen`

### Registry and model management
- Python
- isolated background process or service
- Hugging Face integration for model discovery and retrieval

### UI
- minimal local web interface for runtime validation
- not a product-grade desktop shell

---

## Repository Structure

```text
forgelm/
├── Cargo.toml
├── crates/
│   ├── backend-sys/
│   ├── inference-core/
│   ├── inference-server/
│   ├── api-contracts/
│   └── metrics/
├── registry/
│   ├── service.py
│   └── requirements.txt
├── ui/
│   └── index.html
├── vendor/
│   └── backend/
└── models/