243 lines
7.4 KiB
Markdown
243 lines
7.4 KiB
Markdown
# ForgeLM
|
|
|
|
High-performance local inference runtime for small and mid-sized language models, designed for efficient execution on commodity hardware.
|
|
|
|
## Overview
|
|
|
|
ForgeLM is a local-first inference runtime built for developers who need a clean, controllable, and efficient backend for running quantized language models on consumer GPUs and constrained systems.
|
|
|
|
The project is centered on a practical engineering goal: extracting high throughput and low latency from compact and mid-sized models without turning the runtime into an opaque monolith or an overextended product shell.
|
|
|
|
The initial focus is on text-only inference for models in the range of Qwen 4B / 9B class deployments, with special attention to NVIDIA GPUs such as the RTX 3070 and to fallback paths for weaker hardware.
|
|
|
|
ForgeLM is not intended to be a desktop UI product. It is intended to be a modular inference backend that can be embedded, extended, benchmarked, and reasoned about as infrastructure.
|
|
|
|
---
|
|
|
|
## Design Goals
|
|
|
|
ForgeLM is built around a small number of non-negotiable goals:
|
|
|
|
- **High throughput**
|
|
- Prioritize sustained token generation performance for small and mid-sized models.
|
|
|
|
- **Low latency**
|
|
- Minimize time-to-first-token and reduce generation overhead in the hot path.
|
|
|
|
- **Efficient memory usage**
|
|
- Make disciplined use of VRAM and RAM through careful model loading, batching strategy, and KV-cache policies.
|
|
|
|
- **Predictable architecture**
|
|
- Keep the inference path understandable, inspectable, and modular.
|
|
|
|
- **Hardware realism**
|
|
- Optimize for hardware that developers actually own, including 8 GB consumer GPUs and CPU fallback scenarios.
|
|
|
|
- **Separation of concerns**
|
|
- Keep the inference core, API layer, model registry, and tooling clearly separated.
|
|
|
|
- **Open integration surface**
|
|
- Provide a clean local API compatible with common inference client patterns.
|
|
|
|
---
|
|
|
|
## Project Scope
|
|
|
|
ForgeLM focuses on the runtime layer of local language model serving.
|
|
|
|
### In scope
|
|
|
|
- Local text inference
|
|
- GGUF-based model execution
|
|
- High-performance inference backends
|
|
- OpenAI-compatible HTTP API
|
|
- Streaming responses via SSE
|
|
- Runtime tuning for throughput and memory efficiency
|
|
- Model registry and download workflow
|
|
- Benchmarking and observability
|
|
|
|
### Out of scope for the initial versions
|
|
|
|
- Full desktop product UX
|
|
- Multimodal inference
|
|
- Enterprise orchestration features
|
|
- Training and fine-tuning workflows
|
|
- Plugin-heavy product abstractions
|
|
- Broad feature surface at the expense of runtime efficiency
|
|
|
|
---
|
|
|
|
## Core Principles
|
|
|
|
### 1. The hot path must remain small
|
|
|
|
The token generation path is where performance is won or lost. ForgeLM keeps this path narrow:
|
|
|
|
- model execution in a native backend
|
|
- Rust in the serving and orchestration layer
|
|
- no Python in the inference hot path
|
|
- controlled allocation and bounded runtime overhead
|
|
|
|
### 2. Performance comes before feature breadth
|
|
|
|
A runtime that supports everything but performs poorly is not useful for the target use case. ForgeLM deliberately starts with a constrained feature set in order to produce measurable runtime efficiency first.
|
|
|
|
### 3. Public architecture should remain understandable
|
|
|
|
The repository is intended to be inspectable by engineers. The design should be explainable without product-language shortcuts, hidden runtime behavior, or unnecessary framework complexity.
|
|
|
|
### 4. Consumer hardware is a first-class target
|
|
|
|
Many local inference projects are evaluated on unrealistic hardware assumptions. ForgeLM explicitly targets practical systems, including 8 GB GPUs and weaker fallback environments.
|
|
|
|
---
|
|
|
|
## Technical Direction
|
|
|
|
ForgeLM is structured as a layered system.
|
|
|
|
### Inference Core
|
|
|
|
The inference core is built around a native execution backend for quantized GGUF models. The initial backend strategy is centered on `llama.cpp`-class execution, selected for its strong hardware coverage, mature quantized model support, and practical performance characteristics.
|
|
|
|
Responsibilities:
|
|
|
|
- model loading
|
|
- tokenizer access
|
|
- context management
|
|
- decode loop execution
|
|
- GPU offloading control
|
|
- sampling configuration
|
|
- KV-cache handling
|
|
|
|
### Serving Layer
|
|
|
|
The HTTP serving layer is implemented in Rust and is responsible for:
|
|
|
|
- request validation
|
|
- session orchestration
|
|
- streaming token delivery
|
|
- concurrency control
|
|
- backpressure
|
|
- metrics and health endpoints
|
|
|
|
This layer exists to expose the runtime cleanly without polluting the core execution path.
|
|
|
|
### Model Registry
|
|
|
|
Model search, metadata resolution, and download management are intentionally isolated from the hot path. Registry operations may use a separate process and different tooling, but they do not participate in token generation.
|
|
|
|
Responsibilities:
|
|
|
|
- model discovery
|
|
- metadata normalization
|
|
- artifact download
|
|
- integrity verification
|
|
- local registry indexing
|
|
|
|
### Tooling and Benchmarks
|
|
|
|
Performance claims without reproducible measurement are not useful. Benchmarking and observability are part of the project rather than afterthoughts.
|
|
|
|
Responsibilities:
|
|
|
|
- throughput measurement
|
|
- latency measurement
|
|
- memory usage tracking
|
|
- benchmark replay
|
|
- runtime profiling
|
|
|
|
---
|
|
|
|
## Initial Optimization Target
|
|
|
|
The first optimization target is the practical deployment envelope of small and mid-sized text models on commodity NVIDIA hardware.
|
|
|
|
That means prioritizing:
|
|
|
|
- compact and mid-sized Qwen-class models
|
|
- strong throughput on RTX 3070-class systems
|
|
- viable degraded execution on weaker hardware
|
|
- text-only serving before broader capability expansion
|
|
|
|
In practical terms, the project is designed to explore the trade space between:
|
|
|
|
- model size
|
|
- quantization level
|
|
- context length
|
|
- GPU offloading depth
|
|
- batching strategy
|
|
- latency
|
|
- aggregate throughput
|
|
- output quality
|
|
|
|
---
|
|
|
|
## Architectural Priorities
|
|
|
|
The implementation roadmap is shaped by the following priorities:
|
|
|
|
### Text-only first
|
|
|
|
Text inference is the fastest path to a stable and high-performance serving core. Additional modalities, if ever added, must not distort the original runtime goals.
|
|
|
|
### Single-model efficiency before multi-model orchestration
|
|
|
|
Before adding model multiplexing or dynamic model switching, the runtime must demonstrate excellent single-model performance and stable serving behavior.
|
|
|
|
### Controlled concurrency before maximum concurrency
|
|
|
|
Concurrency is useful only when it improves real system throughput without collapsing latency or wasting memory. The project favors disciplined scheduling over superficial parallelism.
|
|
|
|
### Measured optimization over speculative abstraction
|
|
|
|
Every substantial optimization should be benchmarked. Every abstraction in the serving path should justify its cost.
|
|
|
|
---
|
|
|
|
## Proposed Stack
|
|
|
|
The current technical direction is:
|
|
|
|
### Native backend
|
|
- C/C++
|
|
- `llama.cpp`-class runtime backend
|
|
- CUDA path for NVIDIA acceleration
|
|
|
|
### Serving layer
|
|
- Rust
|
|
- Tokio
|
|
- Axum
|
|
- Hyper
|
|
- FFI bindings generated with `bindgen`
|
|
|
|
### Registry and model management
|
|
- Python
|
|
- isolated background process or service
|
|
- Hugging Face integration for model discovery and retrieval
|
|
|
|
### UI
|
|
- minimal local web interface for runtime validation
|
|
- not a product-grade desktop shell
|
|
|
|
---
|
|
|
|
## Repository Structure
|
|
|
|
```text
|
|
forgelm/
|
|
├── Cargo.toml
|
|
├── crates/
|
|
│ ├── backend-sys/
|
|
│ ├── inference-core/
|
|
│ ├── inference-server/
|
|
│ ├── api-contracts/
|
|
│ └── metrics/
|
|
├── registry/
|
|
│ ├── service.py
|
|
│ └── requirements.txt
|
|
├── ui/
|
|
│ └── index.html
|
|
├── vendor/
|
|
│ └── backend/
|
|
└── models/ |