feat: add API docs and full-featured chat UI with SQLite persistence
This commit is contained in:
Evandro Fonseca Junior committed
1 parent
13ba05b02a
commit
d0c2f86f3f
2 files changed
+1566
No files matched your search
+393
@@ -0,0 +1,393 @@
|
||||
# ForgeLM API Reference
|
||||
|
||||
ForgeLM exposes an **OpenAI-compatible REST API** for local LLM inference. Any client that works with the OpenAI API (VSCode extensions, OpenCode, Cline, Roo Code, curl, Python `openai` SDK, etc.) can point at this server with no code changes.
|
||||
|
||||
---
|
||||
|
||||
## Base URL
|
||||
|
||||
```
|
||||
http://127.0.0.1:8080
|
||||
```
|
||||
|
||||
Configure with the `LOCALLLM_ADDR` environment variable (default `127.0.0.1:8080`).
|
||||
|
||||
---
|
||||
|
||||
## Authentication
|
||||
|
||||
Authentication is **optional** and disabled by default.
|
||||
|
||||
| Env var | Description |
|
||||
|---------|-------------|
|
||||
| `LOCALLLM_API_KEY` | If set, all requests must include this key |
|
||||
| `LOCALLLM_AUTH_HEADER` | Header name to check (default: `authorization`) |
|
||||
| `LOCALLLM_AUTH_SCHEME` | Expected scheme prefix (default: `Bearer`) |
|
||||
|
||||
**Sending a key:**
|
||||
|
||||
```http
|
||||
Authorization: Bearer <your-key>
|
||||
```
|
||||
|
||||
or via `x-api-key`:
|
||||
|
||||
```http
|
||||
x-api-key: <your-key>
|
||||
```
|
||||
|
||||
If no key is configured, all requests are accepted without authentication.
|
||||
|
||||
---
|
||||
|
||||
## CORS
|
||||
|
||||
CORS is enabled for all origins by default (`*`). Restrict with:
|
||||
|
||||
```
|
||||
LOCALLLM_CORS_ORIGINS=https://myapp.com,http://localhost:3000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Endpoints
|
||||
|
||||
### `GET /health`
|
||||
|
||||
Returns server and model status.
|
||||
|
||||
**Response `200 OK`:**
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"model_loaded": true,
|
||||
"model_id": "Qwen3.5-4B-Abliterated.i1-Q6_K",
|
||||
"uptime_secs": 142
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `status` | `string` | Always `"ok"` when server is running |
|
||||
| `model_loaded` | `boolean` | Whether a model is currently loaded |
|
||||
| `model_id` | `string \| null` | Canonical model identifier |
|
||||
| `uptime_secs` | `integer` | Seconds since server start |
|
||||
|
||||
---
|
||||
|
||||
### `GET /v1/models`
|
||||
|
||||
Lists available models. Compatible with the OpenAI models endpoint.
|
||||
|
||||
**Response `200 OK`:**
|
||||
|
||||
```json
|
||||
{
|
||||
"object": "list",
|
||||
"data": [
|
||||
{
|
||||
"id": "Qwen3.5-4B-Abliterated.i1-Q6_K",
|
||||
"object": "model",
|
||||
"created": 1711497600,
|
||||
"owned_by": "local",
|
||||
"root": "Qwen3.5-4B-Abliterated.i1-Q6_K"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Model aliases (configured via `LOCALLLM_MODEL_ALIASES`) also appear in the list and resolve to the same underlying model.
|
||||
|
||||
---
|
||||
|
||||
### `POST /v1/chat/completions`
|
||||
|
||||
The main inference endpoint. Fully compatible with the OpenAI Chat Completions API.
|
||||
|
||||
**Request body:**
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "Qwen3.5-4B-Abliterated.i1-Q6_K",
|
||||
"messages": [
|
||||
{ "role": "system", "content": "You are a helpful assistant." },
|
||||
{ "role": "user", "content": "Explain async/await in Rust." }
|
||||
],
|
||||
"max_tokens": 1024,
|
||||
"temperature": 0.15,
|
||||
"top_p": 0.95,
|
||||
"stream": false
|
||||
}
|
||||
```
|
||||
|
||||
#### Request fields
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `model` | `string` | **required** | Model ID or alias |
|
||||
| `messages` | `array` | **required** | Conversation history |
|
||||
| `max_tokens` | `integer` | `1024` | Max tokens to generate (alias: `max_completion_tokens`) |
|
||||
| `temperature` | `float` | `0.15` | Sampling temperature (0 = greedy) |
|
||||
| `top_p` | `float` | `0.95` | Nucleus sampling threshold |
|
||||
| `stream` | `boolean` | `false` | Enable SSE streaming |
|
||||
| `stop` | `string \| array` | `[]` | Stop sequences |
|
||||
| `seed` | `integer` | `null` | Reproducibility seed |
|
||||
| `n` | `integer` | `1` | Number of completions (only `1` supported) |
|
||||
| `response_format` | `object` | `null` | `{"type": "text"}` or `{"type": "json_object"}` |
|
||||
| `tools` | `array` | `[]` | Tool definitions for function calling |
|
||||
| `tool_choice` | `string \| object` | `null` | Tool selection policy |
|
||||
| `stream_options` | `object` | `null` | `{"include_usage": true}` to get usage in stream |
|
||||
|
||||
#### Message roles
|
||||
|
||||
| Role | Description |
|
||||
|------|-------------|
|
||||
| `system` | System-level instructions |
|
||||
| `developer` | Developer-level instructions (treated as system) |
|
||||
| `user` | User turn |
|
||||
| `assistant` | Previous assistant turn |
|
||||
| `tool` | Tool call result |
|
||||
|
||||
#### Non-streaming response `200 OK`
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "chatcmpl-a1b2c3d4",
|
||||
"object": "chat.completion",
|
||||
"created": 1711497600,
|
||||
"model": "Qwen3.5-4B-Abliterated.i1-Q6_K",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": "In Rust, `async/await` allows you to write asynchronous code..."
|
||||
},
|
||||
"finish_reason": "stop"
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"prompt_tokens": 24,
|
||||
"completion_tokens": 87,
|
||||
"total_tokens": 111,
|
||||
"tokens_per_second": 52.4
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Streaming response (SSE)
|
||||
|
||||
Set `"stream": true` to receive Server-Sent Events:
|
||||
|
||||
```
|
||||
data: {"id":"chatcmpl-a1b2c3d4","object":"chat.completion.chunk","created":1711497600,"model":"Qwen3.5-4B-Abliterated.i1-Q6_K","choices":[{"index":0,"delta":{"role":"assistant","content":"In"},"finish_reason":null}]}
|
||||
|
||||
data: {"id":"chatcmpl-a1b2c3d4","object":"chat.completion.chunk","created":1711497600,"model":"Qwen3.5-4B-Abliterated.i1-Q6_K","choices":[{"index":0,"delta":{"content":" Rust"},"finish_reason":null}]}
|
||||
|
||||
data: {"id":"chatcmpl-a1b2c3d4","object":"chat.completion.chunk","created":1711497600,"model":"Qwen3.5-4B-Abliterated.i1-Q6_K","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
|
||||
|
||||
data: [DONE]
|
||||
```
|
||||
|
||||
To include usage in the final streaming chunk:
|
||||
|
||||
```json
|
||||
{
|
||||
"stream": true,
|
||||
"stream_options": { "include_usage": true }
|
||||
}
|
||||
```
|
||||
|
||||
#### Tool calling
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "Qwen3.5-4B-Abliterated.i1-Q6_K",
|
||||
"messages": [{ "role": "user", "content": "What is the weather in Paris?" }],
|
||||
"tools": [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get current weather for a city",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"city": { "type": "string" }
|
||||
},
|
||||
"required": ["city"]
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"tool_choice": "auto"
|
||||
}
|
||||
```
|
||||
|
||||
When the model decides to call a tool, `finish_reason` is `"tool_calls"` and `message.tool_calls` contains the call:
|
||||
|
||||
```json
|
||||
{
|
||||
"choices": [{
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": null,
|
||||
"tool_calls": [{
|
||||
"id": "call_abc123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": "{\"city\": \"Paris\"}"
|
||||
}
|
||||
}]
|
||||
},
|
||||
"finish_reason": "tool_calls"
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error responses
|
||||
|
||||
All errors follow the OpenAI error format:
|
||||
|
||||
```json
|
||||
{
|
||||
"error": {
|
||||
"message": "The model 'unknown-model' does not exist",
|
||||
"type": "invalid_request_error",
|
||||
"param": "model",
|
||||
"code": "model_not_found"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| HTTP status | `type` | When |
|
||||
|-------------|--------|------|
|
||||
| `400` | `invalid_request_error` | Malformed request, empty messages, unsupported `n` |
|
||||
| `401` | `authentication_error` | Missing or invalid API key |
|
||||
| `404` | `invalid_request_error` | Unknown model |
|
||||
| `422` | `invalid_request_error` | Unsupported `response_format` |
|
||||
| `500` | `server_error` | Internal inference failure |
|
||||
|
||||
---
|
||||
|
||||
## Environment variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `LOCALLLM_MODEL` | *(required)* | Path to the GGUF model file |
|
||||
| `LOCALLLM_ADDR` | `127.0.0.1:8080` | Bind address |
|
||||
| `LOCALLLM_PROFILE` | `balanced` | Runtime profile (`fast`, `balanced`, `quality`) |
|
||||
| `LOCALLLM_GPU_LAYERS` | `0` | Number of layers to offload to GPU |
|
||||
| `LOCALLLM_CTX` | `4096` | Context window size |
|
||||
| `LOCALLLM_BATCH` | `512` | Batch size |
|
||||
| `LOCALLLM_UBATCH` | `512` | Micro-batch size |
|
||||
| `LOCALLLM_THREADS` | *(auto)* | CPU threads for generation |
|
||||
| `LOCALLLM_THREADS_BATCH` | *(auto)* | CPU threads for batch processing |
|
||||
| `LOCALLLM_API_KEY` | *(none)* | API key for authentication |
|
||||
| `LOCALLLM_AUTH_HEADER` | `authorization` | Auth header name |
|
||||
| `LOCALLLM_AUTH_SCHEME` | `Bearer` | Auth scheme prefix |
|
||||
| `LOCALLLM_CORS_ORIGINS` | `*` | Comma-separated allowed origins |
|
||||
| `LOCALLLM_SYSTEM_PROMPT` | *(coding assistant)* | Default system prompt |
|
||||
| `LOCALLLM_ENABLE_TOOL_CALLS` | `false` | Enable tool calling support |
|
||||
| `LOCALLLM_MODEL_ALIASES` | *(none)* | Comma-separated `alias=model` pairs |
|
||||
|
||||
---
|
||||
|
||||
## Quick start
|
||||
|
||||
### Start the server
|
||||
|
||||
```bash
|
||||
LOCALLLM_MODEL=./models/Qwen3.5-4B.gguf cargo run --release -p inference-server
|
||||
```
|
||||
|
||||
### Test with curl
|
||||
|
||||
```bash
|
||||
# Health check
|
||||
curl http://localhost:8080/health
|
||||
|
||||
# List models
|
||||
curl http://localhost:8080/v1/models
|
||||
|
||||
# Chat completion
|
||||
curl http://localhost:8080/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen3.5-4B",
|
||||
"messages": [{"role": "user", "content": "Hello!"}],
|
||||
"max_tokens": 100
|
||||
}'
|
||||
|
||||
# Streaming
|
||||
curl http://localhost:8080/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen3.5-4B",
|
||||
"messages": [{"role": "user", "content": "Count to 5"}],
|
||||
"stream": true
|
||||
}'
|
||||
```
|
||||
|
||||
### Use with Python openai SDK
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8080/v1",
|
||||
api_key="not-needed" # or your LOCALLLM_API_KEY
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen3.5-4B",
|
||||
messages=[{"role": "user", "content": "Write a Python hello world"}],
|
||||
max_tokens=200
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Use with VSCode / OpenCode / Cline / Roo Code
|
||||
|
||||
Set the following in your extension settings:
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| Base URL | `http://localhost:8080/v1` |
|
||||
| API Key | *(leave empty or set `LOCALLLM_API_KEY`)* |
|
||||
| Model | *(your model ID from `/v1/models`)* |
|
||||
|
||||
---
|
||||
|
||||
## Chat template behavior
|
||||
|
||||
ForgeLM automatically selects the correct chat template based on the model family:
|
||||
|
||||
| Model family | Template |
|
||||
|-------------|----------|
|
||||
| `qwen` | `<\|im_start\|>` / `<\|im_end\|>` tokens |
|
||||
| `gemma` | `<start_of_turn>` / `<end_of_turn>` tokens |
|
||||
| `llama` | `[INST]` / `[/INST]` blocks with `<<SYS>>` |
|
||||
| `kimi` / `moonshot` | `<\|im_system\|>` / `<\|im_user\|>` / `<\|im_assistant\|>` |
|
||||
| *(other)* | Generic `System:` / `User:` / `Assistant:` format |
|
||||
|
||||
Template selection uses the requested model name and any configured aliases, so renaming a model to include a family hint (e.g. `qwen-coder`) is sufficient.
|
||||
|
||||
---
|
||||
|
||||
## Coding defaults
|
||||
|
||||
When no system prompt is provided, ForgeLM injects a coding-focused default:
|
||||
|
||||
> *You are a precise coding assistant running locally. Prioritize correct code, minimal speculation, explicit assumptions, and actionable fixes. When editing or generating code, prefer complete snippets, preserve requested APIs, and explain risks briefly.*
|
||||
|
||||
Sampling defaults are also tuned for code generation:
|
||||
- `temperature: 0.15` (low randomness)
|
||||
- `top_p: 0.95`
|
||||
- `max_tokens: 1024`
|
||||
|
||||
Override any of these per-request or globally via `LOCALLLM_SYSTEM_PROMPT`.
|
||||
+1173
File diff suppressed because it is too large.
Load diff
Reference in new issue
Block a user