From d0c2f86f3f5b6908f04f48fef6d8566f8b2c9524 Mon Sep 17 00:00:00 2001 From: Evandro Fonseca Junior Date: Thu, 26 Mar 2026 20:47:09 -0300 Subject: [PATCH] feat: add API docs and full-featured chat UI with SQLite persistence --- docs/API.md | 393 +++++++++++++++++ ui/chat.html | 1173 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 1566 insertions(+) create mode 100644 docs/API.md create mode 100644 ui/chat.html diff --git a/docs/API.md b/docs/API.md new file mode 100644 index 0000000..24efd05 --- /dev/null +++ b/docs/API.md @@ -0,0 +1,393 @@ +# ForgeLM API Reference + +ForgeLM exposes an **OpenAI-compatible REST API** for local LLM inference. Any client that works with the OpenAI API (VSCode extensions, OpenCode, Cline, Roo Code, curl, Python `openai` SDK, etc.) can point at this server with no code changes. + +--- + +## Base URL + +``` +http://127.0.0.1:8080 +``` + +Configure with the `LOCALLLM_ADDR` environment variable (default `127.0.0.1:8080`). + +--- + +## Authentication + +Authentication is **optional** and disabled by default. + +| Env var | Description | +|---------|-------------| +| `LOCALLLM_API_KEY` | If set, all requests must include this key | +| `LOCALLLM_AUTH_HEADER` | Header name to check (default: `authorization`) | +| `LOCALLLM_AUTH_SCHEME` | Expected scheme prefix (default: `Bearer`) | + +**Sending a key:** + +```http +Authorization: Bearer +``` + +or via `x-api-key`: + +```http +x-api-key: +``` + +If no key is configured, all requests are accepted without authentication. + +--- + +## CORS + +CORS is enabled for all origins by default (`*`). Restrict with: + +``` +LOCALLLM_CORS_ORIGINS=https://myapp.com,http://localhost:3000 +``` + +--- + +## Endpoints + +### `GET /health` + +Returns server and model status. + +**Response `200 OK`:** + +```json +{ + "status": "ok", + "model_loaded": true, + "model_id": "Qwen3.5-4B-Abliterated.i1-Q6_K", + "uptime_secs": 142 +} +``` + +| Field | Type | Description | +|-------|------|-------------| +| `status` | `string` | Always `"ok"` when server is running | +| `model_loaded` | `boolean` | Whether a model is currently loaded | +| `model_id` | `string \| null` | Canonical model identifier | +| `uptime_secs` | `integer` | Seconds since server start | + +--- + +### `GET /v1/models` + +Lists available models. Compatible with the OpenAI models endpoint. + +**Response `200 OK`:** + +```json +{ + "object": "list", + "data": [ + { + "id": "Qwen3.5-4B-Abliterated.i1-Q6_K", + "object": "model", + "created": 1711497600, + "owned_by": "local", + "root": "Qwen3.5-4B-Abliterated.i1-Q6_K" + } + ] +} +``` + +Model aliases (configured via `LOCALLLM_MODEL_ALIASES`) also appear in the list and resolve to the same underlying model. + +--- + +### `POST /v1/chat/completions` + +The main inference endpoint. Fully compatible with the OpenAI Chat Completions API. + +**Request body:** + +```json +{ + "model": "Qwen3.5-4B-Abliterated.i1-Q6_K", + "messages": [ + { "role": "system", "content": "You are a helpful assistant." }, + { "role": "user", "content": "Explain async/await in Rust." } + ], + "max_tokens": 1024, + "temperature": 0.15, + "top_p": 0.95, + "stream": false +} +``` + +#### Request fields + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `model` | `string` | **required** | Model ID or alias | +| `messages` | `array` | **required** | Conversation history | +| `max_tokens` | `integer` | `1024` | Max tokens to generate (alias: `max_completion_tokens`) | +| `temperature` | `float` | `0.15` | Sampling temperature (0 = greedy) | +| `top_p` | `float` | `0.95` | Nucleus sampling threshold | +| `stream` | `boolean` | `false` | Enable SSE streaming | +| `stop` | `string \| array` | `[]` | Stop sequences | +| `seed` | `integer` | `null` | Reproducibility seed | +| `n` | `integer` | `1` | Number of completions (only `1` supported) | +| `response_format` | `object` | `null` | `{"type": "text"}` or `{"type": "json_object"}` | +| `tools` | `array` | `[]` | Tool definitions for function calling | +| `tool_choice` | `string \| object` | `null` | Tool selection policy | +| `stream_options` | `object` | `null` | `{"include_usage": true}` to get usage in stream | + +#### Message roles + +| Role | Description | +|------|-------------| +| `system` | System-level instructions | +| `developer` | Developer-level instructions (treated as system) | +| `user` | User turn | +| `assistant` | Previous assistant turn | +| `tool` | Tool call result | + +#### Non-streaming response `200 OK` + +```json +{ + "id": "chatcmpl-a1b2c3d4", + "object": "chat.completion", + "created": 1711497600, + "model": "Qwen3.5-4B-Abliterated.i1-Q6_K", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "In Rust, `async/await` allows you to write asynchronous code..." + }, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 24, + "completion_tokens": 87, + "total_tokens": 111, + "tokens_per_second": 52.4 + } +} +``` + +#### Streaming response (SSE) + +Set `"stream": true` to receive Server-Sent Events: + +``` +data: {"id":"chatcmpl-a1b2c3d4","object":"chat.completion.chunk","created":1711497600,"model":"Qwen3.5-4B-Abliterated.i1-Q6_K","choices":[{"index":0,"delta":{"role":"assistant","content":"In"},"finish_reason":null}]} + +data: {"id":"chatcmpl-a1b2c3d4","object":"chat.completion.chunk","created":1711497600,"model":"Qwen3.5-4B-Abliterated.i1-Q6_K","choices":[{"index":0,"delta":{"content":" Rust"},"finish_reason":null}]} + +data: {"id":"chatcmpl-a1b2c3d4","object":"chat.completion.chunk","created":1711497600,"model":"Qwen3.5-4B-Abliterated.i1-Q6_K","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]} + +data: [DONE] +``` + +To include usage in the final streaming chunk: + +```json +{ + "stream": true, + "stream_options": { "include_usage": true } +} +``` + +#### Tool calling + +```json +{ + "model": "Qwen3.5-4B-Abliterated.i1-Q6_K", + "messages": [{ "role": "user", "content": "What is the weather in Paris?" }], + "tools": [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get current weather for a city", + "parameters": { + "type": "object", + "properties": { + "city": { "type": "string" } + }, + "required": ["city"] + } + } + } + ], + "tool_choice": "auto" +} +``` + +When the model decides to call a tool, `finish_reason` is `"tool_calls"` and `message.tool_calls` contains the call: + +```json +{ + "choices": [{ + "message": { + "role": "assistant", + "content": null, + "tool_calls": [{ + "id": "call_abc123", + "type": "function", + "function": { + "name": "get_weather", + "arguments": "{\"city\": \"Paris\"}" + } + }] + }, + "finish_reason": "tool_calls" + }] +} +``` + +--- + +## Error responses + +All errors follow the OpenAI error format: + +```json +{ + "error": { + "message": "The model 'unknown-model' does not exist", + "type": "invalid_request_error", + "param": "model", + "code": "model_not_found" + } +} +``` + +| HTTP status | `type` | When | +|-------------|--------|------| +| `400` | `invalid_request_error` | Malformed request, empty messages, unsupported `n` | +| `401` | `authentication_error` | Missing or invalid API key | +| `404` | `invalid_request_error` | Unknown model | +| `422` | `invalid_request_error` | Unsupported `response_format` | +| `500` | `server_error` | Internal inference failure | + +--- + +## Environment variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `LOCALLLM_MODEL` | *(required)* | Path to the GGUF model file | +| `LOCALLLM_ADDR` | `127.0.0.1:8080` | Bind address | +| `LOCALLLM_PROFILE` | `balanced` | Runtime profile (`fast`, `balanced`, `quality`) | +| `LOCALLLM_GPU_LAYERS` | `0` | Number of layers to offload to GPU | +| `LOCALLLM_CTX` | `4096` | Context window size | +| `LOCALLLM_BATCH` | `512` | Batch size | +| `LOCALLLM_UBATCH` | `512` | Micro-batch size | +| `LOCALLLM_THREADS` | *(auto)* | CPU threads for generation | +| `LOCALLLM_THREADS_BATCH` | *(auto)* | CPU threads for batch processing | +| `LOCALLLM_API_KEY` | *(none)* | API key for authentication | +| `LOCALLLM_AUTH_HEADER` | `authorization` | Auth header name | +| `LOCALLLM_AUTH_SCHEME` | `Bearer` | Auth scheme prefix | +| `LOCALLLM_CORS_ORIGINS` | `*` | Comma-separated allowed origins | +| `LOCALLLM_SYSTEM_PROMPT` | *(coding assistant)* | Default system prompt | +| `LOCALLLM_ENABLE_TOOL_CALLS` | `false` | Enable tool calling support | +| `LOCALLLM_MODEL_ALIASES` | *(none)* | Comma-separated `alias=model` pairs | + +--- + +## Quick start + +### Start the server + +```bash +LOCALLLM_MODEL=./models/Qwen3.5-4B.gguf cargo run --release -p inference-server +``` + +### Test with curl + +```bash +# Health check +curl http://localhost:8080/health + +# List models +curl http://localhost:8080/v1/models + +# Chat completion +curl http://localhost:8080/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen3.5-4B", + "messages": [{"role": "user", "content": "Hello!"}], + "max_tokens": 100 + }' + +# Streaming +curl http://localhost:8080/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen3.5-4B", + "messages": [{"role": "user", "content": "Count to 5"}], + "stream": true + }' +``` + +### Use with Python openai SDK + +```python +from openai import OpenAI + +client = OpenAI( + base_url="http://localhost:8080/v1", + api_key="not-needed" # or your LOCALLLM_API_KEY +) + +response = client.chat.completions.create( + model="Qwen3.5-4B", + messages=[{"role": "user", "content": "Write a Python hello world"}], + max_tokens=200 +) +print(response.choices[0].message.content) +``` + +### Use with VSCode / OpenCode / Cline / Roo Code + +Set the following in your extension settings: + +| Setting | Value | +|---------|-------| +| Base URL | `http://localhost:8080/v1` | +| API Key | *(leave empty or set `LOCALLLM_API_KEY`)* | +| Model | *(your model ID from `/v1/models`)* | + +--- + +## Chat template behavior + +ForgeLM automatically selects the correct chat template based on the model family: + +| Model family | Template | +|-------------|----------| +| `qwen` | `<\|im_start\|>` / `<\|im_end\|>` tokens | +| `gemma` | `` / `` tokens | +| `llama` | `[INST]` / `[/INST]` blocks with `<>` | +| `kimi` / `moonshot` | `<\|im_system\|>` / `<\|im_user\|>` / `<\|im_assistant\|>` | +| *(other)* | Generic `System:` / `User:` / `Assistant:` format | + +Template selection uses the requested model name and any configured aliases, so renaming a model to include a family hint (e.g. `qwen-coder`) is sufficient. + +--- + +## Coding defaults + +When no system prompt is provided, ForgeLM injects a coding-focused default: + +> *You are a precise coding assistant running locally. Prioritize correct code, minimal speculation, explicit assumptions, and actionable fixes. When editing or generating code, prefer complete snippets, preserve requested APIs, and explain risks briefly.* + +Sampling defaults are also tuned for code generation: +- `temperature: 0.15` (low randomness) +- `top_p: 0.95` +- `max_tokens: 1024` + +Override any of these per-request or globally via `LOCALLLM_SYSTEM_PROMPT`. diff --git a/ui/chat.html b/ui/chat.html new file mode 100644 index 0000000..814a2bd --- /dev/null +++ b/ui/chat.html @@ -0,0 +1,1173 @@ + + + + + + ForgeLM Chat + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + +
+
+

ForgeLM Chat

+

Start a conversation or select a chat from the sidebar. Supports full Markdown, code highlighting, and persistent history.

+
+
+ +
+
+ + +
+
+ + + 0 chars +
+
+
+ +
+ + + +