# ForgeLM API Reference

ForgeLM exposes an **OpenAI-compatible REST API** for local LLM inference. Any client that works with the OpenAI API (VSCode extensions, OpenCode, Cline, Roo Code, curl, Python `openai` SDK, etc.) can point at this server with no code changes.

---

## Base URL

```
http://127.0.0.1:8080
```

Configure with the `LOCALLLM_ADDR` environment variable (default `127.0.0.1:8080`).

---

## Authentication

Authentication is **optional** and disabled by default.

| Env var | Description |
|---------|-------------|
| `LOCALLLM_API_KEY` | If set, all requests must include this key |
| `LOCALLLM_AUTH_HEADER` | Header name to check (default: `authorization`) |
| `LOCALLLM_AUTH_SCHEME` | Expected scheme prefix (default: `Bearer`) |

**Sending a key:**

```http
Authorization: Bearer <your-key>
```

or via `x-api-key`:

```http
x-api-key: <your-key>
```

If no key is configured, all requests are accepted without authentication.

---

## CORS

CORS is enabled for all origins by default (`*`). Restrict with:

```
LOCALLLM_CORS_ORIGINS=https://myapp.com,http://localhost:3000
```

---

## Endpoints

### `GET /health`

Returns server and model status.

**Response `200 OK`:**

```json
{
  "status": "ok",
  "model_loaded": true,
  "model_id": "Qwen3.5-4B-Abliterated.i1-Q6_K",
  "uptime_secs": 142
}
```

| Field | Type | Description |
|-------|------|-------------|
| `status` | `string` | Always `"ok"` when server is running |
| `model_loaded` | `boolean` | Whether a model is currently loaded |
| `model_id` | `string \| null` | Canonical model identifier |
| `uptime_secs` | `integer` | Seconds since server start |

---

### `GET /v1/models`

Lists available models. Compatible with the OpenAI models endpoint.

**Response `200 OK`:**

```json
{
  "object": "list",
  "data": [
    {
      "id": "Qwen3.5-4B-Abliterated.i1-Q6_K",
      "object": "model",
      "created": 1711497600,
      "owned_by": "local",
      "root": "Qwen3.5-4B-Abliterated.i1-Q6_K"
    }
  ]
}
```

Model aliases (configured via `LOCALLLM_MODEL_ALIASES`) also appear in the list and resolve to the same underlying model.

---

### `POST /v1/chat/completions`

The main inference endpoint. Fully compatible with the OpenAI Chat Completions API.

**Request body:**

```json
{
  "model": "Qwen3.5-4B-Abliterated.i1-Q6_K",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "Explain async/await in Rust." }
  ],
  "max_tokens": 1024,
  "temperature": 0.15,
  "top_p": 0.95,
  "stream": false
}
```

#### Request fields

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `model` | `string` | **required** | Model ID or alias |
| `messages` | `array` | **required** | Conversation history |
| `max_tokens` | `integer` | `1024` | Max tokens to generate (alias: `max_completion_tokens`) |
| `temperature` | `float` | `0.15` | Sampling temperature (0 = greedy) |
| `top_p` | `float` | `0.95` | Nucleus sampling threshold |
| `stream` | `boolean` | `false` | Enable SSE streaming |
| `stop` | `string \| array` | `[]` | Stop sequences |
| `seed` | `integer` | `null` | Reproducibility seed |
| `n` | `integer` | `1` | Number of completions (only `1` supported) |
| `response_format` | `object` | `null` | `{"type": "text"}` or `{"type": "json_object"}` |
| `tools` | `array` | `[]` | Tool definitions for function calling |
| `tool_choice` | `string \| object` | `null` | Tool selection policy |
| `stream_options` | `object` | `null` | `{"include_usage": true}` to get usage in stream |

#### Message roles

| Role | Description |
|------|-------------|
| `system` | System-level instructions |
| `developer` | Developer-level instructions (treated as system) |
| `user` | User turn |
| `assistant` | Previous assistant turn |
| `tool` | Tool call result |

#### Non-streaming response `200 OK`

```json
{
  "id": "chatcmpl-a1b2c3d4",
  "object": "chat.completion",
  "created": 1711497600,
  "model": "Qwen3.5-4B-Abliterated.i1-Q6_K",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "In Rust, `async/await` allows you to write asynchronous code..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 24,
    "completion_tokens": 87,
    "total_tokens": 111,
    "tokens_per_second": 52.4
  }
}
```

#### Streaming response (SSE)

Set `"stream": true` to receive Server-Sent Events:

```
data: {"id":"chatcmpl-a1b2c3d4","object":"chat.completion.chunk","created":1711497600,"model":"Qwen3.5-4B-Abliterated.i1-Q6_K","choices":[{"index":0,"delta":{"role":"assistant","content":"In"},"finish_reason":null}]}

data: {"id":"chatcmpl-a1b2c3d4","object":"chat.completion.chunk","created":1711497600,"model":"Qwen3.5-4B-Abliterated.i1-Q6_K","choices":[{"index":0,"delta":{"content":" Rust"},"finish_reason":null}]}

data: {"id":"chatcmpl-a1b2c3d4","object":"chat.completion.chunk","created":1711497600,"model":"Qwen3.5-4B-Abliterated.i1-Q6_K","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]
```

To include usage in the final streaming chunk:

```json
{
  "stream": true,
  "stream_options": { "include_usage": true }
}
```

#### Tool calling

```json
{
  "model": "Qwen3.5-4B-Abliterated.i1-Q6_K",
  "messages": [{ "role": "user", "content": "What is the weather in Paris?" }],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {
            "city": { "type": "string" }
          },
          "required": ["city"]
        }
      }
    }
  ],
  "tool_choice": "auto"
}
```

When the model decides to call a tool, `finish_reason` is `"tool_calls"` and `message.tool_calls` contains the call:

```json
{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": null,
      "tool_calls": [{
        "id": "call_abc123",
        "type": "function",
        "function": {
          "name": "get_weather",
          "arguments": "{\"city\": \"Paris\"}"
        }
      }]
    },
    "finish_reason": "tool_calls"
  }]
}
```

---

## Error responses

All errors follow the OpenAI error format:

```json
{
  "error": {
    "message": "The model 'unknown-model' does not exist",
    "type": "invalid_request_error",
    "param": "model",
    "code": "model_not_found"
  }
}
```

| HTTP status | `type` | When |
|-------------|--------|------|
| `400` | `invalid_request_error` | Malformed request, empty messages, unsupported `n` |
| `401` | `authentication_error` | Missing or invalid API key |
| `404` | `invalid_request_error` | Unknown model |
| `422` | `invalid_request_error` | Unsupported `response_format` |
| `500` | `server_error` | Internal inference failure |

---

## Environment variables

| Variable | Default | Description |
|----------|---------|-------------|
| `LOCALLLM_MODEL` | *(required)* | Path to the GGUF model file |
| `LOCALLLM_ADDR` | `127.0.0.1:8080` | Bind address |
| `LOCALLLM_PROFILE` | `balanced` | Runtime profile (`fast`, `balanced`, `quality`) |
| `LOCALLLM_GPU_LAYERS` | `0` | Number of layers to offload to GPU |
| `LOCALLLM_CTX` | `4096` | Context window size |
| `LOCALLLM_BATCH` | `512` | Batch size |
| `LOCALLLM_UBATCH` | `512` | Micro-batch size |
| `LOCALLLM_THREADS` | *(auto)* | CPU threads for generation |
| `LOCALLLM_THREADS_BATCH` | *(auto)* | CPU threads for batch processing |
| `LOCALLLM_API_KEY` | *(none)* | API key for authentication |
| `LOCALLLM_AUTH_HEADER` | `authorization` | Auth header name |
| `LOCALLLM_AUTH_SCHEME` | `Bearer` | Auth scheme prefix |
| `LOCALLLM_CORS_ORIGINS` | `*` | Comma-separated allowed origins |
| `LOCALLLM_SYSTEM_PROMPT` | *(coding assistant)* | Default system prompt |
| `LOCALLLM_ENABLE_TOOL_CALLS` | `false` | Enable tool calling support |
| `LOCALLLM_MODEL_ALIASES` | *(none)* | Comma-separated `alias=model` pairs |

---

## Quick start

### Start the server

```bash
LOCALLLM_MODEL=./models/Qwen3.5-4B.gguf cargo run --release -p inference-server
```

### Test with curl

```bash
# Health check
curl http://localhost:8080/health

# List models
curl http://localhost:8080/v1/models

# Chat completion
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3.5-4B",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'

# Streaming
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3.5-4B",
    "messages": [{"role": "user", "content": "Count to 5"}],
    "stream": true
  }'
```

### Use with Python openai SDK

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"  # or your LOCALLLM_API_KEY
)

response = client.chat.completions.create(
    model="Qwen3.5-4B",
    messages=[{"role": "user", "content": "Write a Python hello world"}],
    max_tokens=200
)
print(response.choices[0].message.content)
```

### Use with VSCode / OpenCode / Cline / Roo Code

Set the following in your extension settings:

| Setting | Value |
|---------|-------|
| Base URL | `http://localhost:8080/v1` |
| API Key | *(leave empty or set `LOCALLLM_API_KEY`)* |
| Model | *(your model ID from `/v1/models`)* |

---

## Chat template behavior

ForgeLM automatically selects the correct chat template based on the model family:

| Model family | Template |
|-------------|----------|
| `qwen` | `<\|im_start\|>` / `<\|im_end\|>` tokens |
| `gemma` | `<start_of_turn>` / `<end_of_turn>` tokens |
| `llama` | `[INST]` / `[/INST]` blocks with `<<SYS>>` |
| `kimi` / `moonshot` | `<\|im_system\|>` / `<\|im_user\|>` / `<\|im_assistant\|>` |
| *(other)* | Generic `System:` / `User:` / `Assistant:` format |

Template selection uses the requested model name and any configured aliases, so renaming a model to include a family hint (e.g. `qwen-coder`) is sufficient.

---

## Coding defaults

When no system prompt is provided, ForgeLM injects a coding-focused default:

> *You are a precise coding assistant running locally. Prioritize correct code, minimal speculation, explicit assumptions, and actionable fixes. When editing or generating code, prefer complete snippets, preserve requested APIs, and explain risks briefly.*

Sampling defaults are also tuned for code generation:
- `temperature: 0.15` (low randomness)
- `top_p: 0.95`
- `max_tokens: 1024`

Override any of these per-request or globally via `LOCALLLM_SYSTEM_PROMPT`.