evandro/ForgeLM
1
0 Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f5a4a9ee971cff002f69a8b76834e33d61bd1b36
ForgeLM/docs/API.md
T

10 KiB
Raw Blame History

ForgeLM API Reference

ForgeLM exposes an OpenAI-compatible REST API for local LLM inference. Any client that works with the OpenAI API (VSCode extensions, OpenCode, Cline, Roo Code, curl, Python openai SDK, etc.) can point at this server with no code changes.

Base URL

http://127.0.0.1:8080

Configure with the LOCALLLM_ADDR environment variable (default 127.0.0.1:8080).

Authentication

Authentication is optional and disabled by default.

Env var Description
LOCALLLM_API_KEY If set, all requests must include this key
LOCALLLM_AUTH_HEADER Header name to check (default: authorization)
LOCALLLM_AUTH_SCHEME Expected scheme prefix (default: Bearer)

Sending a key:

Authorization: Bearer <your-key>

or via x-api-key:

x-api-key: <your-key>

If no key is configured, all requests are accepted without authentication.

CORS

CORS is enabled for all origins by default (*). Restrict with:

LOCALLLM_CORS_ORIGINS=https://myapp.com,http://localhost:3000

Endpoints

GET /health

Returns server and model status.

Response 200 OK:

{
  "status": "ok",
  "model_loaded": true,
  "model_id": "Qwen3.5-4B-Abliterated.i1-Q6_K",
  "uptime_secs": 142
}
Field Type Description
status string Always "ok" when server is running
model_loaded boolean Whether a model is currently loaded
model_id string | null Canonical model identifier
uptime_secs integer Seconds since server start

GET /v1/models

Lists available models. Compatible with the OpenAI models endpoint.

Response 200 OK:

{
  "object": "list",
  "data": [
    {
      "id": "Qwen3.5-4B-Abliterated.i1-Q6_K",
      "object": "model",
      "created": 1711497600,
      "owned_by": "local",
      "root": "Qwen3.5-4B-Abliterated.i1-Q6_K"
    }
  ]
}

Model aliases (configured via LOCALLLM_MODEL_ALIASES) also appear in the list and resolve to the same underlying model.

POST /v1/chat/completions

The main inference endpoint. Fully compatible with the OpenAI Chat Completions API.

Request body:

{
  "model": "Qwen3.5-4B-Abliterated.i1-Q6_K",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "Explain async/await in Rust." }
  ],
  "max_tokens": 1024,
  "temperature": 0.15,
  "top_p": 0.95,
  "stream": false
}

Request fields

Field Type Default Description
model string required Model ID or alias
messages array required Conversation history
max_tokens integer 1024 Max tokens to generate (alias: max_completion_tokens)
temperature float 0.15 Sampling temperature (0 = greedy)
top_p float 0.95 Nucleus sampling threshold
stream boolean false Enable SSE streaming
stop string | array [] Stop sequences
seed integer null Reproducibility seed
n integer 1 Number of completions (only 1 supported)
response_format object null {"type": "text"} or {"type": "json_object"}
tools array [] Tool definitions for function calling
tool_choice string | object null Tool selection policy
stream_options object null {"include_usage": true} to get usage in stream

Message roles

Role Description
system System-level instructions
developer Developer-level instructions (treated as system)
user User turn
assistant Previous assistant turn
tool Tool call result

Non-streaming response 200 OK

{
  "id": "chatcmpl-a1b2c3d4",
  "object": "chat.completion",
  "created": 1711497600,
  "model": "Qwen3.5-4B-Abliterated.i1-Q6_K",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "In Rust, `async/await` allows you to write asynchronous code..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 24,
    "completion_tokens": 87,
    "total_tokens": 111,
    "tokens_per_second": 52.4
  }
}

Streaming response (SSE)

Set "stream": true to receive Server-Sent Events:

data: {"id":"chatcmpl-a1b2c3d4","object":"chat.completion.chunk","created":1711497600,"model":"Qwen3.5-4B-Abliterated.i1-Q6_K","choices":[{"index":0,"delta":{"role":"assistant","content":"In"},"finish_reason":null}]}

data: {"id":"chatcmpl-a1b2c3d4","object":"chat.completion.chunk","created":1711497600,"model":"Qwen3.5-4B-Abliterated.i1-Q6_K","choices":[{"index":0,"delta":{"content":" Rust"},"finish_reason":null}]}

data: {"id":"chatcmpl-a1b2c3d4","object":"chat.completion.chunk","created":1711497600,"model":"Qwen3.5-4B-Abliterated.i1-Q6_K","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

To include usage in the final streaming chunk:

{
  "stream": true,
  "stream_options": { "include_usage": true }
}

Tool calling

{
  "model": "Qwen3.5-4B-Abliterated.i1-Q6_K",
  "messages": [{ "role": "user", "content": "What is the weather in Paris?" }],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {
            "city": { "type": "string" }
          },
          "required": ["city"]
        }
      }
    }
  ],
  "tool_choice": "auto"
}

When the model decides to call a tool, finish_reason is "tool_calls" and message.tool_calls contains the call:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": null,
      "tool_calls": [{
        "id": "call_abc123",
        "type": "function",
        "function": {
          "name": "get_weather",
          "arguments": "{\"city\": \"Paris\"}"
        }
      }]
    },
    "finish_reason": "tool_calls"
  }]
}

Error responses

All errors follow the OpenAI error format:

{
  "error": {
    "message": "The model 'unknown-model' does not exist",
    "type": "invalid_request_error",
    "param": "model",
    "code": "model_not_found"
  }
}
HTTP status type When
400 invalid_request_error Malformed request, empty messages, unsupported n
401 authentication_error Missing or invalid API key
404 invalid_request_error Unknown model
422 invalid_request_error Unsupported response_format
500 server_error Internal inference failure

Environment variables

Variable Default Description
LOCALLLM_MODEL (required) Path to the GGUF model file
LOCALLLM_ADDR 127.0.0.1:8080 Bind address
LOCALLLM_PROFILE balanced Runtime profile (fast, balanced, quality)
LOCALLLM_GPU_LAYERS 0 Number of layers to offload to GPU
LOCALLLM_CTX 4096 Context window size
LOCALLLM_BATCH 512 Batch size
LOCALLLM_UBATCH 512 Micro-batch size
LOCALLLM_THREADS (auto) CPU threads for generation
LOCALLLM_THREADS_BATCH (auto) CPU threads for batch processing
LOCALLLM_API_KEY (none) API key for authentication
LOCALLLM_AUTH_HEADER authorization Auth header name
LOCALLLM_AUTH_SCHEME Bearer Auth scheme prefix
LOCALLLM_CORS_ORIGINS * Comma-separated allowed origins
LOCALLLM_SYSTEM_PROMPT (coding assistant) Default system prompt
LOCALLLM_ENABLE_TOOL_CALLS false Enable tool calling support
LOCALLLM_MODEL_ALIASES (none) Comma-separated alias=model pairs

Quick start

Start the server

LOCALLLM_MODEL=./models/Qwen3.5-4B.gguf cargo run --release -p inference-server

Test with curl

# Health check
curl http://localhost:8080/health

# List models
curl http://localhost:8080/v1/models

# Chat completion
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3.5-4B",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'

# Streaming
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3.5-4B",
    "messages": [{"role": "user", "content": "Count to 5"}],
    "stream": true
  }'

Use with Python openai SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"  # or your LOCALLLM_API_KEY
)

response = client.chat.completions.create(
    model="Qwen3.5-4B",
    messages=[{"role": "user", "content": "Write a Python hello world"}],
    max_tokens=200
)
print(response.choices[0].message.content)

Use with VSCode / OpenCode / Cline / Roo Code

Set the following in your extension settings:

Setting Value
Base URL http://localhost:8080/v1
API Key (leave empty or set LOCALLLM_API_KEY)
Model (your model ID from /v1/models)

Chat template behavior

ForgeLM automatically selects the correct chat template based on the model family:

Model family Template
qwen <|im_start|> / <|im_end|> tokens
gemma <start_of_turn> / <end_of_turn> tokens
llama [INST] / [/INST] blocks with <<SYS>>
kimi / moonshot <|im_system|> / <|im_user|> / <|im_assistant|>
(other) Generic System: / User: / Assistant: format

Template selection uses the requested model name and any configured aliases, so renaming a model to include a family hint (e.g. qwen-coder) is sufficient.

Coding defaults

When no system prompt is provided, ForgeLM injects a coding-focused default:

You are a precise coding assistant running locally. Prioritize correct code, minimal speculation, explicit assumptions, and actionable fixes. When editing or generating code, prefer complete snippets, preserve requested APIs, and explain risks briefly.

Sampling defaults are also tuned for code generation:

  • temperature: 0.15 (low randomness)
  • top_p: 0.95
  • max_tokens: 1024

Override any of these per-request or globally via LOCALLLM_SYSTEM_PROMPT.

Reference in new issue View Git Blame Copy Permalink