ForgeLM/program.md

Research Program: Response Quality Optimization

Goal

Optimize the SYSTEM_PROMPT to maximize response quality and accuracy from local LLMs (Qwen3 0.8B-4B range).

Current State

• System prompt is minimal (1 sentence)
• Works for basic queries but may lack guidance for:
  • Consistent response format
  • Appropriate detail level
  • Language consistency
  • Avoiding hallucinations

Hypotheses to Test

1. Role Clarity

Adding a clearer role definition might improve helpfulness.

• "You are a helpful assistant" → more specific role?
• Test: "You are ForgeLM, a helpful local AI assistant"

2. Response Guidelines

Adding lightweight guidance on response style.

• Keep it short to avoid token overhead
• Test adding: "Be concise but thorough"

3. Language Consistency

Current: "Respond in the same language as the user"

• Might be too vague for small models
• Test: "Detect the user's language from their message and respond in that language"

4. Hallucination Reduction

Adding a gentle reminder to stick to facts.

• Risk: might cause refusals on small models
• Test: "Only state facts you are confident about"

5. Instruction Following

Testing if specific formatting helps.

• "Use markdown for code blocks"
• "Bullet points for lists"

Constraints

• MUST stay under ~50 tokens to preserve context window
• MUST work with 0.8B parameter models
• MUST NOT cause refusal loops
• Positive framing only (no "don't" statements)

Success Criteria

• Test pass rate > 95%
• No regression in existing functionality
• Maintain compatibility across model sizes

Past Learnings

(Space to record what worked/didn't work)