54 lines
1.6 KiB
Markdown
54 lines
1.6 KiB
Markdown
# Research Program: Response Quality Optimization
|
|
|
|
## Goal
|
|
Optimize the SYSTEM_PROMPT to maximize response quality and accuracy from local LLMs (Qwen3 0.8B-4B range).
|
|
|
|
## Current State
|
|
- System prompt is minimal (1 sentence)
|
|
- Works for basic queries but may lack guidance for:
|
|
- Consistent response format
|
|
- Appropriate detail level
|
|
- Language consistency
|
|
- Avoiding hallucinations
|
|
|
|
## Hypotheses to Test
|
|
|
|
### 1. Role Clarity
|
|
Adding a clearer role definition might improve helpfulness.
|
|
- "You are a helpful assistant" → more specific role?
|
|
- Test: "You are ForgeLM, a helpful local AI assistant"
|
|
|
|
### 2. Response Guidelines
|
|
Adding lightweight guidance on response style.
|
|
- Keep it short to avoid token overhead
|
|
- Test adding: "Be concise but thorough"
|
|
|
|
### 3. Language Consistency
|
|
Current: "Respond in the same language as the user"
|
|
- Might be too vague for small models
|
|
- Test: "Detect the user's language from their message and respond in that language"
|
|
|
|
### 4. Hallucination Reduction
|
|
Adding a gentle reminder to stick to facts.
|
|
- Risk: might cause refusals on small models
|
|
- Test: "Only state facts you are confident about"
|
|
|
|
### 5. Instruction Following
|
|
Testing if specific formatting helps.
|
|
- "Use markdown for code blocks"
|
|
- "Bullet points for lists"
|
|
|
|
## Constraints
|
|
- MUST stay under ~50 tokens to preserve context window
|
|
- MUST work with 0.8B parameter models
|
|
- MUST NOT cause refusal loops
|
|
- Positive framing only (no "don't" statements)
|
|
|
|
## Success Criteria
|
|
- Test pass rate > 95%
|
|
- No regression in existing functionality
|
|
- Maintain compatibility across model sizes
|
|
|
|
## Past Learnings
|
|
(Space to record what worked/didn't work)
|