fix: map local model aliases to Qwen template to stop prompt echo loops
When using local aliases like Cooper, Iron, or Steel (which are Qwen-based architectures), the select_chat_template failed to identify them because the alias names didn't contain 'qwen'. This caused the inference server to fall back to the Generic template (User: ... \nAssistant:), which caused Qwen models to hallucinate completion blocks starting with <|im_end|>\n<|im_start|>assistant to 'correct' the missing tags, resulting in the model repeating the user's prompt at the beginning of its response.
This commit is contained in:
1 parent
99aff0be6f
commit
8f32b677da
1 file changed
+1
-1
@@ -461,7 +461,7 @@ fn select_chat_template(req: &ChatCompletionRequest, config: &ServerConfig) -> C
|
||||
for candidate in candidates {
|
||||
let normalized = candidate.to_ascii_lowercase();
|
||||
|
||||
if normalized.contains("qwen") {
|
||||
if normalized.contains("qwen") || normalized.contains("cooper") || normalized.contains("iron") || normalized.contains("steel") {
|
||||
return ChatTemplate::Qwen;
|
||||
}
|
||||
if normalized.contains("gemma") {
|
||||
|
||||
Reference in new issue
Block a user