fix: map local model aliases to Qwen template to stop prompt echo loops
When using local aliases like Cooper, Iron, or Steel (which are Qwen-based architectures), the select_chat_template failed to identify them because the alias names didn't contain 'qwen'. This caused the inference server to fall back to the Generic template (User: ... \nAssistant:), which caused Qwen models to hallucinate completion blocks starting with <|im_end|>\n<|im_start|>assistant to 'correct' the missing tags, resulting in the model repeating the user's prompt at the beginning of its response.
This commit is contained in:
1 parent
99aff0be6f
commit
8f32b677da
1 file changed
+1
-1
@@ -461,7 +461,7 @@ fn select_chat_template(req: &ChatCompletionRequest, config: &ServerConfig) -> C
|
|||||||
for candidate in candidates {
|
for candidate in candidates {
|
||||||
let normalized = candidate.to_ascii_lowercase();
|
let normalized = candidate.to_ascii_lowercase();
|
||||||
|
|
||||||
if normalized.contains("qwen") {
|
if normalized.contains("qwen") || normalized.contains("cooper") || normalized.contains("iron") || normalized.contains("steel") {
|
||||||
return ChatTemplate::Qwen;
|
return ChatTemplate::Qwen;
|
||||||
}
|
}
|
||||||
if normalized.contains("gemma") {
|
if normalized.contains("gemma") {
|
||||||
|
|||||||
Reference in new issue
Block a user