Merge pull request #864 from Linxiushen/fix-nonstream-max-tokens

[fix] serve_openai_api 非流式端点的 max_tokens 不生效，与流式端点行为不一致
2026-09-20 19:42:05 +08:00 · 2026-09-20 19:42:05 +08:00 · d698c06fc5
parent cc312c1cc6 d4b67f1783
commit d698c06fc5
1 changed files with 1 additions and 1 deletions
--- a/scripts/serve_openai_api.py
+++ b/scripts/serve_openai_api.py
@ -202,7 +202,7 @@ async def chat_completions(request: ChatRequest):
            with torch.no_grad():
                generated_ids = model.generate(
                    inputs["input_ids"],
-                    max_length=inputs["input_ids"].shape[1] + request.max_tokens,
+                    max_new_tokens=request.max_tokens,
                    do_sample=True,
                    attention_mask=inputs["attention_mask"],
                    pad_token_id=tokenizer.pad_token_id,