web_demo.py rendered the full user message in the chat bubble but stored
prompt[-max_new_tokens:] in the message history, so the model received the
user's input front-truncated by *character* count using the generation-length
slider — the UI and the model saw different text, and the beginning of long
questions was silently dropped.
serve_openai_api.py applied the same [-max_tokens:] slice to the rendered
chat template, which corrupts the <|im_start|> structure (e.g. drops the
system block / cuts mid special-token) whenever a client sends a small
max_tokens. max_tokens is the *output* budget and has nothing to do with
trimming the input.
Store the full prompt instead. Input context is already bounded at the token
level by tokenizer(..., truncation=True) (model_max_length=131072).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
swaylq
jingyaogong
gongjy