Problems:
1. Stream response missing 'data: [DONE]' terminator — OpenAI clients
(openai-python, LiteLLM, etc.) expect this SSE sentinel to detect
end-of-stream. Without it, clients hang or raise IncompleteRead.
2. Stream chunks lack required fields (id, object, created, model) —
the OpenAI spec mandates chat.completion.chunk objects contain these
metadata fields. Clients that validate response shape reject the
bare {choices:[{delta:{}}]} payloads.
3. Non-stream path uses max_length instead of max_new_tokens — stream
path correctly uses max_new_tokens, but non-stream passes
max_length=prompt_len+max_tokens. This creates inconsistent behavior
(different effective generation limits) between the two code paths.
4. Generation thread exceptions silently swallowed — if model.generate()
throws (OOM, dtype mismatch, etc.), the queue never receives None,
causing the consumer to block forever on queue.get(). Added sentinel-
based error propagation so exceptions surface to the SSE generator.
5. No CORS headers — browser-based clients (web UIs, chatbots) get
blocked by same-origin policy. Added CORSMiddleware.
6. Generation thread not daemonized — if the main process exits during
generation (e.g. client disconnect + server shutdown), the non-daemon
thread keeps the process alive. Set daemon=True.
Tests:
- 7 tests for parse_response (plain text, think tags, tool calls, edge cases)
- 6 tests for SSE stream format (DONE marker, chunk fields, CORS, max_new_tokens, daemon, error propagation)
voidborne-d
jingyaogong
gongjy