Summary
When a chat upload goes through the central vision model, the generated description is appended to the user's message bare (src/chat_handler.py, enhanced_message = f"{enhanced_message}\n\n[Image: {name}]\n{vl_desc}"). The chat model has no way to tell the description apart from the user's own writing, so it responds to the prose instead of the image — in one documented case complimenting the user's "professional food-critic writing" that was actually written by the vision model.
Note that this holds even when the same model serves both chairs (vision_model == chat default): the description is produced in a separate call that the chat turn has no memory of, so the chat instance still reads its own words as the user's. Only a direct multimodal pass (the chat model receiving the image itself, no describe-then-inject pipeline) makes labeling moot — and that is not this pipeline.
Fix (tested locally)
Label the description as machine-generated at the injection point:
[Attached image: <name> — description below was auto-generated by <vision model>, not written by the user]
One string change in src/chat_handler.py. Models stop reviewing the user's prose and answer about the image. Running in a self-hosted deployment since 13 July.
Related
- The same string is what the client renders as the collapsible "image description" on the user bubble — the label survives that path harmlessly.
Summary
When a chat upload goes through the central vision model, the generated description is appended to the user's message bare (src/chat_handler.py,
enhanced_message = f"{enhanced_message}\n\n[Image: {name}]\n{vl_desc}"). The chat model has no way to tell the description apart from the user's own writing, so it responds to the prose instead of the image — in one documented case complimenting the user's "professional food-critic writing" that was actually written by the vision model.Note that this holds even when the same model serves both chairs (vision_model == chat default): the description is produced in a separate call that the chat turn has no memory of, so the chat instance still reads its own words as the user's. Only a direct multimodal pass (the chat model receiving the image itself, no describe-then-inject pipeline) makes labeling moot — and that is not this pipeline.
Fix (tested locally)
Label the description as machine-generated at the injection point:
[Attached image: <name> — description below was auto-generated by <vision model>, not written by the user]One string change in src/chat_handler.py. Models stop reviewing the user's prose and answer about the image. Running in a self-hosted deployment since 13 July.
Related