Lemonade Experiences: model updates, UI fixes, and tool-use support

[ramkrishna2910]

Summary

• Experience model visibility: Fix frontend normalization rejecting experience models (empty checkpoint)
• Model sizing: Display summed component sizes instead of "N/A" for experience models
• Ultra tier update: Switch to Qwen3.5-35B-A3B-GGUF (LLM) + Flux-2-Klein-9B-GGUF (image gen + edit)
• Image editing scoping: Only Ultra exposes image editing (Edit mode + edit_image tool); Medium/Lite are generation-only
• Experience image rendering: Generated images in experience panel now use full image styling instead of 200x200 thumbnails
• Tool-use integration: LLM chat panel supports tool calling for image generation, editing, TTS, transcription, and vision via experience components
• Experience tools: Server-side experience_tools support and tool definitions JSON

Test plan

• Verify experience models (Ultra/Medium/Lite) appear in the Models tab
• Verify experience model sizes show correctly (not "N/A")
• Verify Ultra shows Edit mode in image panel; Medium/Lite do not
• Verify image generation works in experience mode with proper image rendering
• Verify edit_image tool is only offered in Ultra experience chat
• Verify Medium/Lite do not silently redirect generation to editing
• Run python test/server_endpoints.py against the built server

🤖 Generated with Claude Code

[ramkrishna2910]

@claude review

[github-actions]

Claude finished @ramkrishna2910's task in 2m 36s —— View job

---

PR Review: Lemonade Experiences — model updates, UI fixes, and tool-use support

• Gather context and understand PR scope
• Review security vulnerabilities (XSS/URL redirect flags)
• Review LLMChatPanel.tsx changes (tool-use integration)
• Review new lemonadeTools.ts and toolDefinitions.json
• Review C++ server changes (experience_tools, router, server)
• Review frontend fixes (model visibility, sizing, image rendering)
• Summarize findings

---

Security Issues (must fix before merge)

CodeQL is correct — there are real XSS/URL-redirect risks in the three flagged lines.

1. <img src={item.image_url.url}> — lines 1010 & 1017

// LLMChatPanel.tsx:1010
<img src={item.image_url.url} className="generated-image" />
// LLMChatPanel.tsx:1017
return <img key={index} src={item.image_url.url} className="message-image" />;

image_url.url comes from message history and could in principle hold a javascript: URI or an external URL. In Electron's renderer context (even with nodeIntegration: false) this is an information-leak vector — the browser will make an outbound request to the URL, potentially exfiltrating session context. Fix: validate the URL starts with data:image/ before rendering:

const isSafeDataUrl = (url: string) => url.startsWith('data:image/');
// then:
{isSafeDataUrl(item.image_url.url) && <img src={item.image_url.url} ... />}

Fix this →

2. <audio src={data:${audioItem.audio.mime};base64,...}> — line 1034

<audio controls src={`data:${audioItem.audio.mime};base64,${audioItem.audio.data}`} />

audioItem.audio.mime comes from user-uploaded file .type. A crafted MIME string (e.g. audio/mpeg; charset=utf-8, <script>) won't execute JS here but could produce an invalid data URI or be used to inject characters that break the surrounding attribute context in certain renderer environments. Fix: allowlist the mime type:

const SAFE_AUDIO_MIMES = new Set(['audio/wav', 'audio/mpeg', 'audio/mp3', 'audio/ogg', 'audio/flac', 'audio/webm', 'audio/m4a', 'audio/mp4']);
const safeMime = SAFE_AUDIO_MIMES.has(audioItem.audio.mime) ? audioItem.audio.mime : 'audio/wav';
// then: src={`data:${safeMime};base64,${audioItem.audio.data}`}

Fix this →

---

Dead Code (should be cleaned up)

3. experience_tools.cpp — functions never called

is_experience_model() and get_experience_llm_model() are compiled (CMakeLists.txt:581) but grep finds zero call sites in the C++ codebase. These are server-side stubs with no endpoint or callers. Either wire them up or remove them to avoid confusing future contributors.

4. Router::audio_speech_to_base64() — defined, never called

src/cpp/server/router.cpp adds this method (declared in router.h:74) but it is never invoked from server.cpp or anywhere else. Same situation as above — it's dead code that will silently rot.

---

Architecture / Design Concerns

5. NON_LLM_LABELS duplicated three times

The same set is defined independently in:

• src/app/src/renderer/utils/experienceModels.ts (line 3)
• src/app/src/renderer/utils/lemonadeTools.ts (line 35)
• src/cpp/server/experience_tools.cpp (lines 9-12)

The two TypeScript copies should at minimum share one definition (export from experienceModels.ts and import in lemonadeTools.ts). These will drift.

6. No streaming in experience mode (UX regression)

handleExperienceChat uses stream: false:

// LLMChatPanel.tsx:455
const requestBody = { ..., stream: false, ... };

For models that take 10–30 seconds to respond, the user sees a blank assistant bubble with no progress. The non-experience path streams tokens as they arrive. Consider adding a lightweight typing indicator or streaming the first non-tool response to restore the "live" feel.

7. Agentic loop silently hits MAX_ITERATIONS

When the 5-iteration cap is hit (line 452), buildFinalContent('', artifacts) is called with an empty text string. If no artifacts were produced, the user gets an empty assistant message with no explanation. Add a fallback message:

// after the loop:
setMessages(prev => {
  const updated = [...prev];
  updated[updated.length - 1] = {
    role: 'assistant',
    content: buildFinalContent('(Maximum tool iterations reached.)', artifacts),
  };
  return updated;
});

---

Code Quality

8. console.log in production agentic loop

// LLMChatPanel.tsx:453
console.log(`[LLMChat] Experience loop iteration ${iteration + 1}`);

This fires on every request in experience mode. Should be removed or gated on a debug flag before merge.

9. as any cast for JSON tool definitions

// lemonadeTools.ts:57-58
const requiresLabels = (def as any).requires_labels as string[] | undefined;
const requiresLlmLabels = (def as any).requires_llm_labels as string[] | undefined;

Since toolDefinitions.json has a known shape, declare a typed interface for it:

interface ToolEntry {
  requires_labels?: string[];
  requires_llm_labels?: string[];
  function: { name: string; description: string; parameters: Record<string, any> };
}

10. Silent args = {} on JSON parse failure

// lemonadeTools.ts:99-102
try {
  args = JSON.parse(toolCall.function.arguments);
} catch {
  args = {};  // ← silently swallows malformed arguments
}

This causes the tool to run with empty args (e.g. prompt: '' for image generation) rather than surfacing the error. At minimum log the parse failure.

---

What Looks Good

• Model visibility fix (modelData.ts:66): the recipe !== 'experience' checkpoint exemption correctly unblocks experience models from the normalizer filter.
• supportsEdit scoping (ImageGenerationPanel.tsx): cleanly delegates to getExperienceImageModel so Medium/Lite experience models correctly hide the Edit tab.
• buildLemonadeTools (lemonadeTools.ts): the label-matching approach for selecting component models per tool is clean and data-driven via toolDefinitions.json.
• Experience load in server.cpp: the component-by-component loading with skip-if-loaded logic is correct.
• model_manager.cpp size aggregation: summing component sizes for experience models is a straightforward fix.

---

Summary

The feature work is solid but the two XSS/URL-redirect security issues (flagged by CodeQL at lines 1010, 1017, 1034) should be fixed before merge. The dead C++ code (experience_tools.cpp, audio_speech_to_base64) and the NON_LLM_LABELS duplication are lower priority but worth cleaning up. The missing streaming feedback in experience mode and the silent iteration cap are UX issues worth addressing.

[jeremyfowers]

Suggested change

	curl http://localhost:8000/v1/models
	curl http://localhost:13305/v1/models

[jeremyfowers]

Lots of references to old port below as well

[jeremyfowers]

Should this be a python file so that people can try it?

[jeremyfowers]

Is there any point in having a Medium mode considering it has basically the same RAM requirement as the Ultra mode? Suggest just removing it.

[jeremyfowers]

When I'm first getting started, it says "Size N/A" and doesn't tell me what the download button will actually do.

Then when I click download, it says its downloading "Lemonade Ultra", but the file is actually a flux.gguf.

Can you refine the onboarding experience so that people understand they are signing up for many-gigabyte downloads and can track them appropriately?

Also it would be nice if it was clear they were downloading the models, not downloading "Lemonade Ultra" (since people might think we made our own model).

[jeremyfowers]

It also showed in the UI that the download was complete, even though only Flux had downloaded and I still needed Kokoro. Iterating over each model and pulling it if needed should solve this.

Maybe it was an architectural mistake to put the experiences into server_models.json? For example they also show up in the models endpoint in AnythingLLM.

[jeremyfowers]

I also had to download Ultra, then restart lemond to get Ultra to show up as available.

[jeremyfowers]

Two mic buttons, one overlapping the text entry

[jeremyfowers]

What am I doing wrong?

[jeremyfowers]

Hmm its not generating speech either, and wont admit that it has any tools available.