-
-
Notifications
You must be signed in to change notification settings - Fork 6.6k
💻 feat: Deeper MCP UI integration in the Chat UI #9669
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
9933de0 to
2fc4441
Compare
|
|
||
| // Store UI resources when output is available | ||
| useEffect(() => { | ||
| if (output && attachments && !isSubmitting) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
instead of adding this useEffect, can you leverage the existing useAttachments hook?
client/src/hooks/Messages/useAttachments.ts
packages/api/src/mcp/utils.ts
Outdated
| } | ||
| } | ||
|
|
||
| export const uiResourcesInstructions = `The tool response contains UI resources (i.e. Resource URI starting with "ui://"), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's unnecessary to couple this with artifacts. We can easily show the UI resources adjacent to the tool call, or even in place of the tool call, without needing the LLM to "display" it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's also possible to create a separate plugin for MCP UI resources if LLM input for rendering is what's desired, similar to the one for web search:
client/src/components/Web/plugin.ts
this would also make it "easier" for the LLM so the "artifacts" boilerplate is not necessary. In my experience, it's best to reduce the amount the LLM has to generate and even key the specific "attachment" to render, so it doesn't need to generate a long string to render the expected output.
For example, in web search, we can reference specific links from the search like this:
\ueturn0search3
2fc4441 to
9d8cb66
Compare
9225fbd to
2d7b88e
Compare
2d7b88e to
e21eab2
Compare
| import type { Node } from 'unist'; | ||
| import type { UIResourceNode } from './types'; | ||
|
|
||
| export const UI_RESOURCE_MARKER = '\ue205'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would not recommend using this as a marker, and instead use \ui or something similar, especially since the marker does not vary from \ue205.
As long as the plugin correctly falls back to \ui or the chosen marker, if it has no corresponding resource, to avoid edge cases where the AI writes the marker for purposes not related to UI resources.
Using "unicode-like" markers was a convention introduced by ChatGPT for their web search, which was a large inspiration for LC's UI/LLM interaction
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done! The marker is now \ui0 (or any other index) to show the resources. It works just as well!
e21eab2 to
fcf97ac
Compare
fcf97ac to
42ee75f
Compare
| }; | ||
| continue; | ||
| } else if (tool && mcpToolPattern.test(tool)) { | ||
| toolContextMap[tool] = `# MCP Tool \`${tool}\`: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I take issue with this being added to all MCP tools. These instructions can be part of the response that we format in formatToolContent i.e. packages/api/src/mcp/parsers.ts to ensure it's only applied once per "MCP UI" output.
Even better, would be to make use of serverInstructions for this to be on the MCP server to provide:
https://www.librechat.ai/docs/configuration/librechat_yaml/object_structure/mcp_servers#serverinstructions
Some MCP Servers have 50+ tools and are loaded individually at this level, and we would have repeated instructions for them all.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree that we don't need to add this to all MCP tools and I think formatToolContent is the right place for this.
These instructions aren't about how to use the tools on the server, but rather what to do with the tool response, i.e. how to render them in the client using markers. This is specific to LibreChat's implementation of MCP-UI, and other MCP clients could choose to use a different marker or syntax to render UI resources. This makes no difference for the MCP server, so I don't think the serverInstructions are the right place. What do you think?
| const [showRightArrow, setShowRightArrow] = useState(true); | ||
| const [isContainerHovered, setIsContainerHovered] = useState(false); | ||
| const scrollContainerRef = React.useRef<HTMLDivElement>(null); | ||
| const { submitMessage } = useSubmitMessage(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should not utilize this hook within the message tree as it can cause unnecessary re-renders across all messages and cause significant client latency.
See ~/components/Chat/Messages/Content/Parts/EditTextPart and client/src/components/Chat/Messages/Content/EditMessage.tsx for how this is prevented and submissions are still allowed.
| const localize = useLocalize(); | ||
| const { submitMessage } = useSubmitMessage(); | ||
| const { messageId } = useMessageContext(); | ||
| const { conversation } = useChatContext(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
useChatContext and useSubmitMessage should not be used here as they can cause unnecessary re-renders across all messages and cause significant client latency.
See how other adjacent components access the same things (EditTextPart and EditMessage for submissions)
|
|
||
| export function MCPUIResourceCarousel(props: MCPUIResourceCarouselProps) { | ||
| const { messageId } = useMessageContext(); | ||
| const { conversation } = useChatContext(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No use of useChatContext allowed here (will cause lag within the message tree)
|
I was able to resolve significant client rendering delays in this PR by addressing similar concerns based on my latest review: |
|
Excited for this to get in! |
|
Looking forward to this, saw the great presentation at the MCP London conference, which pushed me to add an MCP-UI to one of our current servers. Found it a little limited when it comes to UX right now. Once this is in place, it will enable make so many things possible! |
0d185fd to
50aa85a
Compare
|
Just addressed your comments by refactoring to a better approach that makes UI Resources available throughout the conversation, and also works when sharing conversations! Give this a spin and let me know what you think! |
|
Looks like I can't mark this PR as ready for review... @danny-avila would you mind giving me the required permissions? |
50aa85a to
4382fa3
Compare
ae89b95 to
b25c657
Compare
|
Hello!
return createUIResource({
uri: uiUri as `ui://${string}`,
content: {
type: 'externalUrl',
iframeUrl: `${baseUrl}${route}?waitForRenderData=true`,
},
encoding: 'text',
uiMetadata: {
'initial-render-data': props,
},
});
window.addEventListener('message', (event) => {
if (event.data.type ==='ui-lifecycle-iframe-render-data') {
console.log('mount the app')
} });
window.parent.postMessage({ type: 'ui-request-render-data', messageId }, '*');
|
|
@Anthony-Jhoiro Thanks for your feedback. We haven't yet considered this use case. We'll do it in the next iteration, after this current PR is reviewed and merged by @danny-avila. Thanks for your patience! |
|
@danny-avila do you think you may have time for this 2-months old PR this week? I know you must be very busy…! @plgodin do you think you can resolve the new merge conflicts to facilitate @danny-avila's final review here? |
|
Sorry for the wait on this. Would like to merge asap, and updating the branch will help! |
|
I just rebased and updated the branch! @danny-avila we should be good for a final review and merge! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR implements deeper MCP UI integration by enabling inline rendering of UI resources directly within chat messages, rather than only in tool response detail sections. The AI can now precisely control where UI elements appear in responses using special markers.
Key Changes:
- Introduced a plugin-based architecture that transforms UI resource markers (e.g.,
\ui{abc123}) into React components during markdown rendering - Changed UI resources from object format to array format for better iteration and carousel support
- Added cross-conversation resource resolution via stable resource IDs generated from content hashes
- Enhanced Share view to support MCP UI resources through a new
ShareMessagesProvider
Reviewed changes
Copilot reviewed 26 out of 27 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
packages/data-provider/src/schemas.ts |
Updated UIResource type to require resourceId and uri fields, removing optional type and data fields |
packages/api/src/mcp/utils.ts |
Fixed trailing whitespace in comment |
packages/api/src/mcp/types/index.ts |
Simplified FormattedContent type by removing metadata field from text content |
packages/api/src/mcp/parsers.ts |
Added resource ID generation, UI marker hints in tool responses, and instructions for LLM on how to use markers |
packages/api/src/mcp/__tests__/parsers.test.ts |
Updated tests to verify resource ID generation and UI marker formatting |
client/src/utils/index.ts |
Added handleUIAction utility to process user interactions with UI resources and submit messages to the LLM |
client/src/store/misc.ts |
Added conversationAttachmentsSelector to filter attachments by conversation for efficient resource lookup |
client/src/locales/en/translation.json |
Added error message translations for UI resource not found and rendering errors |
client/src/hooks/useResourcePermissions.ts |
Added default export for consistency with other hooks |
client/src/hooks/index.ts |
Changed import to use default export for useResourcePermissions |
client/src/hooks/Messages/useConversationUIResources.ts |
New hook that collects all UI resources in a conversation indexed by resource ID for cross-turn references |
client/src/components/Share/ShareView.tsx |
Wrapped MessagesView with ShareMessagesProvider to enable UI resources in shared conversations |
client/src/components/Share/ShareMessagesProvider.tsx |
New provider that supplies message context for share view, enabling existing hooks to work in read-only mode |
client/src/components/MCPUIResource/types.ts |
Defined TypeScript types for UI resource AST nodes used by the markdown plugin |
client/src/components/MCPUIResource/plugin.ts |
Implemented remark plugin that parses \ui{id} markers and transforms them into React component nodes |
client/src/components/MCPUIResource/index.ts |
Barrel export for MCP UI Resource components and utilities |
client/src/components/MCPUIResource/__tests__/plugin.test.ts |
Comprehensive tests for marker parsing, including single resources, carousels, and edge cases |
client/src/components/MCPUIResource/__tests__/MCPUIResourceCarousel.test.tsx |
Tests for carousel rendering with multiple resources and error handling |
client/src/components/MCPUIResource/__tests__/MCPUIResource.test.tsx |
Tests for single resource rendering, cross-conversation lookup, and error states |
client/src/components/MCPUIResource/MCPUIResourceCarousel.tsx |
Component that renders multiple UI resources in a carousel format |
client/src/components/MCPUIResource/MCPUIResource.tsx |
Component that renders individual UI resources inline with error handling |
client/src/components/Chat/Messages/Content/__tests__/UIResourceCarousel.test.tsx |
Updated tests to use handleUIAction instead of console logging |
client/src/components/Chat/Messages/Content/__tests__/Markdown.mcpui.test.tsx |
New integration test verifying UI resources render correctly within markdown |
client/src/components/Chat/Messages/Content/UIResourceCarousel.tsx |
Updated to use handleUIAction for consistent action handling across carousel and inline resources |
client/src/components/Chat/Messages/Content/Markdown.tsx |
Integrated mcpUIResourcePlugin and registered MCP UI components for markdown rendering |
client/src/Providers/MessagesViewContext.tsx |
Exported context and type to allow ShareMessagesProvider to implement the same interface |
api/server/controllers/agents/__tests__/callbacks.spec.js |
Updated tests to use array format for UI resources instead of object format with numeric keys |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.
| onUIAction={async (result) => handleUIAction(result, ask)} | ||
| htmlProps={{ | ||
| autoResizeIframe: { width: true, height: true }, | ||
| sandboxPermissions: 'allow-popups allow-popups-to-escape-sandbox', |
Copilot
AI
Dec 1, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Potential security concern: The sandboxPermissions value 'allow-popups allow-popups-to-escape-sandbox' allows content to escape the iframe sandbox, which could pose security risks when rendering untrusted UI resources from external MCP servers.
Consider whether these permissions are necessary for the use case. If popups must be allowed, document why this is safe in this context, or consider adding user warnings/confirmations for untrusted sources.
| sandboxPermissions: 'allow-popups allow-popups-to-escape-sandbox', | |
| // Only 'allow-popups' is permitted; 'allow-popups-to-escape-sandbox' is removed for security. | |
| sandboxPermissions: 'allow-popups', |
dustinhealy
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi all,
I've just finished my review and everything looks good to me. Once the recent Copilot comments have been addressed I will approve the changes and hand it off to Danny for a final sign off prior to merge.
The only friction I ran into during this review was related to AI responses incorrectly presenting the UI markers (erroneously wrapping markers in LaTex delimiters leading them to be converted into math nodes before they can be handled by the mcpUIResourcePlugin, responding with individual UIResources that are meant to be rendered within a UIResourceCarousel, etc.) causing them not to be rendered properly, but that is due to the relative strength of the models I was testing with and the context provided by the tool call responses from the MCP servers, not the implementation of the mcp-ui integration itself.
My main note which can likely be handled in a future PR is that in a shared conversation, currently there is no feedback mechanism for when a user performs a UIAction on an interactable UIResource. Of course, the core of this behavior is expected since the current strategy for handleUIAction is to send a message within the conversation / call a tool, and shared conversations in their present state are read-only; thus, such a solution is not viable. However, there still should be some indication to the user that interactions will not work as intended within that shared view beyond the console logs currently occurring when something like an "Add to Cart" button is pressed. Perhaps triggering a explanatory toast within handleUIAction for this scenario would be sufficient.
Thank you for your work on this PR - very excited to get this merged in. I believe this is a feature that many users will benefit from.
8a35d16 to
6aa696b
Compare
|
Update: Moving this PR to a new branch so that commits to address the GitHub Copilot comments can be pushed up (currently unable to because I lack permissions on the originating fork / the option to allow collaborators to push commits for this PR is not enabled) @samuelpath @plgodin I would appreciate if you could look over my changes in the new PR when you have a moment prior to this being merged where I attempted to address the Copilot comments and make sure everything looks right to you and if you could share what you think of the validity of the final unresolved Copilot comment and whether you think it requires further review / changes.
|
--------- Co-authored-by: Samuel Path <[email protected]> Co-authored-by: Pierre-Luc Godin <[email protected]>
8e1cfbd to
1e24fbf
Compare
…erencing
- Replace index-based resource markers with stable resource IDs
- Update plugin to parse \ui{resourceId} format instead of \ui0
- Refactor components to use useMessagesOperations instead of useSubmitMessage
- Add ShareMessagesProvider for UI resources in share view
- Add useConversationUIResources hook for cross-turn resource lookups
- Update parsers to generate resource IDs from content hashes
- Update all tests to use resource IDs instead of indices
- Add sandbox permissions for iframe popups
- Remove deprecated MCP tool context instructions
|
@danny-avila @dustinhealy I removed the extra permission from the |
|
Thanks for all your hard work and patience on this! I'm just going to approve while I clean up the We will check for what @dustinhealy if it's included and if not, we will open a new PR |
* 💻 feat: deeper MCP UI integration in the chat UI using plugins --------- Co-authored-by: Samuel Path <[email protected]> Co-authored-by: Pierre-Luc Godin <[email protected]> * 💻 refactor: Migrate MCP UI resources from index-based to ID-based referencing - Replace index-based resource markers with stable resource IDs - Update plugin to parse \ui{resourceId} format instead of \ui0 - Refactor components to use useMessagesOperations instead of useSubmitMessage - Add ShareMessagesProvider for UI resources in share view - Add useConversationUIResources hook for cross-turn resource lookups - Update parsers to generate resource IDs from content hashes - Update all tests to use resource IDs instead of indices - Add sandbox permissions for iframe popups - Remove deprecated MCP tool context instructions --------- Co-authored-by: Pierre-Luc Godin <[email protected]>
* 💻 feat: deeper MCP UI integration in the chat UI using plugins --------- Co-authored-by: Samuel Path <[email protected]> Co-authored-by: Pierre-Luc Godin <[email protected]> * 💻 refactor: Migrate MCP UI resources from index-based to ID-based referencing - Replace index-based resource markers with stable resource IDs - Update plugin to parse \ui{resourceId} format instead of \ui0 - Refactor components to use useMessagesOperations instead of useSubmitMessage - Add ShareMessagesProvider for UI resources in share view - Add useConversationUIResources hook for cross-turn resource lookups - Update parsers to generate resource IDs from content hashes - Update all tests to use resource IDs instead of indices - Add sandbox permissions for iframe popups - Remove deprecated MCP tool context instructions --------- Co-authored-by: Pierre-Luc Godin <[email protected]>
|
Really great to see this merged! Really awesome work @samuelpath , @plgodin ,@dustinhealy and the team 👏 |
|
This is incredible @samuelpath ,@dustinhealy, @plgodin! Thanks for making this happen. It's amazing to see a great project like LibreChat expand its MCP-UI support. As Liad mentioned, the new MCP Apps client will be released soon. You can already play with it in the mcp-ui/client alpha. |
* 💻 feat: deeper MCP UI integration in the chat UI using plugins --------- Co-authored-by: Samuel Path <[email protected]> Co-authored-by: Pierre-Luc Godin <[email protected]> * 💻 refactor: Migrate MCP UI resources from index-based to ID-based referencing - Replace index-based resource markers with stable resource IDs - Update plugin to parse \ui{resourceId} format instead of \ui0 - Refactor components to use useMessagesOperations instead of useSubmitMessage - Add ShareMessagesProvider for UI resources in share view - Add useConversationUIResources hook for cross-turn resource lookups - Update parsers to generate resource IDs from content hashes - Update all tests to use resource IDs instead of indices - Add sandbox permissions for iframe popups - Remove deprecated MCP tool context instructions --------- Co-authored-by: Pierre-Luc Godin <[email protected]>
Note
Most lines of code changed in this PR are tests. Advice for reviewer: first toggle out all test files to focus on the main changes in the logic, since there are not that many of them. And only then can you review the tests.
Summary
The past weeks, we shipped the following 3 PRs to integrate MCP UI with LibreChat:
These PRs made it such that whenever a tool response would send back a UI Resource, it would be displayed in the tool response detail section, as such:
This was only a first step.
The goal of the current PR is to allow the LLM to integrate UI Resource elements directly as part of the chat response. Here is what we would have now:
Or this:
Or when the chat is able to aggregate UI Resources from multiple tool responses:
You can now interact with UI Resources. When you click on an action for which the client should be aware, a message is posted to the client and can be picked up by the LLM.
For instance here I clicked on "Add to cart" in one of the product cards:
We see that LibreChat gets the message from the UI Resource, interprets it. For now, it's as if the UI Resource is impersonating the user. I will explain below how we intend to improve it in a follow-up PR.
The LLM then calls another tool call that flows from the user action (click on add to cart on a given product, product added to the cart, cart displayed):
Key changes made
1. Plugin-Based Architecture for MCP UI Resources
2. Data Structure Improvements
3. New Components Created
4. Enhanced Integration Points
5. Better User Experience
6. Comprehensive Testing
The overall improvement transforms MCP UI from a separate attachment display to a seamless, inline experience where AI models can precisely control where and how UI resources appear within their responses.
Samples of improvement follow-ups to come
Change Type
Please delete any irrelevant options.
Testing
Here are the MCP servers I use for my testing:
For the storefront servers, you can enter a prompt like: "looking for men sneakers", and then try to play with the UI elements displayed.
For the
mcp-aharvardserver, you can ask what the weather is like in some city.I've shown above what kind of result you should be seeing.
Test Configuration:
Checklist
Please delete any irrelevant options.