Add Support for image generation tool#960
Add Support for image generation tool#960TingtingZhou7 wants to merge 15 commits intolightseekorg:mainfrom
Conversation
TingtingZhou7
requested review from
CatherineSue,
key4ng and
slin1237
as code owners
github-actions
bot
added
grpc
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds first-class image-generation tool support across protocols, gateway routing, MCP config/transformer, and streaming: new Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant Gateway as ModelGateway
participant MCP
participant Transformer as ResponseTransformer
Client->>Gateway: Request with tool_choice "image_generation"
Gateway->>Gateway: recognize builtin tool -> route decision
Gateway->>MCP: Forward request to MCP image-generation server
MCP->>Transformer: Return ResponseFormat::ImageGenerationCall payload
Transformer->>Gateway: to_image_generation_call -> ImageGenerationCall output item
Gateway->>Client: Respond with `image_generation_call` output item
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
TingtingZhou7
changed the title
|
Hi @TingtingZhou7, the DCO sign-off check has failed. All commits must include a To fix existing commits: # Sign off the last N commits (replace N with the number of unsigned commits)
git rebase HEAD~N --signoff
git push --force-with-leaseTo sign off future commits automatically:
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds support for a new 'image_generation' tool across the protocol and gateway layers. I have noted that the new functionality lacks corresponding tests, which should be addressed to ensure reliability. Additionally, I recommend formatting the 'matches!' expression in 'model_gateway/src/routers/grpc/harmony/builder.rs' to span multiple lines for better consistency and readability.
| matches!( | ||
| self.tool_type.as_str(), | ||
| "web_search_preview" | "code_interpreter" | "container" | ||
| "web_search_preview" | "code_interpreter" | "image_generation" | "container" |
There was a problem hiding this comment.
For consistency with other multi-line matches! expressions in this pull request and to improve readability, consider formatting this by putting each pattern on its own line.
"web_search_preview"
| "code_interpreter"
| "image_generation"
| "container"
TingtingZhou7
force-pushed
the
tingzhou/add-img
branch
2 times, most recently
from
fb61255 to
3814457
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d6e92d7239
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.oca/custom_code_review_guidelines.txt:
- Around line 1-24: The "Sample guideline" content (the Java streams example /
"Use streams instead of a loop for better readability") appears unrelated to the
PR's image-generation scope; either remove this
.oca/custom_code_review_guidelines.txt change from the PR, or explicitly justify
its inclusion by updating the PR description and the file to state how these
review guidelines relate to the "Add Support for image generation tool" feature
(or move the guidelines to a separate documentation-only PR); ensure the file
text clearly references the image-generation context if kept.
- Line 22: Document that the example's use of Collection.toList() requires Java
16+ and either add a note stating the minimum Java version (Java 16) or change
the example to use Stream.collect(Collectors.toList()) for Java 8–15
compatibility; if you choose the latter, also note the mutability difference
(Collection.toList() returns an immutable List, while Collectors.toList()
returns a mutable list on older JDKs).
- Around line 9-22: The Java example in the template is inappropriate for this
Rust repo and contains API/type mismatches: replace the Java snippet
(references: List<Integer>, ArrayList, for (int number : numbers),
Arrays.stream(numbers), .toList()) with either a corrected Java variant that
preserves semantics (use numbers.stream() for collections and
collect(Collectors.toList()) if you need a mutable ArrayList) or—preferably—swap
the entire example for an equivalent Rust iterator sample using Vec and
iter().map(...).collect() so the example matches the repository language and
avoids the type and mutability problems.
In `@oracle-schema-config.local.yaml`:
- Line 1: The repo-wide config currently enables automatic DB DDL by setting
auto_migrate: true; change this key to auto_migrate: false in this committed
config so startup won't execute Oracle migrations by default, and instead enable
auto-migrate only in local/runtime overrides or explicit non-repo configs;
verify that the startup migration code paths (oracle.rs startup logic and
versioning.rs migration runner) will respect the config and that the default
safety check in schema.rs remains effective when the flag is false.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: d029e5cb-9b7c-444d-95d1-137b236a3f9f
📒 Files selected for processing (6)
.oca/custom_code_review_guidelines.txtcrates/protocols/src/responses.rsmodel_gateway/src/routers/grpc/common/responses/utils.rsmodel_gateway/src/routers/grpc/harmony/builder.rsmodel_gateway/src/routers/openai/responses/utils.rsoracle-schema-config.local.yaml
| # ```java | ||
| # // Calculate squares of numbers | ||
| # List<Integer> squares = new ArrayList<>(); | ||
| # for (int number : numbers) { | ||
| # squares.add(number * number); | ||
| # } | ||
| # ``` | ||
| # Recommendation: | ||
| # | ||
| # ```java | ||
| # // Calculate squares of numbers | ||
| # List<Integer> squares = Arrays.stream(numbers) | ||
| # .map(n -> n * n) // Map each number to its square | ||
| # .toList(); |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: Check Java version and stream usage patterns
# Check if Java version is documented anywhere
rg -i 'java.*(version|[0-9]{2})' --glob '*.md' --glob '*.txt' --glob 'pom.xml' --glob 'build.gradle*' -C2
# Check for existing .toList() usage (Java 16+ API)
rg '\.toList\(\)' --type=java -C2
# Check for Collectors.toList() usage (pre-Java 16 pattern)
rg 'Collectors\.toList\(\)' --type=java -C2Repository: lightseekorg/smg
Length of output: 357
🏁 Script executed:
#!/bin/bash
# Explore repository structure and primary language
echo "=== Repository Structure ==="
git ls-files | head -30
echo -e "\n=== File Extensions ==="
git ls-files | sed 's/.*\.//' | sort | uniq -c | sort -rn | head -20
echo -e "\n=== Read the file being reviewed ==="
if [ -f .oca/custom_code_review_guidelines.txt ]; then
cat -n .oca/custom_code_review_guidelines.txt
else
echo "File not found"
fiRepository: lightseekorg/smg
Length of output: 2505
Java example in template has type and API compatibility issues; consider replacing with Rust example matching repository language.
This is a template guideline file in a Rust repository. While the Java code example (lines 9-22) is intended as a sample pattern, it contains technical issues that would mislead developers:
- Type mismatch: The original loop
for (int number : numbers)works with anyIterable, but the recommendedArrays.stream(numbers)requires an array type. Ifnumbersis aList<Integer>, the recommendation won't compile. - Mutability difference:
ArrayList(original) is mutable;.toList()(recommended, Java 16+) returns an immutable list—a significant behavioral change. - Language scope: A Rust project's review guidelines should ideally feature Rust examples, not Java, to match contributor context.
Consider either fixing the Java example to use numbers.stream() if it's a collection, or replacing it with a Rust iterator example that's more relevant to the project.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.oca/custom_code_review_guidelines.txt around lines 9 - 22, The Java example
in the template is inappropriate for this Rust repo and contains API/type
mismatches: replace the Java snippet (references: List<Integer>, ArrayList, for
(int number : numbers), Arrays.stream(numbers), .toList()) with either a
corrected Java variant that preserves semantics (use numbers.stream() for
collections and collect(Collectors.toList()) if you need a mutable ArrayList)
or—preferably—swap the entire example for an equivalent Rust iterator sample
using Vec and iter().map(...).collect() so the example matches the repository
language and avoids the type and mutability problems.
| # // Calculate squares of numbers | ||
| # List<Integer> squares = Arrays.stream(numbers) | ||
| # .map(n -> n * n) // Map each number to its square | ||
| # .toList(); |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
What Java version introduced the Stream.toList() method and what are the differences between toList() and collect(Collectors.toList())?
💡 Result:
The Stream.toList method was introduced in Java 16. Key differences between Stream.toList and collect(Collectors.toList) are: - Mutability: toList returns an immutable List (throws UnsupportedOperationException on mutator methods). collect(Collectors.toList) returns a mutable List (typically ArrayList, modifiable). - Syntax: toList is more concise (no need for collect or Collectors import). collect requires stream.collect(Collectors.toList). - Null handling: Both allow null elements. - Type inference: collect(Collectors.toList) is more flexible with supertypes (e.g., Stream.collect(toList) to List); toList infers exact stream type, may need casts or explicit typing. - Performance: toList is slightly more optimized (direct array deposition, no Collector overhead). - Availability: toList since Java 16; Collectors.toList since Java 8. Note: In Java 21+, Collectors.toList was updated to return immutable lists matching toList behavior.
Citations:
- 1: https://bugs.openjdk.org/browse/JDK-8257870
- 2: https://stackoverflow.com/questions/65969919/differences-of-java-16s-stream-tolist-and-stream-collectcollectors-tolist
- 3: https://bugs.openjdk.org/browse/JDK-8256441
- 4: https://docs.oracle.com/en/java/javase/22/docs/api/java.base/java/util/stream/Stream.html
- 5: https://docs.oracle.com/en/java/javase/22/docs/api/java.base/java/util/stream/Collectors.html
Document Java version requirement for .toList().
The .toList() method was introduced in Java 16. If the project targets earlier Java versions, this example will not compile. For Java 8-15 compatibility, use .collect(Collectors.toList()) instead. Note: .toList() returns an immutable List, while Collectors.toList() returns a mutable one (though in Java 21+, Collectors.toList() also returns an immutable list).
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.oca/custom_code_review_guidelines.txt at line 22, Document that the
example's use of Collection.toList() requires Java 16+ and either add a note
stating the minimum Java version (Java 16) or change the example to use
Stream.collect(Collectors.toList()) for Java 8–15 compatibility; if you choose
the latter, also note the mutability difference (Collection.toList() returns an
immutable List, while Collectors.toList() returns a mutable list on older JDKs).
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
model_gateway/src/routers/grpc/common/responses/utils.rs (1)
25-29: 🧹 Nitpick | 🔵 TrivialUpdate docstring to include
image_generation.The docstring mentions
web_search_previewandcode_interpreteras builtin tool types but doesn't includeimage_generation, which is now also handled.📝 Suggested fix
/// Ensure MCP connection succeeds if MCP tools or builtin tools are declared /// -/// Checks if request declares MCP tools or builtin tool types (web_search_preview, -/// code_interpreter), and if so, validates that the MCP clients can be created +/// Checks if request declares MCP tools or builtin tool types (web_search_preview, +/// code_interpreter, image_generation), and if so, validates that the MCP clients can be created /// and connected.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@model_gateway/src/routers/grpc/common/responses/utils.rs` around lines 25 - 29, Update the docstring for the MCP connection check to also mention the builtin tool type image_generation; locate the comment/block above the function that "Ensure MCP connection succeeds if MCP tools or builtin tools are declared" (the docstring referencing builtin tool types web_search_preview and code_interpreter) and add image_generation to that list so the documentation matches current behavior.model_gateway/src/routers/openai/responses/utils.rs (1)
206-209: 🧹 Nitpick | 🔵 TrivialUpdate docstring to include
image_generation.The function docstring mentions "MCP tools", "web_search_preview", and "code_interpreter" but should also include
image_generationsince it's now handled.📝 Suggested fix
/// Convert a single ResponseTool back to its original JSON representation. /// -/// Handles MCP tools (with server metadata), web_search_preview, and code_interpreter. +/// Handles MCP tools (with server metadata), web_search_preview, code_interpreter, and image_generation. /// Returns None for function tools and other types that don't need restoration.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@model_gateway/src/routers/openai/responses/utils.rs` around lines 206 - 209, Update the function docstring that begins "Convert a single ResponseTool back to its original JSON representation." to also mention image_generation alongside MCP tools, web_search_preview, and code_interpreter; explicitly state that image_generation is handled and that the function still returns None for function tools and other types that don't need restoration so readers see image_generation is supported.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@crates/protocols/src/responses.rs`:
- Around line 39-41: Add the missing ImageGeneration tool into the MCP routing
surfaces: update the BuiltinToolType enum in crates/mcp/src/core/config.rs to
include ImageGeneration and add the corresponding ImageGenerationCall variant to
ResponseFormatConfig; then modify extract_builtin_types() in
model_gateway/src/routers/mcp_utils.rs so it extracts ImageGeneration (and
ensure FileSearch is also extracted there), and update the response_format()
implementation for BuiltinToolType to handle the new ImageGeneration variant
(return the proper ResponseFormatConfig::ImageGenerationCall). Use the enum and
function names BuiltinToolType, ResponseFormatConfig, extract_builtin_types, and
response_format to locate the spots to change.
---
Outside diff comments:
In `@model_gateway/src/routers/grpc/common/responses/utils.rs`:
- Around line 25-29: Update the docstring for the MCP connection check to also
mention the builtin tool type image_generation; locate the comment/block above
the function that "Ensure MCP connection succeeds if MCP tools or builtin tools
are declared" (the docstring referencing builtin tool types web_search_preview
and code_interpreter) and add image_generation to that list so the documentation
matches current behavior.
In `@model_gateway/src/routers/openai/responses/utils.rs`:
- Around line 206-209: Update the function docstring that begins "Convert a
single ResponseTool back to its original JSON representation." to also mention
image_generation alongside MCP tools, web_search_preview, and code_interpreter;
explicitly state that image_generation is handled and that the function still
returns None for function tools and other types that don't need restoration so
readers see image_generation is supported.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 5dfee3c3-70d3-451b-a54f-8bb74542dcde
📒 Files selected for processing (6)
.oca/custom_code_review_guidelines.txtcrates/protocols/src/responses.rsmodel_gateway/src/routers/grpc/common/responses/utils.rsmodel_gateway/src/routers/grpc/harmony/builder.rsmodel_gateway/src/routers/openai/responses/utils.rsoracle-schema-config.local.yaml
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ec689199c9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@crates/mcp/src/transform/transformer.rs`:
- Around line 396-420: Add additional unit tests for
ResponseTransformer::transform covering the edge cases for
ResponseFormat::ImageGenerationCall: one test that passes a direct JSON string
(to exercise the fallback path that treats the whole payload as a string), one
test that passes a non-string/non-object JSON value (to exercise the branch that
calls to_string() on the payload), and one test that asserts error payload
handling (if ResponseTransformer returns an error variant or Error payload
field) to verify the transformer produces the expected ImageGenerationCall
status and result; keep the same test structure as
test_image_generation_transform but vary the input payloads and expected id
("ig_req-..."), status, and result assertions.
- Around line 106-124: The to_image_generation_call function currently always
returns ResponseOutputItem::ImageGenerationCall with
ImageGenerationCallStatus::Completed and Some(result); update it to inspect the
MCP result for error/partial indicators (e.g., presence of "error" key, an
explicit "status" field like "failed"/"incomplete"/"generating", or non-2xx
error payload shapes) and map those to the appropriate ImageGenerationCallStatus
variants (InProgress/Generating/Incomplete/Failed) from the protocol, and set
result to None or include the error message only for Failed/Incomplete as
appropriate; locate and change logic in to_image_generation_call and the
construction of ResponseOutputItem::ImageGenerationCall (id, status, result) so
downstream consumers see correct status instead of always Completed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: ca785e4b-e333-4866-b2ad-c5634359f518
📒 Files selected for processing (4)
crates/mcp/src/core/config.rscrates/mcp/src/transform/transformer.rscrates/mcp/src/transform/types.rsmodel_gateway/src/routers/mcp_utils.rs
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e03b4b7c90
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
model_gateway/src/routers/grpc/common/responses/streaming.rs (2)
476-484: 🧹 Nitpick | 🔵 TrivialConsider defining a constant for the event type string.
The other variants use constants (e.g.,
WebSearchCallEvent::IN_PROGRESS). For consistency and maintainability, consider adding a similar constant likeImageGenerationCallEvent::IN_PROGRESSin the event_types module rather than using a hardcoded string.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@model_gateway/src/routers/grpc/common/responses/streaming.rs` around lines 476 - 484, The match arm for ResponseFormat::ImageGenerationCall is using a hardcoded string; instead add a constant like ImageGenerationCallEvent::IN_PROGRESS in the event_types module and use that constant here. Update the event_types module to define ImageGenerationCallEvent::IN_PROGRESS (matching the naming/pattern of WebSearchCallEvent::IN_PROGRESS), then change the match to return ImageGenerationCallEvent::IN_PROGRESS so all ResponseFormat variants use the same constant pattern and maintain consistency with emit_tool_event.
125-158:⚠️ Potential issue | 🔴 CriticalField mismatch:
ImageGenerationCallusesidandresult, notcall_idandoutput.Per
crates/protocols/src/responses.rs(Context snippet 2),ResponseOutputItem::ImageGenerationCallhas fieldsid,status, andresult. However, this function:
- Looks up
call_id(line 141) which doesn't exist onImageGenerationCall- Sets
output(line 147) instead ofresultAs a result,
ImageGenerationCallitems will pass theis_tool_callcheck but never match, silently failing to update.🐛 Proposed fix to handle ImageGenerationCall field differences
let is_tool_call = matches!( item_type, Some("mcp_call") | Some("web_search_call") | Some("code_interpreter_call") | Some("file_search_call") | Some("image_generation_call") ); - if is_tool_call - && item_data.get("call_id").and_then(|c| c.as_str()) - == Some(&tool_result.call_id) - { + // ImageGenerationCall uses `id` instead of `call_id` + let call_id_field = if item_type == Some("image_generation_call") { + "id" + } else { + "call_id" + }; + if is_tool_call + && item_data.get(call_id_field).and_then(|c| c.as_str()) + == Some(&tool_result.call_id) + { // Add output field let output_str = serde_json::to_string(&tool_result.output) .unwrap_or_else(|_| "{}".to_string()); - item_data["output"] = json!(output_str); + // ImageGenerationCall uses `result` instead of `output` + let output_field = if item_type == Some("image_generation_call") { + "result" + } else { + "output" + }; + item_data[output_field] = json!(output_str);🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@model_gateway/src/routers/grpc/common/responses/streaming.rs` around lines 125 - 158, The update_mcp_call_outputs function currently matches tool call items by looking for "call_id" and writes an "output" field, which fails for ImageGenerationCall items that use "id" and "result"; modify update_mcp_call_outputs (and the loop that checks item_type == "image_generation_call") to treat image_generation_call specially: compare item_data.get("id").and_then(|v| v.as_str()) to tool_result.call_id (instead of "call_id") and write the tool_result.output into item_data["result"] (preserving the same serialization strategy you use elsewhere), while keeping the existing behavior for other tool types that use "call_id" and "output"; ensure you still set item_data["status"] = "failed" when tool_result.is_error.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Outside diff comments:
In `@model_gateway/src/routers/grpc/common/responses/streaming.rs`:
- Around line 476-484: The match arm for ResponseFormat::ImageGenerationCall is
using a hardcoded string; instead add a constant like
ImageGenerationCallEvent::IN_PROGRESS in the event_types module and use that
constant here. Update the event_types module to define
ImageGenerationCallEvent::IN_PROGRESS (matching the naming/pattern of
WebSearchCallEvent::IN_PROGRESS), then change the match to return
ImageGenerationCallEvent::IN_PROGRESS so all ResponseFormat variants use the
same constant pattern and maintain consistency with emit_tool_event.
- Around line 125-158: The update_mcp_call_outputs function currently matches
tool call items by looking for "call_id" and writes an "output" field, which
fails for ImageGenerationCall items that use "id" and "result"; modify
update_mcp_call_outputs (and the loop that checks item_type ==
"image_generation_call") to treat image_generation_call specially: compare
item_data.get("id").and_then(|v| v.as_str()) to tool_result.call_id (instead of
"call_id") and write the tool_result.output into item_data["result"] (preserving
the same serialization strategy you use elsewhere), while keeping the existing
behavior for other tool types that use "call_id" and "output"; ensure you still
set item_data["status"] = "failed" when tool_result.is_error.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 4aa4c854-ad76-461e-acc3-b967a6e7aed9
📒 Files selected for processing (2)
model_gateway/src/routers/grpc/common/responses/streaming.rsmodel_gateway/src/routers/openai/mcp/tool_loop.rs
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 648d3a748f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
TingtingZhou7
changed the title
TingtingZhou7
force-pushed
the
tingzhou/add-img
branch
from
4630856 to
bb24298
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6ca57cc806
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| ItemType::WEB_SEARCH_CALL => WebSearchCallEvent::COMPLETED, | ||
| ItemType::CODE_INTERPRETER_CALL => CodeInterpreterCallEvent::COMPLETED, | ||
| ItemType::FILE_SEARCH_CALL => FileSearchCallEvent::COMPLETED, | ||
| "image_generation_call" => ImageGenerationCallEvent::COMPLETED, |
There was a problem hiding this comment.
This new arm emits response.image_generation_call.completed, but in the same OpenAI streaming flow transform_streaming_event still maps non-web-search builtins to mcp_call with mcp_ IDs (model_gateway/src/routers/openai/responses/streaming.rs:146-149). For image-generation requests, clients will see response.output_item.added as mcp_call and then completion/done as image_generation_call for the same output index, which changes type/ID mid-stream and breaks consumers that correlate events by item identity.
Useful? React with 👍 / 👎.
| if item_type == Some("image_generation_call") { | ||
| let result_str = tool_result | ||
| .output |
There was a problem hiding this comment.
The new image-generation update path only extracts result from tool_result.output and never reads a returned status; afterwards status is only changed when tool_result.is_error is true. In the Harmony Responses streaming path (which calls this updater before response.completed), payloads such as {"status":"generating"} or {"status":"incomplete"} will be surfaced as completed calls, so clients lose the newly added status semantics and can treat unfinished generations as final.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2d4662056a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| | Some("web_search_call") | ||
| | Some("code_interpreter_call") | ||
| | Some("file_search_call") | ||
| | Some("image_generation_call") |
There was a problem hiding this comment.
By including image_generation_call in this shared updater, image tool items now pass through the generic error-status logic below, which writes "failed" when tool_result.is_error is true. However ResponseOutputItem::ImageGenerationCall uses ImageGenerationCallStatus (in_progress|completed|incomplete) and does not accept failed, so failed image items become schema-invalid; in finalize() they fail serde_json::from_value(...).ok() and get dropped from the persisted/final response. This occurs in the gRPC Harmony streaming path whenever an MCP image tool returns an error.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 79070250c7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4a8cd870d8
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
3 similar comments
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
TingtingZhou7
force-pushed
the
tingzhou/add-img
branch
from
c867045 to
466a5f6
Compare
github-actions
bot
added
mcp
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 466a5f6511
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if !requested_allowed_tools.is_empty() { | ||
| session_tools.retain(|tool| { | ||
| let Some(name) = tool.get("name").and_then(|v| v.as_str()) else { | ||
| return false; | ||
| }; | ||
| if requested_allowed_tools.contains(name) { | ||
| return true; | ||
| } | ||
| name.rsplit_once("__") | ||
| .is_some_and(|(_, suffix)| requested_allowed_tools.contains(suffix)) | ||
| }); |
There was a problem hiding this comment.
This global requested_allowed_tools filter applies one merged name set to all session tools, so a request that includes allowed_tools for one MCP server can silently remove unrelated tools (including builtin-routed tools like image_generation or tools from another server). McpToolSession::new already enforces per-server allowlists, so this extra cross-server retain step introduces regressions in mixed-tool requests where valid tools disappear before model execution.
Useful? React with 👍 / 👎.
| if !mcp_servers | ||
| .iter() | ||
| .any(|b| b.server_key == server_name || b.label == server_name) | ||
| { |
There was a problem hiding this comment.
Including b.label == server_name in this dedupe check lets a request-scoped MCP tool suppress the configured builtin server just by choosing the same label, even when it points to a different URL. In that case the static builtin binding is skipped, so builtin tool exposure/routing can break or target the wrong server for that request. Matching should use stable server identity (server key/URL), not user-provided labels.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8114c8ee8a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
crates/mcp/src/core/session.rs
Outdated
| let image_builtin_name = orchestrator | ||
| .find_builtin_server(BuiltinToolType::ImageGeneration) | ||
| .map(|(server_name, _, _)| server_name); |
There was a problem hiding this comment.
McpToolSession::new now computes visibility from only BuiltinToolType::ImageGeneration, so builtin servers for web_search_preview/code_interpreter are no longer removed from mcp_servers(). That regresses the non-builtin visibility contract and causes downstream list-tools/output paths that iterate session.mcp_servers() to surface builtin routing servers as regular MCP servers, producing duplicated or misleading tool exposure in mixed-tool requests.
Useful? React with 👍 / 👎.
| let mut headers = default_headers.clone(); | ||
| for (k, v) in mcp.headers.clone().unwrap_or_default() { | ||
| headers.insert(k, v); |
There was a problem hiding this comment.
This now merges default_headers into every user-provided ResponseTool::Mcp connection. The OpenAI response handlers populate default_headers from opc-compartment-id, so requests that target third-party server_url values will leak compartment metadata to external MCP services through connect_mcp_servers; this header propagation should be scoped to builtin-routed servers instead of all dynamic MCP URLs.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 15c4a9e894
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| || obj.contains_key("background") | ||
| || obj.contains_key("output_format") | ||
| || obj.contains_key("quality") | ||
| || obj.contains_key("size") | ||
| || obj.contains_key("action") |
There was a problem hiding this comment.
In ResponseFormat::Passthrough, this heuristic treats any payload containing generic keys like action, size, or quality as an image-generation result, so non-image MCP tools can be mis-typed as image_generation_call and lose their original mcp_call structure. A normal passthrough tool returning { "action": "..." } will now be transformed incorrectly; conversion should require an explicit discriminator (for example type == "image_generation_call") instead of these broad key checks.
Useful? React with 👍 / 👎.
| let revised_prompt = arguments.get("revised_prompt").cloned().unwrap_or(Value::Null); | ||
|
|
||
| let background = get_non_empty_string_option(image_tool_options, "background") |
There was a problem hiding this comment.
This path only reads revised_prompt and then replaces the entire argument object, which drops all original tool arguments for any call routed as ImageGenerationCall. That breaks configurations where the image tool schema uses different fields (for example prompt) or additional required parameters, because they are silently removed before execute_tool. Merging defaults into the parsed arguments (instead of rebuilding from revised_prompt) avoids null/invalid calls.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5f8888988b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| ItemType::WEB_SEARCH_CALL => WebSearchCallEvent::COMPLETED, | ||
| ItemType::CODE_INTERPRETER_CALL => CodeInterpreterCallEvent::COMPLETED, | ||
| ItemType::FILE_SEARCH_CALL => FileSearchCallEvent::COMPLETED, | ||
| ItemType::IMAGE_GENERATION_CALL => ImageGenerationCallEvent::COMPLETED, | ||
| _ => McpEvent::CALL_COMPLETED, // Default to mcp_call for mcp_call and unknown types |
There was a problem hiding this comment.
send_tool_call_completion_events always emits response.image_generation_call.completed for image items, but tool_output.to_response_item() can legitimately produce non-terminal statuses (in_progress/generating). In streaming requests where an MCP image backend returns those statuses, clients receive a completion event and response.output_item.done even though generation is still ongoing, which can cause consumers to stop waiting for the final image.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c0da9898d9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| name.rsplit_once("__") | ||
| .is_some_and(|(_, suffix)| requested_allowed_tools.contains(suffix)) |
There was a problem hiding this comment.
This retain check only strips a "__" suffix, but McpToolSession exposes duplicate tool names with the mcp_<server>_<tool> pattern (see crates/mcp/src/core/session.rs), so the allowlist no longer matches those tools in the OpenAI Responses interception path. When any request has allowed_tools and there are duplicate tool names across servers, even explicitly allowed tools are filtered out before model execution, causing tool calls to disappear unexpectedly.
Useful? React with 👍 / 👎.
| output: &Value, | ||
| is_error: bool, | ||
| ) -> String { | ||
| if tool_name == "generate_image" { |
There was a problem hiding this comment.
Image output compaction is gated on tool_name == "generate_image", but in this codebase the invoked tool name can be aliased (for duplicate names) or configured with a different builtin tool name. In those cases image-generation results are treated as generic JSON and the full base64 payload is written into tool context, which can bloat follow-up prompts and trigger context-length failures in multi-turn tool loops.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 02767ec0d0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| let already_bound = mcp_servers.iter().any(|b| { | ||
| b.server_key == server_name | ||
| || builtin_server_url | ||
| .as_ref() | ||
| .is_some_and(|url| b.server_key == *url) | ||
| }); |
There was a problem hiding this comment.
Do not treat a dynamic binding with the same URL as equivalent to a configured builtin binding here. This check suppresses adding the static builtin server whenever any dynamic MCP input shares its URL (including the per-request builtin connection path), but dynamic bindings are created without builtin tool metadata (tools/builtin_type), so their discovered tools keep default passthrough formatting. In that case builtin calls can lose their configured response format and stream/event shape, which breaks image/web/code builtin behavior whenever URL-based dedupe is hit.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cbad86d6dd
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| "parameters".to_string(), | ||
| json!({ | ||
| "type": "object", | ||
| "properties": { | ||
| "revised_prompt": { "type": "string" } |
There was a problem hiding this comment.
This block overwrites the MCP-discovered parameters schema for generate_image with a hardcoded schema that only requires revised_prompt. If the underlying MCP server expects other required fields (for example prompt, or provider-specific required options), the model will no longer be guided to produce valid arguments and tool execution can fail at runtime. The safer approach is to augment or merge with the server-provided schema rather than replacing it wholesale.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
smg/model_gateway/src/routers/grpc/regular/responses/streaming.rs
Lines 794 to 797 in f939768
Avoid sending output_item.done for non-terminal image statuses
The code correctly suppresses response.image_generation_call.completed when the tool reports in_progress/generating, but it still always emits response.output_item.done and marks the item complete. When an MCP image tool returns a non-terminal status, clients still receive a terminal done signal and may stop waiting for subsequent progress/final image data.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aa13923fc6
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // image_generation_call stores image bytes in `result` instead of `output`. | ||
| if item_type.as_deref() == Some(ItemType::IMAGE_GENERATION_CALL) { | ||
| let result_str = tool_result | ||
| .output | ||
| .as_object() |
There was a problem hiding this comment.
In the Harmony Responses streaming path, update_mcp_call_outputs only copies result for image_generation_call items and drops other payload fields like revised_prompt, background, output_format, quality, size, and action. Because finalize() rebuilds the final response from item_data, any metadata returned by the MCP image tool is silently lost in both streamed output_item.done state and persisted final output, even when the tool returned it.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 776205cb14
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| "name": tool_output.tool_name, | ||
| "status": status_str, | ||
| "arguments": tool_output.arguments_str, | ||
| "result": result_str |
There was a problem hiding this comment.
This branch constructs image_generation_call items with only result, which drops structured fields (for example revised_prompt, size, quality, background, output_format, action) that tool_output.to_response_item() already computed. Because emit_output_item_done stores this object as the canonical output item used in response.completed, regular gRPC streaming responses lose image metadata even when the MCP tool returned it.
Useful? React with 👍 / 👎.
| .and_then(|v| v.as_str()) | ||
| .map(String::from) | ||
| .or_else(|| tool_output.output.as_str().map(String::from)) | ||
| .unwrap_or_else(|| tool_output.output.to_string()); |
There was a problem hiding this comment.
When the tool output has no result field, this fallback serializes the entire JSON payload into result. For non-terminal image statuses like in_progress/generating, clients can receive result values such as a stringified status object instead of image data, which breaks consumers that treat result as the binary/base64 payload. result should stay unset unless an actual result value is returned.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b5187667b5
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| *output = Some(compact_tool_output_for_model_context( | ||
| false, | ||
| &tool_result.output, |
There was a problem hiding this comment.
This call hardcodes is_image_generation to false, so Harmony Responses always serializes full MCP output into the next-turn function_call_output context. When the tool is image generation, that output is usually large base64 content, which can blow up prompt size and cause follow-up iterations to fail or be heavily truncated instead of using the intended compact summary path. The same commit added image-output compaction elsewhere, so this branch should also detect image tool results before calling compact_tool_output_for_model_context.
Useful? React with 👍 / 👎.
|
Hi @TingtingZhou7, this PR has merge conflicts that must be resolved before it can be merged. Please rebase your branch: git fetch origin main
git rebase origin/main
# resolve any conflicts, then:
git push --force-with-lease |
mergify
bot
added
the
needs-rebase
1 participant
Description
Support image generation tool mcp server from smg for openSource customers.
Problem
No image generation tool support today via mcp server.
The reponse api format will follow openAI format: https://developers.openai.com/api/docs/guides/image-generation
Solution
Add support.
Changes
Two layer of ImageGenerationTool is supported in this PR:
Test Plan
Test Case 1: Call to Remote MCP Server to generate image
Step1: I set up the OCI Image Tool Generation tool MCP Server locally at port 8080. And run the cargo command as below with --mcp-config-path /Users/tingzhou/workspace/genAI/smg/mcp.local.yaml added. This will allow smg to read the mcp server information dynamically:
which include the tool and mcp server information. Here is the example of the mcp.local.yaml file:
Test with streaming disabled
Request
Success log from local mcp server:
Image attached:
Respons json:
response-new1.json
Test with streaming enabled
And also verified image events are generated properly. Response can be found here:
response.txt
Test Case 2
Direct call to OpenAI to generate image
Step1: I run this command to start the smg locally without pass the mcp server configuration:
Step1: Run image generation tool end-to-end
API call:
And here is the output:
openai-response.txt
Checklist
cargo +nightly fmtpassescargo clippy --all-targets --all-features -- -D warningspassesSummary by CodeRabbit
New Features
Tests