API bug: functionResponse with $ref + displayName returns 400 "does not match to a display_name" (gemini-3.5-flash, gemini-3.6-flash)

[turicas]

This is a product/API issue, not a client library issue. I'm filing here because there is no public Issue Tracker component for the Gemini API (AI Studio / generativelanguage.googleapis.com). The reproducer uses stdlib only (no SDK).

Environment details

• Programming language: Python 3.12 (stdlib only, no SDK)
• OS: Debian Linux
• Endpoint: https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent
• Models tested: gemini-3.5-flash-preview-05-20, gemini-3.6-flash-preview-05-20

Problem

The Gemini generateContent API rejects the documented pattern for returning media in a functionResponse using $ref and displayName. The error is:

The referenced name `speed-limit.jpg` in function_response.response does not match
to a display_name in the function_response.parts.

This affects both gemini-3.5-flash and gemini-3.6-flash. The alternative layout (inline image in parts without $ref) works correctly.

Affected models

Tested 2026-07-29:

Model	$ref + displayName (documented)	Inline image without $ref	functionResponse + sibling image
gemini-3.5-flash	HTTP 400	HTTP 200, correct	HTTP 200, correct
gemini-3.6-flash	HTTP 400	HTTP 200, correct	HTTP 200, correct

Steps to reproduce

export GEMINI_API_KEY=...
python3 gemini_tool_error.py

(see the script code below)

Minimal failing payload

This follows the pattern described in the function calling documentation for returning media alongside function results:

{
  "contents": [
    {"role": "user", "parts": [{"text": "Call get_test_image and tell me what you see."}]},
    {"role": "model", "parts": [{"functionCall": {"name": "get_test_image", "args": {}}}]},
    {
      "role": "user",
      "parts": [
        {
          "functionResponse": {
            "name": "get_test_image",
            "response": {"image_ref": {"$ref": "speed-limit.jpg"}},
            "parts": [
              {
                "inlineData": {
                  "displayName": "speed-limit.jpg",
                  "mimeType": "image/jpeg",
                  "data": "<BASE64>"
                }
              }
            ]
          }
        }
      ]
    }
  ],
  "tools": [{"functionDeclarations": [{"name": "get_test_image", "description": "Returns an image.", "parameters": {"type": "object", "properties": {}, "required": []}}]}]
}

Response: HTTP 400

{
  "error": {
    "code": 400,
    "message": "The referenced name `speed-limit.jpg` in function_response.response does not match to a display_name in the function_response.parts.",
    "status": "INVALID_ARGUMENT"
  }
}

Working alternative (inline image without $ref)

Replacing the $ref response with a plain result and keeping the image in parts works:

{
  "functionResponse": {
    "name": "get_test_image",
    "response": {"result": "The requested image is attached."},
    "parts": [
      {
        "inlineData": {
          "displayName": "speed-limit.jpg",
          "mimeType": "image/jpeg",
          "data": "<BASE64>"
        }
      }
    ]
  }
}

Response: HTTP 200, correct answer.

Console output

=== gemini-3.5-flash ===
  google-native/gemini-3.5-flash: initial HTTP 200
    documented_ref_and_displayName         HTTP 400: The referenced name `speed-limit.jpg` in function_response.response does not match to a display_name in the function_response.parts.
    nested_image_without_ref               HTTP 200: 60
    function_response_plus_sibling_image   HTTP 200: 0 60
    documented_without_signatures          HTTP 400: The referenced name `speed-limit.jpg` in function_response.response does not match to a display_name in the function_response.parts.
 
=== gemini-3.6-flash ===
  google-native/gemini-3.6-flash: initial HTTP 200
    documented_ref_and_displayName         HTTP 400: The referenced name `speed-limit.jpg` in function_response.response does not match to a display_name in the function_response.parts.
    nested_image_without_ref               HTTP 200: 60
    function_response_plus_sibling_image   HTTP 200: 60
    documented_without_signatures          HTTP 400: The referenced name `speed-limit.jpg` in function_response.response does not match to a display_name in the function_response.parts.

gemini_tool_error.py

"""Minimal multimodal tool-result reproducer for Gemini 3.5/3.6.

Tests the same image-returning tool workflow through:
  1. OpenRouter Chat Completions
  2. Google's OpenAI-compatible Chat Completions
  3. Google's native generateContent API

stdlib only. Required environment variables: OPENROUTER_API_KEY and GEMINI_API_KEY
"""

import base64
import copy
import json
import os
import sys
import urllib.error
import urllib.request
from pathlib import Path
from typing import Any

IMAGE_URL = "https://commons.wikimedia.org/wiki/Special:FilePath/Canada%20Speed%20Limit%2060%20sign.jpg?width=512"
IMAGE_FILE = Path("image-001.jpg")
RESULT_FILE = Path("gemini-tool-image-repro-results.json")
MODELS = ("gemini-3.5-flash", "gemini-3.6-flash")
PROMPT = (
    "Call get_test_image exactly once. Do not answer before receiving its result. "
    "Then answer with only the largest speed-limit number visible in the image."
)
USER_AGENT = "python/urllib"
TOOL_OPENAI = {
    "type": "function",
    "function": {
        "name": "get_test_image",
        "description": "Returns the test image requested by the user.",
        "parameters": {"type": "object", "properties": {}, "required": []},
    },
}
TOOL_GEMINI = {
    "functionDeclarations": [
        {
            "name": "get_test_image",
            "description": "Returns the test image requested by the user.",
            "parameters": {"type": "object", "properties": {}, "required": []},
        }
    ]
}
JSON = dict[str, Any]


def download_image() -> tuple[str, str]:
    if not IMAGE_FILE.exists():
        print(f"Downloading {IMAGE_FILE}...")
        request = urllib.request.Request(IMAGE_URL, headers={"User-Agent": USER_AGENT})
        with urllib.request.urlopen(request, timeout=60) as response:
            IMAGE_FILE.write_bytes(response.read())
    return "image/jpeg", base64.b64encode(IMAGE_FILE.read_bytes()).decode("ascii")


def post(url: str, headers: dict[str, str], payload: JSON) -> tuple[int, Any]:
    request = urllib.request.Request(
        url,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json", "User-Agent": USER_AGENT, **headers},
        method="POST",
    )
    try:
        with urllib.request.urlopen(request, timeout=120) as response:
            raw = response.read().decode("utf-8", errors="replace")
            return response.status, json.loads(raw)
    except urllib.error.HTTPError as error:
        raw = error.read().decode("utf-8", errors="replace")
        try:
            body: Any = json.loads(raw)
        except json.JSONDecodeError:
            body = raw
        return error.code, body
    except (urllib.error.URLError, TimeoutError, OSError) as error:
        return 0, {"network_error": f"{type(error).__name__}: {error}"}


def remove_signatures(value: Any) -> Any:
    """Diagnostic only: recursively strips provider reasoning/signature metadata."""
    if isinstance(value, list):
        return [remove_signatures(item) for item in value]
    if not isinstance(value, dict):
        return value
    blocked = {
        "reasoning",
        "reasoning_content",
        "reasoning_details",
        "thoughtSignature",
        "thought_signature",
        "signature",
    }
    return {key: remove_signatures(item) for key, item in value.items() if key not in blocked}


def response_text(protocol: str, body: Any) -> str:
    try:
        if protocol == "openai-chat":
            return str(body["choices"][0]["message"].get("content") or "")
        parts = body["candidates"][0]["content"]["parts"]
        return "\n".join(str(part.get("text", "")) for part in parts if isinstance(part, dict))
    except (KeyError, IndexError, TypeError):
        return ""


def error_text(body: Any) -> str:
    if isinstance(body, dict) and "error" in body:
        error = body["error"]
        if isinstance(error, dict):
            return str(error.get("message") or error)
        return str(error)
    if isinstance(body, dict) and "network_error" in body:
        return str(body["network_error"])
    return ""


def show(label: str, status: int, protocol: str, body: Any) -> None:
    text = response_text(protocol, body).strip().replace("\n", " ")
    detail = text or error_text(body)
    if len(detail) > 180:
        detail = detail[:177] + "..."
    print(f"    {label:<38} HTTP {status}: {detail}")


def openai_compatible(
    endpoint_name: str,
    url: str,
    key: str,
    model: str,
    image_uri: str,
) -> list[JSON]:
    model_id = f"google/{model}" if endpoint_name == "openrouter" else model
    initial: JSON = {
        "model": model_id,
        "messages": [{"role": "user", "content": PROMPT}],
        "tools": [TOOL_OPENAI],
        "tool_choice": {"type": "function", "function": {"name": "get_test_image"}},
        "stream": False,
    }
    if endpoint_name == "openrouter":
        initial["reasoning"] = {"effort": "low"}

    status, first = post(url, {"Authorization": f"Bearer {key}"}, initial)
    print(f"  {endpoint_name}/{model}: initial HTTP {status}")
    if status != 200:
        show("initial", status, "openai-chat", first)
        return [{"endpoint": endpoint_name, "model": model, "case": "initial", "status": status, "response": first}]

    try:
        assistant = first["choices"][0]["message"]
        call_id = assistant["tool_calls"][0]["id"]
    except (KeyError, IndexError, TypeError) as error:
        return [
            {
                "endpoint": endpoint_name,
                "model": model,
                "case": "initial_parse",
                "status": status,
                "response": first,
                "error": str(error),
            }
        ]

    tool_text = {
        "role": "tool",
        "tool_call_id": call_id,
        "content": "The tool returned the requested image.",
    }
    image_part = {"type": "image_url", "image_url": {"url": image_uri}}
    cases: list[tuple[str, JSON, JSON]] = [
        (
            "tool_then_user_image_only",
            assistant,
            {"messages": [tool_text, {"role": "user", "content": [image_part]}]},
        ),
        (
            "tool_then_user_text_plus_image",
            assistant,
            {
                "messages": [
                    tool_text,
                    {
                        "role": "user",
                        "content": [
                            {"type": "text", "text": "This image is the output of get_test_image."},
                            image_part,
                        ],
                    },
                ]
            },
        ),
        (
            "image_inside_tool_message",
            assistant,
            {
                "messages": [
                    {
                        "role": "tool",
                        "tool_call_id": call_id,
                        "content": [
                            {"type": "text", "text": "The tool returned this image."},
                            image_part,
                        ],
                    }
                ]
            },
        ),
        (
            "image_only_without_signatures",
            remove_signatures(assistant),
            {"messages": [tool_text, {"role": "user", "content": [image_part]}]},
        ),
    ]

    results: list[JSON] = []
    for case_name, assistant_message, suffix in cases:
        body: JSON = {
            "model": model_id,
            "messages": [
                {"role": "user", "content": PROMPT},
                copy.deepcopy(assistant_message),
                *copy.deepcopy(suffix["messages"]),
            ],
            "tools": [TOOL_OPENAI],
            "stream": False,
        }
        if endpoint_name == "openrouter":
            body["reasoning"] = {"effort": "low"}
        case_status, case_response = post(url, {"Authorization": f"Bearer {key}"}, body)
        show(case_name, case_status, "openai-chat", case_response)
        results.append(
            {
                "endpoint": endpoint_name,
                "model": model,
                "case": case_name,
                "status": case_status,
                "text": response_text("openai-chat", case_response),
                "response": case_response,
            }
        )
    return results


def native_gemini(key: str, model: str, mime: str, encoded: str) -> list[JSON]:
    url = f"https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent"
    initial: JSON = {
        "contents": [{"role": "user", "parts": [{"text": PROMPT}]}],
        "tools": [TOOL_GEMINI],
        "toolConfig": {
            "functionCallingConfig": {
                "mode": "ANY",
                "allowedFunctionNames": ["get_test_image"],
            }
        },
    }
    headers = {"x-goog-api-key": key}
    status, first = post(url, headers, initial)
    print(f"  google-native/{model}: initial HTTP {status}")
    if status != 200:
        show("initial", status, "gemini-native", first)
        return [{"endpoint": "google-native", "model": model, "case": "initial", "status": status, "response": first}]

    try:
        model_content = first["candidates"][0]["content"]
        function_call = next(part["functionCall"] for part in model_content["parts"] if "functionCall" in part)
        call_id = str(function_call.get("id") or "")
        call_name = str(function_call["name"])
    except (KeyError, IndexError, StopIteration, TypeError) as error:
        return [
            {
                "endpoint": "google-native",
                "model": model,
                "case": "initial_parse",
                "status": status,
                "response": first,
                "error": str(error),
            }
        ]

    def function_response(response: JSON, parts: list[JSON] | None = None) -> JSON:
        value: JSON = {"name": call_name, "response": response}
        if call_id:
            value["id"] = call_id
        if parts is not None:
            value["parts"] = parts
        return {"functionResponse": value}

    blob_with_name = {
        "inlineData": {
            "displayName": "speed-limit.jpg",
            "mimeType": mime,
            "data": encoded,
        }
    }
    plain_blob = {"inlineData": {"mimeType": mime, "data": encoded}}
    documented = function_response(
        {"image_ref": {"$ref": "speed-limit.jpg"}},
        [blob_with_name],
    )
    nested_no_ref = function_response(
        {"result": "The requested image is attached."},
        [blob_with_name],
    )
    text_result = function_response({"result": "The requested image follows."})

    cases: list[tuple[str, JSON, list[JSON]]] = [
        (
            "documented_ref_and_displayName",
            model_content,
            [{"role": "user", "parts": [documented]}],
        ),
        (
            "nested_image_without_ref",
            model_content,
            [{"role": "user", "parts": [nested_no_ref]}],
        ),
        (
            "function_response_plus_sibling_image",
            model_content,
            [{"role": "user", "parts": [text_result, plain_blob]}],
        ),
        (
            "documented_without_signatures",
            remove_signatures(model_content),
            [{"role": "user", "parts": [documented]}],
        ),
    ]

    results: list[JSON] = []
    for case_name, returned_model_content, suffix in cases:
        body: JSON = {
            "contents": [
                {"role": "user", "parts": [{"text": PROMPT}]},
                copy.deepcopy(returned_model_content),
                *copy.deepcopy(suffix),
            ],
            "tools": [TOOL_GEMINI],
        }
        case_status, case_response = post(url, headers, body)
        show(case_name, case_status, "gemini-native", case_response)
        results.append(
            {
                "endpoint": "google-native",
                "model": model,
                "case": case_name,
                "status": case_status,
                "text": response_text("gemini-native", case_response),
                "response": case_response,
            }
        )
    return results


def main() -> int:
    openrouter_key = os.environ.get("OPENROUTER_API_KEY", "")
    gemini_key = os.environ.get("GEMINI_API_KEY", "")
    if not openrouter_key and not gemini_key:
        print("Set OPENROUTER_API_KEY and/or GEMINI_API_KEY.", file=sys.stderr)
        return 2

    mime, encoded = download_image()
    image_uri = f"data:{mime};base64,{encoded}"
    results: list[JSON] = []

    for model in MODELS:
        print(f"\n=== {model} ===")
        if openrouter_key:
            results.extend(
                openai_compatible(
                    "openrouter",
                    "https://openrouter.ai/api/v1/chat/completions",
                    openrouter_key,
                    model,
                    image_uri,
                )
            )
        if gemini_key:
            results.extend(
                openai_compatible(
                    "google-openai-compat",
                    "https://generativelanguage.googleapis.com/v1beta/openai/chat/completions",
                    gemini_key,
                    model,
                    image_uri,
                )
            )
            results.extend(native_gemini(gemini_key, model, mime, encoded))

    RESULT_FILE.write_text(json.dumps(results, ensure_ascii=False, indent=2), encoding="utf-8")
    print(f"\nFull responses: {RESULT_FILE}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

[Venkaiahbabuneelam]

Hi @turicas,
Thank you for the detailed report! The recommended approach for multimodal function responses has been updated in the current documentation please refer to the Multimodal function responses section here: https://ai.google.dev/gemini-api/docs/function-calling#multimodal-function-responses
The working pattern you've already identified (image in functionResponse.parts without $ref) aligns with the current guidance.

Thanks