Skip to content

Commit

Permalink
Python: Bug: Type Definition Error ImageContent (#10395)
Browse files Browse the repository at this point in the history
### Motivation and Context  
1. This change is required because IDE type-checking throws an error
when manually passing values (e.g., Base64 strings) to ImageContent’s
constructor.
2. It fixes the missing argument definitions (data, data_uri,
data_format, mime_type), which caused the type error in IDEs like VS
Code.
3. It resolves the issue reported where users observed IDE-type errors
despite code functioning at runtime.
   
### Description  
• Added explicit type definitions (data, data_uri, data_format,
mime_type) in the ImageContent class constructor and fields.
• Updated the docstring and __init__ to reflect the new arguments.    
• Ensured the code maintains backward compatibility by preserving
existing methods like from_image_file.
• Verified that type-checking in major Python IDEs now passes without
warnings.

### Example

```
import asyncio
import base64

from semantic_kernel.connectors.ai.open_ai.services.azure_chat_completion import AzureChatCompletion,
from semantic_kernel.contents import AuthorRole, ChatHistory, ImageContent
from semantic_kernel.contents.chat_message_content import ChatMessageContent
from semantic_kernel.contents.text_content import TextContent


async def main():
    service = AzureChatCompletion(
        api_key="************************",
        deployment_name="gpt-4o-mini",
        endpoint="https://************************.openai.azure.com",
        api_version="2024-12-01-preview",
    )

    chat_history = ChatHistory()

    # example of base64 image
    # or `ImageContent.from_image_file`
    with open("weather_icon.png", "rb") as image_file:
        b64_string = base64.b64encode(image_file.read()).decode("utf-8")

    chat_history.add_message(
        ChatMessageContent(
            role=AuthorRole.USER,
            items=[
                ImageContent(
                    data_uri=f"data:image/png;base64,{b64_string}",
                    data_format="base64",
                    mime_type="image/png",
                ),
                TextContent(text="what is the weather"),
            ],
        )
    )
    response = await service.get_chat_message_contents(
        chat_history=chat_history,
        settings=service.get_prompt_execution_settings_class()(),
    )

    print(response[0].content)
    # >> The icon you provided typically represents sunny or clear weather.
    # >> If you need specific weather information for a location,
    # >> please provide the location or check a reliable weather service for the latest updates.


asyncio.run(main())
```   

### Contribution Checklist  
- [x] The code builds clean without any errors or warnings    
- [x] The PR follows the SK Contribution Guidelines and the
pre-submission formatting script raises no violations
- [x] All unit tests pass, and I have added new tests where possible    
- [x] I didn’t break anyone ☺

---------

Co-authored-by: Tao Chen <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
  • Loading branch information
3 people authored Feb 5, 2025
1 parent 0a8c4d8 commit 4d2b2be
Showing 1 changed file with 34 additions and 0 deletions.
34 changes: 34 additions & 0 deletions python/semantic_kernel/contents/image_content.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,40 @@ class ImageContent(BinaryContent):
ValidationError: If neither uri or data is provided.
"""

def __init__(
self,
uri: str | None = None,
data_uri: str | None = None,
data: str | bytes | None = None,
data_format: str | None = None,
mime_type: str | None = None,
**kwargs: Any,
):
"""Create an Image Content object, either from a data_uri or data.
Args:
uri (Url | None): The reference uri of the content.
data_uri (DataUrl | None): The data uri of the content.
data (str | bytes | None): The data of the content.
data_format (str | None): The format of the data (e.g. base64).
mime_type (str | None): The mime type of the image, only used with data.
kwargs (Any): Any additional arguments:
inner_content (Any): The inner content of the response,
this should hold all the information from the response so even
when not creating a subclass a developer
can leverage the full thing.
ai_model_id (str | None): The id of the AI model that generated this response.
metadata (dict[str, Any]): Any metadata that should be attached to the response.
"""
super().__init__(
uri=uri,
data_uri=data_uri,
data=data,
data_format=data_format,
mime_type=mime_type,
**kwargs,
)

content_type: Literal[ContentTypes.IMAGE_CONTENT] = Field(IMAGE_CONTENT_TAG, init=False) # type: ignore
tag: ClassVar[str] = IMAGE_CONTENT_TAG

Expand Down

0 comments on commit 4d2b2be

Please sign in to comment.