Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion skills/firebase-ai-logic-basics/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,8 @@ snippets for Android (Kotlin), iOS (Swift), Flutter (Dart), and Web.
> for the currently supported image generation (Nano Banana) model names.

- Requires an upgraded Blaze pay-as-you-go billing plan.
- See the
[Image Generation Guide](references/sdk/capabilities/image_generation.md).

### Search Grounding with the built in googleSearch tool

Expand Down Expand Up @@ -207,7 +209,8 @@ Hybrid inference runs prompts with an on-device model when one is available and
falls back to a cloud-hosted model otherwise (or vice versa). Availability and
setup steps differ by platform, more info in the
[hybrid and on-device documentation](https://firebase.google.com/docs/ai-logic/hybrid.md.txt),
which links to each platform's guide.
which links to each platform's guide, and in the
[On-Device & Hybrid Inference Guide](references/sdk/capabilities/on_device_hybrid.md).

______________________________________________________________________

Expand Down Expand Up @@ -313,4 +316,6 @@ ______________________________________________________________________
[Structured output (JSON Schema)](references/sdk/capabilities/structured_output.md)
[Function calling (tools)](references/sdk/capabilities/function_calling.md)
[Search grounding (googleSearch)](references/sdk/capabilities/search_grounding.md)
[Image generation (Nano Banana)](references/sdk/capabilities/image_generation.md)
[On-device & hybrid inference](references/sdk/capabilities/on_device_hybrid.md)
[Client-side Text-to-Speech (TTS) generation](references/sdk/capabilities/text_to_speech.md)
Original file line number Diff line number Diff line change
@@ -1,5 +1,69 @@
# Image Generation (Nano Banana)

Gemini Image models ("Nano Banana") generate and edit images directly from the
client. Use this capability when an app needs to create a picture from a text
prompt, render text inside an image, produce interleaved text-and-image output
(for example an illustrated recipe), edit a user-supplied image, or refine an
image over several chat turns. Image generation is a Preview feature, works only
with dedicated Gemini Image models (never the general-purpose text models), and
requires the Blaze pay-as-you-go billing plan regardless of the Gemini API
provider.

## Configuration

> [!WARNING] **Blaze plan required.** Image generation requires an upgraded
> Blaze pay-as-you-go billing plan regardless of your Gemini API provider.
> Always check the
> [Firebase AI Logic Models documentation](https://firebase.google.com/docs/ai-logic/models.md.txt)
> for the currently supported image generation (Nano Banana) model names.

- **Model**: Use a dedicated Gemini Image model. General-use Gemini models do
not generate images, and legacy Imagen models (`ImagenModel` / `imagen-*`) are
shut down and removed in the latest SDKs (see the
[Imagen to Gemini migration guide](https://firebase.google.com/docs/ai-logic/imagen-models-migration.md.txt)).
Every snippet below uses the placeholder `<latest_supported_image_model>`;
replace it with the current image model from the models page.
- **Response modality (required)**: The model must be configured with
`responseModalities` in `generationConfig` that include `IMAGE`. Add `TEXT`
only when you want interleaved text and images in the same response (the model
then returns text parts and image parts in order); for image-only output
request `IMAGE` alone.
- **Reading the result**: Generated images come back as inline data parts (PNG).
Kotlin exposes them as `ImagePart` (a `Bitmap`) in
`candidates.first().content.parts`; Swift as `response.inlineDataParts`
(`Data`); Dart as `response.inlineDataParts` (`bytes`); Web as
`result.response.inlineDataParts()` (base64 `data` plus `mimeType`).
- **Output size**: By default images are square (1:1) at 1024x1024. Set
`imageConfig` inside `generationConfig` to choose an aspect ratio and image
size (supported sizes depend on the model):
- **Android**:
`imageConfig = imageConfig { aspectRatio = AspectRatio.LANDSCAPE_16x9; imageSize = ImageSize.SIZE_2K }`
- **iOS**:
`imageConfig: ImageConfig(aspectRatio: .landscape16x9, imageSize: .size2K)`
- **Flutter**:
`imageConfig: ImageConfig(aspectRatio: ImageAspectRatio.landscape16x9, imageSize: ImageSize.size2K)`
- **Web**: `imageConfig: { aspectRatio: ImageConfigAspectRatio.LANDSCAPE_16x9, imageSize: ImageConfigImageSize.SIZE_2K }`
(or the underlying string literals: aspect ratios `1:1`, `1:4`, `1:8`, `2:3`,
`3:2`, `3:4`, `4:1`, `4:3`, `4:5`, `5:4`, `8:1`, `9:16`, `16:9`, `21:9`;
sizes `512`, `1K`, `2K`, `4K` with uppercase `K`).
- **Input**: All Gemini Image models accept image input (for editing, and as
reference images for objects, characters, or style; per-model limits are in
the official guide); some accept video input; none accept audio. Inline
request data is limited to 20 MB.
- **Editing and iteration**: Pass the source image together with the text
prompt. For iterative edits, use a chat session (`startChat()` /
`sendMessage()`) with the same model configuration; follow-up turns do not
need to resend the image.
- **Images are not guaranteed**: The model may answer with text only
(`FinishReason` `NO_IMAGE`), especially for ambiguous prompts, or refuse an
unsafe prompt (`FinishReason` `STOP`). On iOS
(`GenerateContentError.responseStoppedEarly`) and Web (`AIError` from
`inlineDataParts()`), `NO_IMAGE` throws an error rather than returning empty
image parts. Ask for images explicitly ("generate an image", "update the
image") and handle the no-image case in the UI.
- Grounding with Google Search and structured output are supported on some image
models only; check the official guide for the current model matrix.

## Android (Kotlin)

Create the model with `ResponseModality.IMAGE`, call `generateContent` from a
Expand Down Expand Up @@ -254,3 +318,37 @@ try {
console.error('Prompt or candidate was blocked:', err);
}
```

## Troubleshooting

- **Response contains text but no image, or throws `NO_IMAGE`** (`FinishReason`
is `NO_IMAGE`; throws `GenerateContentError.responseStoppedEarly` on iOS and
`AIError` from `inlineDataParts()` on Web): the prompt was ambiguous. Ask for
image output explicitly ("generate an image", "provide images as you go
along", "update the image") or retry.
- **Model refuses and returns only text** (`FinishReason` is `STOP`): the prompt
was judged potentially unsafe; the model will not create the image.
- **No image parts and the model is a general-use Gemini model**: image
generation is only supported by Gemini Image models. Pick an image model from
the models page.
- **Billing or permission errors**: the project must be on the Blaze plan for
image generation, even with the Gemini Developer API.
- **`imageConfig` rejected or fails to compile**: Android (`AspectRatio`,
`ImageSize`), iOS (`ImageConfig.AspectRatio`, `ImageConfig.ImageSize`), and
Flutter (`ImageAspectRatio`, `ImageSize`) use strongly-typed constants rather
than raw strings; on Web, use `ImageConfigAspectRatio` / `ImageConfigImageSize`
or string literals with an uppercase `K` suffix (`1K`, `2K`, `4K`; `512` has
no suffix).
- **HTTP 413 when editing**: inline request data is limited to 20 MB; resize or
compress the input image before sending it.
- **Generated text appears inside the image instead of as text**: ask for text
output explicitly ("generate narrative text along with illustrations"). When
an image must contain text, generate the text first, then the image.

See the official image generation guide:
https://firebase.google.com/docs/ai-logic/generate-images-gemini

Last verified against
https://firebase.google.com/docs/ai-logic/generate-images-gemini.md.txt and the
Firebase SDK sources (firebase-android-sdk, firebase-ios-sdk, firebase-js-sdk,
flutterfire; `main` branches) on 2026-10-09.
Original file line number Diff line number Diff line change
@@ -1,5 +1,95 @@
# On-Device & Hybrid Inference

Hybrid inference lets an app run an on-device model when available and fall back
to a cloud-hosted Gemini model (or the reverse) depending on availability:

- **Android (`com.google.firebase:firebase-ai` +
`com.google.firebase:firebase-ai-ondevice`)**: uses AICore / ML Kit GenAI
Prompt (Gemini Nano) on supported Android devices.
- **Web (`firebase/ai`)**: uses Chrome's built-in `LanguageModel` Prompt API
(Gemini Nano) on Chrome on Desktop (v139+).
- **iOS (`FirebaseAILogic` via Swift Package Manager)**: in `firebase-ios-sdk`
v13.0.0+ (iOS 27+ / macOS 27+), `ai.geminiLanguageModel(name:)` conforms to
Apple's `FoundationModels.LanguageModel` protocol so apps can route between
Apple's on-device `SystemLanguageModel.default` and cloud Gemini models with
`LanguageModelSession` (replacing the experimental `HybridModel` /
`GenerativeModelSession` wrapper from v12.13.0–12.x).
- **Flutter (`firebase_ai`)**: not available via Firebase AI Logic (always calls
the cloud-hosted Gemini API).

Reach for hybrid inference when a feature benefits from on-device execution
(enhanced privacy, local context, inference at no cost, offline use) while still
reaching every user through the cloud fallback.

## Configuration

> [!WARNING] **Preview feature.** On-device and hybrid inference on Android
> (`@PublicPreviewAPI` in `firebase-ai` + `firebase-ai-ondevice`) and Web
> (Chrome Desktop v139+) are in Preview and may change in backwards-incompatible
> ways.

### Inference modes

Set `mode` (Web) or `OnDeviceConfig(mode = ...)` (Android) when creating the
model. Both platforms share the same four `InferenceMode` constants:

| Mode | Behavior |
| ------------------ | ----------------------------------------------------------------------------------- |
| `PREFER_ON_DEVICE` | Use the on-device model if it's available; otherwise fall back to the cloud model. |
| `ONLY_ON_DEVICE` | Use the on-device model if it's available; otherwise throw an exception. |
| `PREFER_IN_CLOUD` | Use the cloud-hosted model if it's available; otherwise fall back to the on-device. |
| `ONLY_IN_CLOUD` | Use the cloud-hosted model if it's available; otherwise throw an exception. |

- **Inspecting which path ran**: read `response.inferenceSource` (Android) or
`result.response.inferenceSource` (Web) and compare it with
`InferenceSource.ON_DEVICE` / `InferenceSource.IN_CLOUD`.
- **Model lifecycle**:
- **Android**: inspect `model.onDeviceExtension?.checkStatus()`
(`OnDeviceModelStatus.AVAILABLE`, `DOWNLOADABLE`, `DOWNLOADING`,
`UNAVAILABLE`) and trigger downloads with
`model.onDeviceExtension?.download()` (returns `Flow<DownloadStatus>`).
- **Web**: call `await hybridModel.initializeDeviceModel(onDownloadProgress?)`
on or after an end-user interaction (such as a button click) to trigger and
await the Gemini Nano download in Chrome. If `initializeDeviceModel()` has
not been called and the model is still `"downloadable"`, `PREFER_ON_DEVICE`
falls back to the cloud without starting a download.
- **Cloud fallback configuration**:
- **Android**: pass `modelName`, `generationConfig`, `safetySettings`, and
`systemInstruction` directly to
`generativeModel(..., onDeviceConfig = OnDeviceConfig(...))`.
- **Web**: pass cloud settings in `inCloudParams` (`model`,
`generationConfig`, etc.; default cloud model is `gemini-3.5-flash-lite` in
`firebase@13.0.0+`, or `gemini-2.5-flash-lite` in `firebase@12.x`) and
Chrome Prompt API options in `onDeviceParams` (`createOptions`,
`promptOptions.responseConstraint`). Note that `createOptions.temperature`
and `topK` were removed in `firebase@13.0.0` (`@firebase/ai@3.0.0`).

### Supported on-device vs cloud

Supported for on-device inference:

- Single-turn content generation, streaming (`generateContentStream`) and
non-streaming (`generateContent`)
- Text from text-only input (plus text `systemInstruction` on Android; up to
4096 tokens on iOS)
- Text from text-and-image input (single `Bitmap` + text on Android with a
4000-token request limit; JPEG/PNG on Web with a 9216-token limit)
- Structured output (via `generateObject` / `@Generable` KSP on Android, or
`onDeviceParams.promptOptions.responseConstraint` on Web)
- `countTokens` is supported on Android on-device, but throws on Web
`ONLY_ON_DEVICE` (and calls the cloud in hybrid modes). Avoid relying on
`countTokens` to decide hybrid fallback thresholds because token counts differ
between cloud-hosted and on-device models.

Not supported on-device (with `PREFER_ON_DEVICE` the SDK falls back to the cloud
automatically; with `ONLY_ON_DEVICE` the request throws):

- Audio, video, and document (PDF) inputs, or file URLs (`gs://`)
- Image generation with Gemini or Imagen models
- Multi-turn chat (`startChat`) and bidirectional Live API streaming
- Tools (function calling, Google Search grounding)
- AI monitoring in the Firebase console for on-device requests

## Android (Kotlin)

Add `com.google.firebase:firebase-ai-ondevice` alongside
Expand Down Expand Up @@ -181,3 +271,37 @@ const hybridModel = getGenerativeModel(ai, {
For enum output use `Schema.enumString({ enum: [...] })` with
`responseMimeType: "text/x.enum"` in `inCloudParams` and the same schema as the
`responseConstraint`.

## Troubleshooting

- **Web: requests always fall back to the cloud when Gemini Nano is
`"downloadable"`**: call `await hybridModel.initializeDeviceModel()` inside a
user gesture handler (or pre-download with `await LanguageModel.create()` in
Chrome DevTools) — `generateContent` does not trigger a new model download on
its own.
- **Android: `OnDeviceModelStatus.UNAVAILABLE`**: the device does not support
AICore / Gemini Nano, or `com.google.firebase:firebase-ai-ondevice` is missing
from `dependencies`. With `PREFER_ON_DEVICE`, requests automatically fall back
to the cloud model.
- **`ONLY_ON_DEVICE` throws**: the on-device model is unavailable or not yet
downloaded, or the request uses a feature that is not supported on-device
(chat, tools, audio/video/PDF, file URLs). Use `PREFER_ON_DEVICE` if a cloud
fallback is acceptable.
- **Requests unexpectedly run in the cloud**: `PREFER_ON_DEVICE` silently falls
back for unsupported features or when Gemini Nano is unavailable. Check
`response.inferenceSource` / `result.response.inferenceSource`.
- **`countTokens` throws on Web or differs from on-device**: token counting
throws on Web in `ONLY_ON_DEVICE` mode (and queries the cloud model in
`PREFER_ON_DEVICE` / `PREFER_IN_CLOUD`). Even on Android (where on-device
`countTokens` is supported), token counts differ between cloud-hosted and
on-device models, so do not rely on `countTokens` for fallback decisions.
- **Web: works on localhost but not for end users**: the Chrome Origin Trial
token is missing, expired, or not served on that page.

See the official hybrid on-device inference guide:
https://firebase.google.com/docs/ai-logic/hybrid

Last verified against
https://firebase.google.com/docs/ai-logic/hybrid.md.txt and
the Firebase SDK sources (firebase-android-sdk, firebase-ios-sdk,
firebase-js-sdk, flutterfire; `main` branches) on 2026-10-09.
Loading