Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Imagen Documentation #8776

Open
wants to merge 7 commits into
base: dl/vertex-imagen
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 14 additions & 50 deletions common/api-review/vertexai.api.md
Original file line number Diff line number Diff line change
Expand Up @@ -346,7 +346,7 @@ export class GenerativeModel extends VertexAIModel {
// @public
export function getGenerativeModel(vertexAI: VertexAI, modelParams: ModelParams, requestOptions?: RequestOptions): GenerativeModel;

// @public (undocumented)
// @beta
export function getImagenModel(vertexAI: VertexAI, modelParams: ImagenModelParams, requestOptions?: RequestOptions): ImagenModel;

// @public
Expand Down Expand Up @@ -430,122 +430,87 @@ export enum HarmSeverity {
HARM_SEVERITY_NEGLIGIBLE = "HARM_SEVERITY_NEGLIGIBLE"
}

// @public (undocumented)
// @beta
export enum ImagenAspectRatio {
// (undocumented)
LANDSCAPE_16x9 = "16:9",
// (undocumented)
LANDSCAPE_3x4 = "3:4",
// (undocumented)
PORTRAIT_4x3 = "4:3",
// (undocumented)
PORTRAIT_9x16 = "9:16",
// (undocumented)
SQUARE = "1:1"
}

// @public (undocumented)
// @beta
export interface ImagenGCSImage {
// (undocumented)
gcsURI: string;
// (undocumented)
mimeType: string;
}

// @public (undocumented)
// @beta
export interface ImagenGenerationConfig {
// (undocumented)
addWatermark?: boolean;
// (undocumented)
aspectRatio?: ImagenAspectRatio;
// (undocumented)
imageFormat?: ImagenImageFormat;
// (undocumented)
negativePrompt?: string;
// (undocumented)
numberOfImages?: number;
}

// @public (undocumented)
// @beta
export interface ImagenGenerationResponse<T extends ImagenInlineImage | ImagenGCSImage> {
// (undocumented)
filteredReason?: string;
// (undocumented)
images: T[];
}

// @public
// @beta
export class ImagenImageFormat {
// (undocumented)
compressionQuality?: number;
// (undocumented)
static jpeg(compressionQuality?: number): ImagenImageFormat;
// (undocumented)
mimeType: string;
// (undocumented)
static png(): ImagenImageFormat;
}

// @public
// @beta
export interface ImagenInlineImage {
// (undocumented)
bytesBase64Encoded: string;
// (undocumented)
mimeType: string;
}

// @public (undocumented)
// @beta
export class ImagenModel extends VertexAIModel {
constructor(vertexAI: VertexAI, modelParams: ImagenModelParams, requestOptions?: RequestOptions | undefined);
// (undocumented)
generateImages(prompt: string): Promise<ImagenGenerationResponse<ImagenInlineImage>>;
// (undocumented)
// @internal
generateImagesGCS(prompt: string, gcsURI: string): Promise<ImagenGenerationResponse<ImagenGCSImage>>;
// (undocumented)
generationConfig?: ImagenGenerationConfig;
// (undocumented)
requestOptions?: RequestOptions | undefined;
// (undocumented)
safetySettings?: ImagenSafetySettings;
}

// @public (undocumented)
// @beta
export interface ImagenModelParams {
// (undocumented)
generationConfig?: ImagenGenerationConfig;
// (undocumented)
model: string;
// (undocumented)
safetySettings?: ImagenSafetySettings;
}

// @public (undocumented)
// @beta
export enum ImagenPersonFilterLevel {
// (undocumented)
ALLOW_ADULT = "allow_adult",
// (undocumented)
ALLOW_ALL = "allow_all",
// (undocumented)
BLOCK_ALL = "dont_allow"
}

// @public (undocumented)
// @beta
export enum ImagenSafetyFilterLevel {
// (undocumented)
BLOCK_LOW_AND_ABOVE = "block_low_and_above",
// (undocumented)
BLOCK_MEDIUM_AND_ABOVE = "block_medium_and_above",
// (undocumented)
BLOCK_NONE = "block_none",
// (undocumented)
BLOCK_ONLY_HIGH = "block_only_high"
}

// @public (undocumented)
// @beta
export interface ImagenSafetySettings {
// (undocumented)
personFilterLevel?: ImagenPersonFilterLevel;
// (undocumented)
safetyFilterLevel?: ImagenSafetyFilterLevel;
}

Expand Down Expand Up @@ -838,13 +803,12 @@ export const enum VertexAIErrorCode {
RESPONSE_ERROR = "response-error"
}

// @public (undocumented)
// @public
export abstract class VertexAIModel {
// @internal
protected constructor(vertexAI: VertexAI, modelName: string);
// @internal (undocumented)
protected _apiSettings: ApiSettings;
// (undocumented)
readonly model: string;
static normalizeModelName(modelName: string): string;
}
Expand Down
25 changes: 23 additions & 2 deletions docs-devsite/vertexai.imagengcsimage.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ https://github.com/firebase/firebase-js-sdk
{% endcomment %}

# ImagenGCSImage interface
> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
>
An image generated by Imagen, stored in Cloud Storage (GCS) for Firebase.

<b>Signature:</b>

```typescript
Expand All @@ -20,19 +25,35 @@ export interface ImagenGCSImage

| Property | Type | Description |
| --- | --- | --- |
| [gcsURI](./vertexai.imagengcsimage.md#imagengcsimagegcsuri) | string | |
| [mimeType](./vertexai.imagengcsimage.md#imagengcsimagemimetype) | string | |
| [gcsURI](./vertexai.imagengcsimage.md#imagengcsimagegcsuri) | string | <b><i>(BETA)</i></b> The URI of the file in Cloud Storage (GCS) for Firestore. |
Copy link
Contributor

@rachelsaunders rachelsaunders Feb 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| [gcsURI](./vertexai.imagengcsimage.md#imagengcsimagegcsuri) | string | <b><i>(BETA)</i></b> The URI of the file in Cloud Storage (GCS) for Firestore. |
| [gcsURI](./vertexai.imagengcsimage.md#imagengcsimagegcsuri) | string | <b><i>(BETA)</i></b> The URI of the file stored in a Cloud Storage for Firebase bucket. |

| [mimeType](./vertexai.imagengcsimage.md#imagengcsimagemimetype) | string | <b><i>(BETA)</i></b> The MIME type of the image; either <code>&quot;image/png&quot;</code> or <code>&quot;image/jpeg&quot;</code>.<!-- -->To request a different format, set the <code>imageFormat</code> property in your <code>[ImagenGenerationConfig](./vertexai.imagengenerationconfig.md#imagengenerationconfig_interface)</code>. |

## ImagenGCSImage.gcsURI

> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
>

The URI of the file in Cloud Storage (GCS) for Firestore.

<b>Signature:</b>

```typescript
gcsURI: string;
```

### Example

`"gs://bucket-name/path/sample_0.jpg"`<!-- -->.

## ImagenGCSImage.mimeType

> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
>

The MIME type of the image; either `"image/png"` or `"image/jpeg"`<!-- -->.

To request a different format, set the `imageFormat` property in your <code>[ImagenGenerationConfig](./vertexai.imagengenerationconfig.md#imagengenerationconfig_interface)</code>.

<b>Signature:</b>

```typescript
Expand Down
54 changes: 49 additions & 5 deletions docs-devsite/vertexai.imagengenerationconfig.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,13 @@ https://github.com/firebase/firebase-js-sdk
{% endcomment %}

# ImagenGenerationConfig interface
> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
>
Configuration options for generating images with Imagen.

See the [Cloud documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/image/generate-images) for more details.
Copy link
Contributor

@rachelsaunders rachelsaunders Feb 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll have Firebase specific docs about this that I'd prefer we link our devs to.

The URL will be http://firebase.google.com/docs/vertex-ai/generate-images

Note that the URL will 404 until launch day.


<b>Signature:</b>

```typescript
Expand All @@ -20,14 +27,23 @@ export interface ImagenGenerationConfig

| Property | Type | Description |
| --- | --- | --- |
| [addWatermark](./vertexai.imagengenerationconfig.md#imagengenerationconfigaddwatermark) | boolean | |
| [aspectRatio](./vertexai.imagengenerationconfig.md#imagengenerationconfigaspectratio) | [ImagenAspectRatio](./vertexai.md#imagenaspectratio) | |
| [imageFormat](./vertexai.imagengenerationconfig.md#imagengenerationconfigimageformat) | [ImagenImageFormat](./vertexai.imagenimageformat.md#imagenimageformat_class) | |
| [negativePrompt](./vertexai.imagengenerationconfig.md#imagengenerationconfignegativeprompt) | string | |
| [numberOfImages](./vertexai.imagengenerationconfig.md#imagengenerationconfignumberofimages) | number | |
| [addWatermark](./vertexai.imagengenerationconfig.md#imagengenerationconfigaddwatermark) | boolean | <b><i>(BETA)</i></b> Whether to add an invisible watermark to generated images.<!-- -->If set to <code>true</code>, an invisible SynthID watermark is embedded in generated images to indicate that they are AI generated. If set to <code>false</code>, watermarking will be disabled.<!-- -->The default value depends on the Imagen model; see the [addWatermark](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/imagen-api#parameter_list) documentation for more details. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addWatermark

^ should be tagged as code

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same thing for all these references to specific parameters

Copy link
Contributor

@rachelsaunders rachelsaunders Feb 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any way that we can use "Public Preview" or "Preview" here and throughout instead of "(BETA)"?

| [aspectRatio](./vertexai.imagengenerationconfig.md#imagengenerationconfigaspectratio) | [ImagenAspectRatio](./vertexai.md#imagenaspectratio) | <b><i>(BETA)</i></b> The aspect ratio of the generated images. The default value is square 1:1. Supported aspect ratios depend on the Imagen model, see <code>[ImagenAspectRatio](./vertexai.md#imagenaspectratio)</code> for more details. |
| [imageFormat](./vertexai.imagengenerationconfig.md#imagengenerationconfigimageformat) | [ImagenImageFormat](./vertexai.imagenimageformat.md#imagenimageformat_class) | <b><i>(BETA)</i></b> The image format of the generated images. The default is PNG.<!-- -->See <code>[ImagenImageFormat](./vertexai.imagenimageformat.md#imagenimageformat_class)</code> for more details. |
| [negativePrompt](./vertexai.imagengenerationconfig.md#imagengenerationconfignegativeprompt) | string | <b><i>(BETA)</i></b> A description of what should be omitted from the generated images.<!-- -->Support for negative prompts depends on the Imagen model.<!-- -->See the [Cloud documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/image/generate-images#negative-prompt) for more details. |
| [numberOfImages](./vertexai.imagengenerationconfig.md#imagengenerationconfignumberofimages) | number | <b><i>(BETA)</i></b> The number of images to generate. The default value is 1.<!-- -->The number of sample images that may be generated in each request depends on the model (typically up to 4); see the [sampleCount](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/imagen-api#parameter_list) documentation for more details. |

## ImagenGenerationConfig.addWatermark

> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
>

Whether to add an invisible watermark to generated images.

If set to `true`<!-- -->, an invisible SynthID watermark is embedded in generated images to indicate that they are AI generated. If set to `false`<!-- -->, watermarking will be disabled.

The default value depends on the Imagen model; see the [addWatermark](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/imagen-api#parameter_list) documentation for more details.

<b>Signature:</b>

```typescript
Expand All @@ -36,6 +52,11 @@ addWatermark?: boolean;

## ImagenGenerationConfig.aspectRatio

> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
>

The aspect ratio of the generated images. The default value is square 1:1. Supported aspect ratios depend on the Imagen model, see <code>[ImagenAspectRatio](./vertexai.md#imagenaspectratio)</code> for more details.

<b>Signature:</b>

```typescript
Expand All @@ -44,6 +65,13 @@ aspectRatio?: ImagenAspectRatio;

## ImagenGenerationConfig.imageFormat

> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
>

The image format of the generated images. The default is PNG.

See <code>[ImagenImageFormat](./vertexai.imagenimageformat.md#imagenimageformat_class)</code> for more details.

<b>Signature:</b>

```typescript
Expand All @@ -52,6 +80,15 @@ imageFormat?: ImagenImageFormat;

## ImagenGenerationConfig.negativePrompt

> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
>

A description of what should be omitted from the generated images.

Support for negative prompts depends on the Imagen model.

See the [Cloud documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/image/generate-images#negative-prompt) for more details.

<b>Signature:</b>

```typescript
Expand All @@ -60,6 +97,13 @@ negativePrompt?: string;

## ImagenGenerationConfig.numberOfImages

> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
>

The number of images to generate. The default value is 1.

The number of sample images that may be generated in each request depends on the model (typically up to 4); see the [sampleCount](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/imagen-api#parameter_list) documentation for more details.

<b>Signature:</b>

```typescript
Expand Down
25 changes: 23 additions & 2 deletions docs-devsite/vertexai.imagengenerationresponse.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ https://github.com/firebase/firebase-js-sdk
{% endcomment %}

# ImagenGenerationResponse interface
> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
>
The response from a request to generate images with Imagen.

<b>Signature:</b>

```typescript
Expand All @@ -20,11 +25,20 @@ export interface ImagenGenerationResponse<T extends ImagenInlineImage | ImagenGC

| Property | Type | Description |
| --- | --- | --- |
| [filteredReason](./vertexai.imagengenerationresponse.md#imagengenerationresponsefilteredreason) | string | |
| [images](./vertexai.imagengenerationresponse.md#imagengenerationresponseimages) | T\[\] | |
| [filteredReason](./vertexai.imagengenerationresponse.md#imagengenerationresponsefilteredreason) | string | <b><i>(BETA)</i></b> The reason that images were filtered out. This property will only be defined if one or more images were filtered.<!-- -->Images may be filtered out due to the <code>[ImagenSafetyFilterLevel](./vertexai.md#imagensafetyfilterlevel)</code>, <code>[ImagenPersonFilterLevel](./vertexai.md#imagenpersonfilterlevel)</code>, or filtering included in the model. The filter levels may be adjusted in your <code>[ImagenSafetySettings](./vertexai.imagensafetysettings.md#imagensafetysettings_interface)</code>.<!-- -->See the [Responsible AI and usage guidelines for Imagen](https://cloud.google.com/vertex-ai/generative-ai/docs/image/responsible-ai-imagen) for more details. |
| [images](./vertexai.imagengenerationresponse.md#imagengenerationresponseimages) | T\[\] | <b><i>(BETA)</i></b> The images generated by Imagen.<!-- -->The number of images generated may be fewer than the number requested if one or more were filtered out; see <code>filteredReason</code>. |

## ImagenGenerationResponse.filteredReason

> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
>

The reason that images were filtered out. This property will only be defined if one or more images were filtered.

Images may be filtered out due to the <code>[ImagenSafetyFilterLevel](./vertexai.md#imagensafetyfilterlevel)</code>, <code>[ImagenPersonFilterLevel](./vertexai.md#imagenpersonfilterlevel)</code>, or filtering included in the model. The filter levels may be adjusted in your <code>[ImagenSafetySettings](./vertexai.imagensafetysettings.md#imagensafetysettings_interface)</code>.

See the [Responsible AI and usage guidelines for Imagen](https://cloud.google.com/vertex-ai/generative-ai/docs/image/responsible-ai-imagen) for more details.

<b>Signature:</b>

```typescript
Expand All @@ -33,6 +47,13 @@ filteredReason?: string;

## ImagenGenerationResponse.images

> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
>

The images generated by Imagen.

The number of images generated may be fewer than the number requested if one or more were filtered out; see `filteredReason`<!-- -->.

<b>Signature:</b>

```typescript
Expand Down
Loading
Loading