Extending Search with AI Image and Video Analysis

[zawilliams]

Hey all,

I've got a large amount of assets in a Craft install (40k+) - photos and video from a marketing company. I'm looking at integrating an AI search API like Google Vision and Algolia to analyze and index photos and videos so I could allow someone to search for assets with natural language (e.g. "horses on a sunny day").

Would overriding the asset search even be possible in order to enable this kind of functionality?

[brandonkelly]

You can latch onto the ElementQuery::EVENT_BEFORE_PREPARE event, and make whatever modifications you want to it, like scraping the search param value and passing it to an API, and then removing search and adding $query->andWhere(['elements.id' => $filteredElementIds]).

[vitordiniz22]

I ran into this exact problem and ended up building a plugin for it: Lens. Disclosure: I'm the author.

It takes the approach Brandon describes one step further: instead of calling a vision API at search time, it analyzes each image once (alt text, long description, semantic tags, OCR'd text) and indexes the results into Craft's native search. Queries like "team meeting outdoors", or a word from a sign in the background, work in the asset browser and in every image picker, with no per-search API calls and relevance-ranked results.

For a 40,000-asset library specifically: it bulk-processes whole volumes through Craft's queue with a cost estimate up front, and you bring your own OpenAI, Gemini, or Anthropic key, so you pay the provider directly. One honest limitation versus your original question: it covers images only, not video.