Ability to use filters on index for additional searchparams/options

[ToshY]

Notice
Backstory: I've been migrating majority of heavy-duty queries from Doctrine ORM to Meilisearch.

Description
I would like to the possibility to be able to create filters that would allow the search / rawSearch methods to have additional searchParams / options from these filters.

Basic example

• The idea would be similar to Doctrine Filters, where an additional condition is added to the query when a filter is enabled.
• What I'm after is to create event listeners to add filters to seaches by default (also see this Symfony eventlistener example with Doctrine filters).

Other
An example use-case for this is as follows:

• Let's say I have authenticated and guest users.
• Both these types of users can perform searches on the same index, but the authenticated users can have access to more results.
• In order to achieve this, an additional field is added to the documents, e.g. freeTier.
  • The freeTier = 1 is available for guests and authenticated users.
  • The freeTier = 0 is explicitly available for authenticated users.
• If the guest user seaches, it should add an additional filter to the searchParams which essentially would add AND (freeTier = 1) filter.
• If the authenticated user searches, it should not add this additional filter as authenticated users are allowed to search for both freeTier values 0 and 1.

Now if similar, but not exact same, searches are performed on the same index, I'd still like it to take into consideration this freeTier = 1 field without manually concatenating this filter to every place a search on this index is performed. But with that in mind, if for some reason it is not desired to use the filter at a certain search, I should also have the possibility to disable/enable at any time.

---

The feature request boils down to a kind of Doctrine filters functionality for the Meilisearch search methods.

[norkunas]

I think if considering this then it should more belong to https://github.com/meilisearch/meilisearch-symfony than to this repo

[ToshY]

Hey @norkunas 👋

As a Symfony user myself I'm okay with that, and I can recreate the issue there if needed (or maybe you can move it over there?).

However, the initial thought of creating the issue in this repo was that Doctrine filters also aren't specific to Symfony. With that in mind, making a more generic implementation for filters would allow users that don't use Symfony to benefit from this feature as well.

[norkunas]

However, the initial thought of creating the issue in this repo was that Doctrine filters also aren't specific to Symfony.

But those filters are specific to ORM while this library is more like DBAL because it's just an api client, that's why I think this way.

With that in mind, making a more generic implementation for filters would allow users that don't use Symfony to benefit from this feature as well.

There are meilisearch api clients in many languages, so those wouldn't benefit as well

[ToshY]

There are meilisearch api clients in many languages, so those wouldn't benefit as well

I was referring to those are using this (PHP) library without Symfony framework: Laravel, Laminas, no frameworks, etc.

---

In an attempt to convince you that it has a more righful place here, I've started tinkering on this myself yesterday and came up with a relatively easy setup to achieve what I'm looking for myself.

In short, by overriding/extending the existing classes Meilisearch\Client, Meilisearch\Endpoints\Indexes and Meilisearch\Endpoints\Delegates\HandlesIndex, I'm able to pass an additional array property $filters = [] to the constructor of the client (and subsequent method calls). So without the need of events/listeners, I can create a client and initialize it with certain filters of my choice, and if I want to remove the filter on the index for certain searches, I'll just call $index->removeFilter(...) to remove it, or call $index->addFilter(...) to add/replace one.

Starting with my sample project structure.

App/
├── Filter/
│   └── Meilisearch/
│       └── FreeTierFilter.php
└── Service/
    └── External/
        └── Request/
            └── Meilisearch/
                ├── Enum/
                │   └── MergeAction.php
                ├── Filter/
                │   ├── AbstractFilterInterface.php
                │   ├── ArrayFilter.php
                │   └── ScalarFilter.php
                ├── MeilisearchClient.php
                ├── MeilisearchHandlesIndexes.php
                └── MeilisearchIndexes.php

• The ./App/Filter/Meilisearch/FreeTierFilter.php denotes an application specific filter, e.g. FreeTierFilter.php
• The ./App/Service/External/Request/Meilisearch namespace denotes custom logic to override/extend existing meilisearch classes.

From top-to-bottom here comes the implementation of these classes.

./App/Filter/Meilisearch/FreeTierFilter.php

<?php

declare(strict_types=1);

namespace App\Filter\Meilisearch;

use App\Service\External\Request\Meilisearch\Enum\MergeAction;
use App\Service\External\Request\Meilisearch\Filter\ScalarFilter;
use Symfony\Bundle\SecurityBundle\Security;
use Symfony\Component\Security\Core\User\UserInterface;

class FreeTierFilter extends ScalarFilter
{
    public function __construct(
        private Security $security,
    ) {
    }

    public static function name(): string
    {
        return 'free_tier_filter';
    }

    public static function key(): string
    {
        return 'filter';
    }

    public function value()
    {
        $user = $this->security->getUser();

        if ($user instanceof UserInterface === true) {
            return;
        }

        return '(freeTier = 1)';
    }

    public static function comparison(): string
    {
        return 'AND';
    }

    public static function action(): MergeAction
    {
        return MergeAction::UNION_ACTION;
    }
}

As an example, I've added the Security from Symfony in the constructor and used in the value() method to check if the current user is authenticated or not. If the user is authenticated, all good! It will return (nothing) and not add anything. If it's however a guest user, it will add an filter with the value (freeTier = 1) with the comparison AND, giving the expression AND (freeTier = 1) that eventually be concatenated to the existing filter key (denoted by key method) of the search params.

./App/Service/External/Request/Meilisearch/Enum/MergeAction.php

<?php

declare(strict_types=1);

namespace App\Service\External\Request\Meilisearch\Enum;

enum MergeAction
{
    case UNION_ACTION;

    case REPLACE_ACTION;
}

The actions denote the ways the filters will be merged, either by "union" (for strings = concatenation; for arrays merge distinct) or "replace" (filter value takes precedence).

./App/Service/External/Request/Meilisearch/Filter/AbstractFilterInterface.php

<?php

declare(strict_types=1);

namespace App\Service\External\Request\Meilisearch\Filter;

use App\Service\External\Request\Meilisearch\Enum\MergeAction;

interface AbstractFilterInterface
{
    public static function name(): string;

    public static function key(): string;

    public function value();

    public static function action(): MergeAction;

    public function merge(mixed $value): mixed;
}

./App/Service/External/Request/Meilisearch/Filter/ArrayFilter.php

<?php

declare(strict_types=1);

namespace App\Service\External\Request\Meilisearch\Filter;

use App\Service\External\Request\Meilisearch\Enum\MergeAction;

abstract class ArrayFilter implements AbstractFilterInterface
{
    public function merge(mixed $value): mixed
    {
        return match ($this->action()) {
            MergeAction::UNION_ACTION => array_merge(
                array_intersect($value, $this->value()),
                array_diff($value, $this->value()),
                array_diff($this->value(), $value)
            ),
            MergeAction::REPLACE_ACTION => $this->value(),
        };
    }
}

./App/Service/External/Request/Meilisearch/Filter/ScalarFilter.php

<?php

declare(strict_types=1);

namespace App\Service\External\Request\Meilisearch\Filter;

use App\Service\External\Request\Meilisearch\Enum\MergeAction;

abstract class ScalarFilter implements AbstractFilterInterface
{
    abstract public static function comparison(): string;

    public function merge(mixed $value): mixed
    {
        $isNotEmpty = empty($this->value()) === false;
        return match ($this->action()) {
            MergeAction::UNION_ACTION => $isNotEmpty ? implode(
                sprintf(
                    ' %s ',
                    static::comparison(),
                ),
                [
                    $value,
                    $this->value(),
                ]
            ) : $value,
            MergeAction::REPLACE_ACTION => $this->value(),
        };
    }
}

./App/Service/External/Request/Meilisearch/MeilisearchClient.php

<?php

declare(strict_types=1);

namespace App\Service\External\Request\Meilisearch;

use Meilisearch\Endpoints\Delegates\HandlesDumps;
use Meilisearch\Endpoints\Delegates\HandlesKeys;
use Meilisearch\Endpoints\Delegates\HandlesMultiSearch;
use Meilisearch\Endpoints\Delegates\HandlesSystem;
use Meilisearch\Endpoints\Delegates\HandlesTasks;
use Meilisearch\Endpoints\Dumps;
use Meilisearch\Endpoints\Health;
use Meilisearch\Endpoints\Keys;
use Meilisearch\Endpoints\Stats;
use Meilisearch\Endpoints\Tasks;
use Meilisearch\Endpoints\TenantToken;
use Meilisearch\Endpoints\Version;
use Meilisearch\Http\Client as MeilisearchClientAdapter;
use Symfony\Component\DependencyInjection\Attribute\Autowire;
use Symfony\Component\HttpClient\Psr18Client;

final class MeilisearchClient
{
    use HandlesDumps;
    use MeilisearchHandlesIndex;
    use HandlesTasks;
    use HandlesKeys;
    use HandlesSystem;
    use HandlesMultiSearch;

    public function __construct(
        #[Autowire(env: 'MEILISEARCH_URL')]
        private readonly string $url,
        #[Autowire(env: 'MEILISEARCH_MASTER_KEY')]
        private readonly string $apiKey,
        private readonly array $filters,
    ) {
        $this->http = new MeilisearchClientAdapter(
            $this->url,
            $this->apiKey,
            new Psr18Client(),
        );
        $this->index = new MeilisearchIndexes($this->http, filters: $this->filters);
        $this->health = new Health($this->http);
        $this->version = new Version($this->http);
        $this->stats = new Stats($this->http);
        $this->tasks = new Tasks($this->http);
        $this->keys = new Keys($this->http);
        $this->dumps = new Dumps($this->http);
        $this->tenantToken = new TenantToken($this->http, $apiKey);
    }
}

./App/Service/External/Request/Meilisearch/MeilisearchHandlesIndexes.php

<?php

declare(strict_types=1);

namespace App\Service\External\Request\Meilisearch;

use Meilisearch\Contracts\IndexesQuery;
use Meilisearch\Contracts\IndexesResults;

trait MeilisearchHandlesIndex
{
    protected MeilisearchIndexes $index;

    public function getIndexes(IndexesQuery $options = null): IndexesResults
    {
        return $this->index->all($options ?? null);
    }

    public function getRawIndex(string $uid): array
    {
        return $this->index($uid)->fetchRawInfo();
    }

    public function index(string $uid): MeilisearchIndexes
    {
        return new MeilisearchIndexes($this->http, $uid, filters: $this->filters);
    }

    public function getIndex(string $uid): MeilisearchIndexes
    {
        return $this->index($uid)->fetchInfo();
    }

    public function deleteIndex(string $uid): array
    {
        return $this->index($uid)->delete();
    }

    public function createIndex(string $uid, array $options = []): array
    {
        return $this->index->create($uid, $options);
    }

    public function updateIndex(string $uid, array $options = []): array
    {
        return $this->index($uid)->update($options);
    }
}

./App/Service/External/Request/Meilisearch/MeilisearchIndexes.php

<?php

declare(strict_types=1);

namespace App\Service\External\Request\Meilisearch;

use App\Service\External\Request\Meilisearch\Filter\AbstractFilterInterface;
use Meilisearch\Contracts\Http;
use Meilisearch\Endpoints\Indexes;

final class MeilisearchIndexes extends Indexes
{
    protected string|null $uid = null;

    public function __construct(
        Http $http,
        $uid = null,
        $primaryKey = null,
        $createdAt = null,
        $updatedAt = null,
        private array $filters = [],
    ) {
        parent::__construct($http, $uid, $primaryKey, $createdAt, $updatedAt);

        $this->uid = $uid;
    }

    public function rawSearch(?string $query, array $searchParams = []): array
    {
        $parameters = array_merge(
            ['q' => $query],
            $searchParams
        );

        /** @var AbstractFilterInterface $filterItem */
        foreach ($this->filters as $filterItem) {
            $searchParamKey = $filterItem->key();
            if (in_array($searchParamKey, array_keys($parameters), true) === false) {
                continue;
            }

            $currentValue = $parameters[$searchParamKey];
            $parameters[$searchParamKey] = $filterItem->merge($currentValue);
        }

        $result = $this->http->post(self::PATH . '/' . $this->uid . '/search', $parameters);

        // patch to prevent breaking in laravel/scout getTotalCount method,
        // affects only Meilisearch >= v0.28.0.
        if (isset($result['estimatedTotalHits'])) {
            $result['nbHits'] = $result['estimatedTotalHits'];
        }

        return $result;
    }

    public function addFilter(AbstractFilterInterface $filter): void
    {
        $this->filters[$filter->name()] = $filter;
    }

    public function removeFilter(AbstractFilterInterface $filter): void
    {
        if (in_array($filter->name(), array_keys($this->filters), true) === false) {
            return;
        }

        unset($this->filters[$filter->name()]);
    }
}

This is where the magic happens. The rawSearch method is adapted to apply filters if given and apply them to the specific keys of the search params. The additional methods addFilter and removeFilter are added to easily add/remove filters when making searches on an index.

Now create an instance of new search client and pass the initial FreeTierFilter to the filters argument (example with Symfony config given below)

config/services.yaml

  App\Service\External\Request\Meilisearch\MeilisearchClient:
    $filters:
      free_tier_filter: '@App\Filter\Meilisearch\FreeTierFilter'

Now everything's setup to use the client.

class SearchController
    public function __construct(
        private readonly MeilisearchClient $meilisearchClient,
        private readonly FreeTierFilter $freeTierFilter,
    ) {
    }

   public function action() {
       $index = $this->meilisearchClient->getIndex('movies');

       // Less results as the `filter` key contains the additional statement ` AND (freeTier = 1)`.
       $result = $index->search(query: 'Jungle', searchParams: ['filter' => 'score > 8']);
        
       // Now remove the filter
       $index->removeFilter($this->freeTierFilter);

       // More results are returned as statement `(freeTier = 1)` will now no longer be added to the `filter` key.
       $result = $index->search(query: 'Jungle', searchParams: ['filter' => 'score > 8']);

       // Add the filter back and it's applied again
       $index->addFilter($this->freeTierFilter);
   }
}

---

In the example above, everything that was added to ./App/Service/External/Request/Meilisearch/ directory is based on the classes available from this library, which is not something Symfony specific.

The whole point of the example is to show that it's possible to create a generic implementation in this repo.

[curquiza]

Hello thank you @ToshY for opening the issue
I move the issue to the symfony repo 😊

[tacman]

Interesting. I put something together for being able to use ApiPlatform to filter on facets. It's complicated and not very elegant, but it works great.

#[ApiFilter(FacetsFieldSearchFilter::class, properties: ['marking', 'locale', 'countryCode'])]

When indexing, it searches the class attributes, and creates the settings that are passed to Meilisearch.

            $properties = $arguments['properties'];
            foreach ($properties as $property) {
                if (!array_key_exists($property, $settings)) {
                    $settings[$property] = [
                        'name' => $property,
                        'browsable' => false,
                        'sortable' => false,
                        'searchable' => false
                    ];
                }
                switch ($filter) {
                    case FacetsFieldSearchFilter::class:
                        $settings[$property]['browsable'] = true;
                        break;
                    case SortFilter::class:
                    case OrderFilter::class:
                        $settings[$property]['sortable'] = true;
                        break;

                    case SearchFilter::class:
                    case MeiliMultiFieldSearchFilter::class:
                    case RangeFilter::class:
                    case MultiFieldSearchFilter::class:
                        $settings[$property]['searchable'] = true;
                        break;
                }
            }
        }

And eventually after some more tweaking

        $index = $this->meiliService->getIndex($indexName, $primaryKey);
            $results = $index->updateSettings($settings)

It looks like schranz-search handles this more elegantly. The schema is interesting, but ugh, yet another schema to figure out. Schema is probably a better word than settings, and filter v. search is also confusing at least initially.