Use jsonb when using postgresql #671

DanielBadura · 2025-01-06T20:21:24Z

We should use jsonb when the user is using postgresql. This give the user better query capabilities on the json fields in the store.

When this PR is released doctrine/dbal#6693 the migration tooling should pick this change up. So the user does not need to update the table themselfes besides of the normal call to bin/console do:mi:di & bin/console do:mi:mi.

github-actions · 2025-01-06T20:31:59Z

Hello 👋

here is the most recent benchmark result:

SimpleSetupBench
================

+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
|                                        | time (kde mode)                                     | memory                                     |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| subject                                | Tag: <current>     | Tag: base          | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| benchLoad1Event ()                     | 1.002ms (±0.00%)   | 1.095ms (±0.00%)   | -8.45%    | 34.629mb        | 34.637mb   | -0.02%      |
| benchLoad10000Events ()                | 51.643ms (±0.00%)  | 49.726ms (±0.00%)  | +3.86%    | 34.629mb        | 34.637mb   | -0.02%      |
| benchSave1Event ()                     | 1.373ms (±0.00%)   | 1.114ms (±0.00%)   | +23.25%   | 34.128mb        | 34.128mb   | 0.00%       |
| benchSave10000Events ()                | 246.649ms (±0.00%) | 225.547ms (±0.00%) | +9.36%    | 34.128mb        | 34.128mb   | 0.00%       |
| benchSave10000Aggregates ()            | 9.428s (±0.00%)    | 8.988s (±0.00%)    | +4.89%    | 34.128mb        | 34.128mb   | 0.00%       |
| benchSave10000AggregatesTransaction () | 4.960s (±0.00%)    | 4.954s (±0.00%)    | +0.13%    | 34.128mb        | 34.128mb   | 0.00%       |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+

SnapshotsBench
==============

+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
|                                        | time (kde mode)                                     | memory                                     |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| subject                                | Tag: <current>     | Tag: base          | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| benchLoad10000EventsMissingSnapshot () | 52.693ms (±0.00%)  | 51.353ms (±0.00%)  | +2.61%    | 34.132mb        | 34.132mb   | 0.00%       |
| benchLoad10000Events ()                | 985.000μs (±0.00%) | 974.900μs (±0.00%) | +1.04%    | 34.132mb        | 34.132mb   | 0.00%       |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+

SubscriptionEngineBatchBench
============================

+---------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+
|                           | time (kde mode)                                   | memory                                     |
+---------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+
| subject                   | Tag: <current>    | Tag: base         | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+---------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+
| benchHandle10000Events () | 77.425ms (±0.00%) | 78.550ms (±0.00%) | -1.43%    | 34.596mb        | 34.596mb   | 0.00%       |
+---------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+

SubscriptionEngineBench
=======================

+---------------------------+-----------------+-----------------+-----------+-----------------+------------+-------------+
|                           | time (kde mode)                               | memory                                     |
+---------------------------+-----------------+-----------------+-----------+-----------------+------------+-------------+
| subject                   | Tag: <current>  | Tag: base       | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+---------------------------+-----------------+-----------------+-----------+-----------------+------------+-------------+
| benchHandle10000Events () | 3.223s (±0.00%) | 3.252s (±0.00%) | -0.88%    | 46.745mb        | 46.745mb   | 0.00%       |
+---------------------------+-----------------+-----------------+-----------+-----------------+------------+-------------+

PersonalDataBench
=================

+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
|                                        | time (kde mode)                                     | memory                                     |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| subject                                | Tag: <current>     | Tag: base          | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| benchLoad1Event ()                     | 958.800μs (±0.00%) | 939.400μs (±0.00%) | +2.07%    | 35.258mb        | 35.258mb   | 0.00%       |
| benchLoad10000Events ()                | 91.978ms (±0.00%)  | 85.902ms (±0.00%)  | +7.07%    | 35.258mb        | 35.258mb   | 0.00%       |
| benchSave1Event ()                     | 1.609ms (±0.00%)   | 1.706ms (±0.00%)   | -5.67%    | 35.258mb        | 35.258mb   | 0.00%       |
| benchSave10000Events ()                | 292.028ms (±0.00%) | 260.288ms (±0.00%) | +12.19%   | 35.259mb        | 35.259mb   | 0.00%       |
| benchSave10000Aggregates ()            | 13.188s (±0.00%)   | 13.375s (±0.00%)   | -1.39%    | 35.258mb        | 35.258mb   | 0.00%       |
| benchSave10000AggregatesTransaction () | 9.227s (±0.00%)    | 9.186s (±0.00%)    | +0.44%    | 35.759mb        | 35.759mb   | 0.00%       |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+

SimpleSetupStreamStoreBench
===========================

+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
|                                        | time (kde mode)                                     | memory                                     |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| subject                                | Tag: <current>     | Tag: base          | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| benchLoad1Event ()                     | 980.200μs (±0.00%) | 1.036ms (±0.00%)   | -5.34%    | 35.155mb        | 35.155mb   | 0.00%       |
| benchLoad10000Events ()                | 60.835ms (±0.00%)  | 58.122ms (±0.00%)  | +4.67%    | 35.155mb        | 35.155mb   | 0.00%       |
| benchSave1Event ()                     | 1.044ms (±0.00%)   | 1.416ms (±0.00%)   | -26.27%   | 35.155mb        | 35.155mb   | 0.00%       |
| benchSave10000Events ()                | 335.346ms (±0.00%) | 301.169ms (±0.00%) | +11.35%   | 35.155mb        | 35.155mb   | 0.00%       |
| benchSave10000Aggregates ()            | 9.177s (±0.00%)    | 9.457s (±0.00%)    | -2.97%    | 35.155mb        | 35.155mb   | 0.00%       |
| benchSave10000AggregatesTransaction () | 5.198s (±0.00%)    | 5.168s (±0.00%)    | +0.58%    | 35.155mb        | 35.155mb   | 0.00%       |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+

SplitStreamBench
================

+-------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
|                         | time (kde mode)                                     | memory                                     |
+-------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| subject                 | Tag: <current>     | Tag: base          | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+-------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| benchLoad10000Events () | 4.451ms (±0.00%)   | 4.418ms (±0.00%)   | +0.74%    | 37.434mb        | 37.434mb   | 0.00%       |
| benchSave10000Events () | 374.001ms (±0.00%) | 344.186ms (±0.00%) | +8.66%    | 37.507mb        | 37.506mb   | +0.00%      |
+-------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+

This comment gets update everytime a new commit comes in!

… when using jsonb.

DanielBadura · 2025-01-06T21:03:53Z

src/Subscription/ThrowableToErrorContextTransformer.php

@@ -56,6 +56,10 @@ private static function transformThrowable(Throwable $error): array
     */
    private static function transformTrace(array $trace): array
    {
+        if (array_key_exists('class', $trace) && is_string($trace['class'])) {
+            $trace['class'] = str_replace("\x00", '', $trace['class']);


Any better suggestions @DavidBadura ?

Shouldn't that be its own PR?

Right now, this is not doing any harm. When we adjust the table type to jsonb Postgres checks if the json is valid and doesn't like null bytes in there.

I don't think that this must be in a separate bugfix PR but I could extract it. The only thing is if the user already has null bytes in the table. Then he would need to remove them. Or do you have a case in mind?

DanielBadura · 2025-01-16T18:52:53Z

New version for dbal was made which makes this change easier for our users.

Still need to evaluate if this is worth it due to slower write speed compared to the only limited gain for querying the eventstore.

Use jsonb when using postgresql

dd53bd1

Filter null bytes from trace due to anon class. This leads to a crash…

fac8d7c

… when using jsonb.

DanielBadura commented Jan 6, 2025

View reviewed changes

Fix cs

f6d5520

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use jsonb when using postgresql #671

Use jsonb when using postgresql #671

DanielBadura commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

DanielBadura Jan 6, 2025

DavidBadura Jan 7, 2025

DanielBadura Jan 7, 2025

DanielBadura commented Jan 16, 2025

Use jsonb when using postgresql #671

Are you sure you want to change the base?

Use jsonb when using postgresql #671

Conversation

DanielBadura commented Jan 6, 2025 • edited Loading

github-actions bot commented Jan 6, 2025 • edited Loading

DanielBadura Jan 6, 2025

Choose a reason for hiding this comment

DavidBadura Jan 7, 2025

Choose a reason for hiding this comment

DanielBadura Jan 7, 2025

Choose a reason for hiding this comment

DanielBadura commented Jan 16, 2025

DanielBadura commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading