diff --git a/documentation/guides/mat-views.md b/documentation/guides/mat-views.md index ef06be9d..02649c09 100644 --- a/documentation/guides/mat-views.md +++ b/documentation/guides/mat-views.md @@ -55,10 +55,10 @@ As data grows in size, the performance of certain queries can degrade. Materialized views store the result of a `SAMPLE BY` or time-based `GROUP BY` query on disk, and keep it automatically up to date. -The refresh of a materialized view is `INCREMENTAL` and very efficient, and -using materialized views can offer 100x or higher query speedups. If you require -the lowest latency queries, for example, for charts and dashboards, use -materialized views! +The refresh of a materialized view is incremental and very efficient, and using +materialized views can offer 100x or higher query speedups. If you require the +lowest latency queries, for example, for charts and dashboards, use materialized +views! For a better understanding of what materialized views are for, read the [introduction to materialized views](/docs/concept/mat-views/) documentation. @@ -106,7 +106,7 @@ If you are unfamiliar with the OHLC concept, please see our ```questdb-sql title="trades_OHLC_15m DDL" CREATE MATERIALIZED VIEW 'trades_OHLC_15m' -WITH BASE 'trades' REFRESH INCREMENTAL AS +WITH BASE 'trades' REFRESH IMMEDIATE AS SELECT timestamp, symbol, first(price) AS open, @@ -124,9 +124,9 @@ In this example: 2. The base table is `trades` - This is the data source, and will trigger incremental refresh when new data is written. -3. The refresh strategy is `INCREMENTAL` - - The data is automatically refreshed and incrementally written; efficient, - fast, low maintenance. +3. The refresh strategy is `IMMEDIATE` + - The data is automatically refreshed and incrementally written after a base + table transaction occurs; efficient, fast, low maintenance. 4. The `SAMPLE BY` query contains two key column (`timestamp`, `symbol`) and five aggregates (`first`, `max`, `min`, `last`, `price`) calculated in `15m` time buckets. @@ -172,13 +172,61 @@ will not trigger any sort of refresh. #### Refresh strategies -Currently, only `INCREMENTAL` refresh is supported. This strategy incrementally -updates the view when new data is inserted into the base table. This means that -only new data is written to the view, so there is minimal write overhead. +The `IMMEDIATE` refresh strategy incrementally updates the view when new data is +inserted into the base table. This means that only new data is written to the +view, so there is minimal write overhead. Upon creation, or when the view is invalidated, a full refresh will occur, which rebuilds the view from scratch. +Other than `IMMEDIATE` refresh, QuestDB supports `MANUAL` and timer +(`EVERY `) strategies for materialized views. Manual strategy means +that to refresh the view, you need to run the +[`REFRESH` SQL](/docs/reference/sql/refresh-mat-view/) explicitly. In case of +timer-based refresh the view is refreshed periodically, at the specified +interval. + +The refresh strategy of an existing view can be changed any time with the +[`ALTER SET REFRESH`](/docs/reference/sql/alter-mat-view-set-refresh/) command. + +## Period materialized views + +In certain use cases, like storing trading day information, the data becomes +available at fixed time intervals. In this case, `PERIOD` variant of +materialized views can be used: + +```questdb-sql title="Period materialized view" +CREATE MATERIALIZED VIEW trades_daily_prices +REFRESH PERIOD (LENGTH 1d TIME ZONE 'Europe/London' DELAY 2h) AS +SELECT + timestamp, + symbol, + avg(price) AS avg_price +FROM trades +SAMPLE BY 1d; +``` + +Refer to the following +[documentation page](/docs/reference/sql/create-mat-view/#period-materialized-views) +to learn more on period materialized views. + +## Initial refresh + +As soon as a materialized view is created an asynchronous refresh is started. In +situations when this is not desirable, `DEFERRED` keyword can be specified along +with the refresh strategy: + +```questdb-sql title="Deferred manual refresh" +CREATE MATERIALIZED VIEW trades_daily_prices +REFRESH MANUAL DEFERRED AS +... +``` + +The `DEFERRED` keyword can be specified for any refresh strategy. Refer to the +following +[documentation page](/docs/reference/sql/create-mat-view/#initial-refresh) to +learn more on the keyword. + #### SAMPLE BY Materialized views are populated using `SAMPLE BY` or time-based `GROUP BY` @@ -357,7 +405,7 @@ useful. ## Limitations - Not all `SAMPLE BY` syntax is supported, for example, `FILL`. -- `INCREMENTAL` refresh is only triggered by inserts into the `base` table, not +- `IMMEDIATE` refresh is only triggered by inserts into the `base` table, not join tables. ## LATEST ON materialized views @@ -555,15 +603,6 @@ partitioning capabilities. ### Refresh mechanism -:::note - -Currently, QuestDB only supports **incremental refresh** for materialized views. - -Future releases will include additional refresh types, such as time-interval and -manual refreshes. - -::: - Unlike regular views, which recompute their results at query time, materialized views in QuestDB are incrementally refreshed as new data is added to the base table. This approach ensures that only the **relevant time slices** of the view diff --git a/documentation/reference/function/meta.md b/documentation/reference/function/meta.md index cd8dca8e..4056b1ba 100644 --- a/documentation/reference/function/meta.md +++ b/documentation/reference/function/meta.md @@ -518,8 +518,8 @@ materialized_views(); | view_name | refresh_type | base_table_name | last_refresh_start_timestamp | last_refresh_finish_timestamp | view_sql | view_table_dir_name | invalidation_reason | view_status | refresh_base_table_txn | base_table_txn | refresh_limit_value | refresh_limit_unit | timer_start | timer_interval_value | timer_interval_unit | |------------------|--------------|-----------------|------------------------------|-------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------|---------------------|-------------|------------------------|----------------|---------------------|--------------------|-------------|----------------------|---------------------| -| trades_OHLC_15m | incremental | trades | 2025-05-30T16:40:37.562421Z | 2025-05-30T16:40:37.568800Z | SELECT timestamp, symbol, first(price) AS open, max(price) as high, min(price) as low, last(price) AS close, sum(amount) AS volume FROM trades SAMPLE BY 15m | trades_OHLC_15m~27 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null | -| trades_latest_1d | incremental | trades | 2025-05-30T16:40:37.554274Z | 2025-05-30T16:40:37.562049Z | SELECT timestamp, symbol, side, last(price) AS price, last(amount) AS amount, last(timestamp) as latest FROM trades SAMPLE BY 1d | trades_latest_1d~28 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null | +| trades_OHLC_15m | immediate | trades | 2025-05-30T16:40:37.562421Z | 2025-05-30T16:40:37.568800Z | SELECT timestamp, symbol, first(price) AS open, max(price) as high, min(price) as low, last(price) AS close, sum(amount) AS volume FROM trades SAMPLE BY 15m | trades_OHLC_15m~27 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null | +| trades_latest_1d | immediate | trades | 2025-05-30T16:40:37.554274Z | 2025-05-30T16:40:37.562049Z | SELECT timestamp, symbol, side, last(price) AS price, last(amount) AS amount, last(timestamp) as latest FROM trades SAMPLE BY 1d | trades_latest_1d~28 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null | ## version/pg_catalog.version diff --git a/documentation/reference/sql/alter-mat-view-set-refresh-start.md b/documentation/reference/sql/alter-mat-view-set-refresh-start.md deleted file mode 100644 index be27d210..00000000 --- a/documentation/reference/sql/alter-mat-view-set-refresh-start.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -title: ALTER MATERIALIZED VIEW SET REFRESH START -sidebar_label: SET REFRESH START -description: - ALTER MATERIALIZED VIEW SET REFRESH START SQL keyword reference documentation. ---- - -Changes a materialized view's refresh to run on a schedule. - -## Syntax - -![Flow chart showing the syntax of ALTER MATERIALIZED VIEW SET REFRESH START command](/images/docs/diagrams/alterMatViewSetRefreshStart.svg) - -## Description - -Sometimes, the view may not need to be updated eagerly. For example, perhaps the data is only queried every five minutes. - -In this circumstance, you can defer updating the view in small pieces, and instead let it be updated in a larger -incremental write every five minutes. - -The schedule is defined using a start time and then a timing unit, with a minimum of `1m`. - -Each triggered refresh is itself incremental, and will only consider data since the last refresh. - -The unit follows the same format as [SAMPLE BY](/docs/reference/sql/sample-by/). - -## Examples - -```questdb-sql -ALTER MATERIALIZED VIEW trades_hourly_prices SET REFRESH START '2025-05-30T00:00:00Z' EVERY '1h'; -``` diff --git a/documentation/reference/sql/alter-mat-view-set-refresh.md b/documentation/reference/sql/alter-mat-view-set-refresh.md new file mode 100644 index 00000000..59f5867b --- /dev/null +++ b/documentation/reference/sql/alter-mat-view-set-refresh.md @@ -0,0 +1,38 @@ +--- +title: ALTER MATERIALIZED VIEW SET REFRESH +sidebar_label: SET REFRESH +description: + ALTER MATERIALIZED VIEW SET REFRESH SQL keyword reference documentation. +--- + +Changes a materialized view's refresh strategy and parameters. + +## Syntax + +![Flow chart showing the syntax of ALTER MATERIALIZED VIEW SET REFRESH command](/images/docs/diagrams/alterMatViewSetRefresh.svg) + +## Description + +Sometimes, the view's refresh strategy and its parameters may need to be changed. +Say, you may want to change the view to be timer refreshed instead of immediate +refresh. + +The `REFRESH` follows the same format as [CREATE MATERIALIZED VIEW](/docs/reference/sql/create-mat-view/). + +## Examples + +```questdb-sql +ALTER MATERIALIZED VIEW trades_hourly_prices SET REFRESH EVERY '1h'; +``` + +```questdb-sql +ALTER MATERIALIZED VIEW trades_hourly_prices SET REFRESH PERIOD (LENGTH 1d DELAY 1h); +``` + +```questdb-sql +ALTER MATERIALIZED VIEW trades_hourly_prices SET REFRESH IMMEDIATE; +``` + +```questdb-sql +ALTER MATERIALIZED VIEW trades_hourly_prices SET REFRESH MANUAL; +``` diff --git a/documentation/reference/sql/create-mat-view.md b/documentation/reference/sql/create-mat-view.md index 7b52b659..169c30df 100644 --- a/documentation/reference/sql/create-mat-view.md +++ b/documentation/reference/sql/create-mat-view.md @@ -91,38 +91,146 @@ with the designated timestamp as the grouping key. ::: -## Alternate refresh modes +## Alternative refresh strategies -By default, QuestDB will incrementally refresh the view each time new data is written to the base table. +With the default `IMMEDIATE` refresh strategy, QuestDB will incrementally +refresh the view each time new data is written to the base table. If your data +is written rapidly in small transactions, this will trigger additional small +writes to the view. -If your data is written rapidly in small transactions, this will trigger additional small writes to the view. +Instead, you can use timer-based refresh, which trigger an incremental refresh +after certain time intervals: -Instead, you can specify a refresh schedule, which trigger and incremental refresh after certain time intervals: +```questdb-sql +CREATE MATERIALIZED VIEW price_1h +REFRESH EVERY 1h START '2025-05-30T00:00:00.000000Z' TIME ZONE 'Europe/Berlin' +AS ... +``` + +In this example, the view will start refreshing from the specified timestamp in +Berlin time zone on an hourly schedule. The refresh itself will still be +incremental, but will no longer be triggered on every new insert. You can omit +the `START ` and `TIME ZONE ` clauses in order to just +start refreshing from `now`. + +:::tip + +The minimum timed interval is one minute (`1m`). If you need to refresh faster +than this, please use the default incremental refresh. + +::: + +In case you want to be in full control of when the incremental refresh happens, +you can use `MANUAL` refresh: ```questdb-sql -CREATE MATERIALIZED VIEW price_1h REFRESH START '2025-05-30T00:00:00.000000Z' EVERY 1h AS ... +CREATE MATERIALIZED VIEW price_1h +REFRESH MANUAL +AS ... ``` -In this example, the view will start refreshing from the specified timestamp on an hourly schedule. +Manual strategy means that to refresh the view, you need to run the +[`REFRESH` SQL](/docs/reference/sql/refresh-mat-view/) explicitly. -The refresh itself will still be incremental, but will no longer be triggered on every new insert. +For all these strategies, the refresh itself stays incremental, i.e. the +materialized view is only updated for base table time intervals that received +modifications since the previous refresh. -You can omit the `START ` clause in order to just start refreshing from `now`. +## Period materialized views +In certain use cases, like storing trading day information, the data becomes +available at fixed time intervals. In this case, `PERIOD` variant of +materialized views can be used: -:::tip +```questdb-sql title="Period materialized view" +CREATE MATERIALIZED VIEW trades_hourly_prices +REFRESH PERIOD (LENGTH 1d TIME ZONE 'Europe/London' DELAY 2h) AS +SELECT + timestamp, + symbol, + avg(price) AS avg_price +FROM trades +SAMPLE BY 1h; +``` -The minimum timed interval is one minute (`1m`). If you need to refresh faster than this, please use -the default incremental refresh. +The `PERIOD` clause above defines an in-flight time interval (period) in the +`trades_daily_prices` materialized view that will not receive data until it +finishes. In this example, the interval is one day (`LENGTH 1d`) in London time +zone. The `DELAY 2h` clause here means that the data for the trading day may +have 2 hour lag until it's fully written. So, in our example the current +in-flight period in the view is considered complete and gets refreshed +automatically each day at 2AM, London time. Since the default `IMMEDIATE` +refresh strategy is used, all writes to older, complete periods in the base +table lead to an immediate and asynchronous refresh in the view once the +transaction is committed. + +Period materialized views can be used with any supported refresh strategy, not +only with the `IMMEDIATE` one. For instance, they can be configured for +timer-based refresh: + +```questdb-sql title="Period materialized view with timer refresh" +CREATE MATERIALIZED VIEW trades_hourly_prices +REFRESH EVERY 10m PERIOD (LENGTH 1d TIME ZONE 'Europe/London' DELAY 2h) AS +... +``` -::: +Here, the `PERIOD` refresh still takes place once a period completes, but +refreshes for older rows take place each 10 minutes. + +Finally, period materialized views can be configure for manual refresh: + +```questdb-sql title="Period materialized view with timer refresh" +CREATE MATERIALIZED VIEW trades_hourly_prices +REFRESH MANUAL PERIOD (LENGTH 1d TIME ZONE 'Europe/London' DELAY 2h) AS +... +``` + +The only way to refresh data on such a materialized view is to run +[`REFRESH` SQL](/docs/reference/sql/refresh-mat-view/) explicitly. When run, +`REFRESH` statement will refresh incrementally all recently completed periods, +as well as all time intervals touched by the recent write transactions. + +## Initial refresh + +As soon as a materialized view is created an asynchronous refresh is started. In +situations when this is not desirable, `DEFERRED` keyword can be specified along +with the refresh strategy: + +```questdb-sql title="Deferred manual refresh" +CREATE MATERIALIZED VIEW trades_hourly_prices +REFRESH MANUAL DEFERRED AS +... +``` + +In the above example, the view has manual refresh strategy and it does not +refresh after creation. It will only refresh when you run the +[`REFRESH` SQL](/docs/reference/sql/refresh-mat-view/) explicitly. + +The `DEFERRED` keyword can be also specified for `IMMEDIATE` and timer-based +refresh strategies. Here is an example: + +```questdb-sql title="Deferred timer refresh" +CREATE MATERIALIZED VIEW trades_hourly_prices +REFRESH EVERY 1h DEFERRED START '2026-01-01T00:00:00' AS +... +``` + +In such cases, the view will be refreshed only when the corresponding event +occurs: + +- After the next base table transaction in case of `IMMEDIATE` refresh strategy. +- At the next trigger time in case of timer-based refresh strategy. + +Once a materialized view is created, its refresh strategy can be changed any time +with the [`ALTER SET REFRESH`](/docs/reference/sql/alter-mat-view-set-refresh/) +command. ## Base table -Incrementally refreshed views require that the base table is specified, so that -the server refreshes the materialized view each time the base table is updated. -When creating a materialized view that queries multiple tables, you must specify -one of them as the base table. +Materialized views require that the base table is specified, so that the last +base table transaction number can be saved and later on checked by the +incremental refresh. When creating a materialized view that queries multiple +tables, you must specify one of them as the base table. ```questdb-sql title="Hourly materialized view with LT JOIN" CREATE MATERIALIZED VIEW trades_ext_hourly_prices @@ -195,6 +303,39 @@ CREATE MATERIALIZED VIEW trades_hourly_prices AS ( ) PARTITION BY DAY TTL 7 DAYS; ``` +```questdb-sql title="Creating a materialized view with one day period" +CREATE MATERIALIZED VIEW trades_hourly_prices +REFRESH PERIOD (LENGTH 1d TIME ZONE 'Europe/London' DELAY 2h) AS +SELECT + timestamp, + symbol, + avg(price) AS avg_price +FROM trades +SAMPLE BY 1h; +``` + +```questdb-sql title="Creating a materialized view with timer refresh each 10 minutes" +CREATE MATERIALIZED VIEW trades_hourly_prices +REFRESH EVERY 10m START '2025-06-18T00:00:00.000000000' AS +SELECT + timestamp, + symbol, + avg(price) AS avg_price +FROM trades +SAMPLE BY 1h; +``` + +```questdb-sql title="Creating a materialized view with manual refresh" +CREATE MATERIALIZED VIEW trades_hourly_prices +REFRESH MANUAL AS +SELECT + timestamp, + symbol, + avg(price) AS avg_price +FROM trades +SAMPLE BY 1h; +``` + ## IF NOT EXISTS An optional `IF NOT EXISTS` clause may be added directly after the @@ -220,7 +361,7 @@ Materialized view names follow the When a user creates a new materialized view, they are automatically assigned all materialized view level permissions with the `GRANT` option for that view. This -behaviour can can be overridden using `OWNED BY`. +behavior can can be overridden using `OWNED BY`. If the `OWNED BY` clause is used, the permissions instead go to the user, group, or service account named in that clause. diff --git a/documentation/sidebars.js b/documentation/sidebars.js index 5cf993eb..30e93c14 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -230,7 +230,7 @@ module.exports = { items: [ "reference/sql/alter-mat-view-change-symbol-capacity", "reference/sql/alter-mat-view-set-refresh-limit", - "reference/sql/alter-mat-view-set-refresh-start", + "reference/sql/alter-mat-view-set-refresh", "reference/sql/alter-mat-view-set-ttl", "reference/sql/alter-mat-view-resume-wal", ], diff --git a/static/images/docs/diagrams/.railroad b/static/images/docs/diagrams/.railroad index e88bae5b..70ca3b6b 100644 --- a/static/images/docs/diagrams/.railroad +++ b/static/images/docs/diagrams/.railroad @@ -363,7 +363,10 @@ enableDedup createMatViewDef ::= 'CREATE' 'MATERIALIZED' 'VIEW' ('IF' 'NOT' 'EXISTS')? viewName ('WITH BASE' baseTableName)? - ('REFRESH' ( 'INCREMENTAL' | ( ('START' timestamp)? 'EVERY' interval) ))? + ( + 'REFRESH' ((('IMMEDIATE' | 'MANUAL') ('DEFERRED')?) | ('EVERY' interval ('DEFERRED')? ('START' timestamp)? ('TIME' 'ZONE' timezone)?) )? + ('PERIOD' '(' 'LENGTH' length ('TIME' 'ZONE' timezone)? ('DELAY' delay)? ')')? + )? 'AS' ('(')? (query) @@ -393,8 +396,10 @@ alterMatViewSetTtl alterMatViewSetRefreshLimit ::= 'ALTER' 'MATERIALIZED' 'VIEW' viewName 'SET' 'REFRESH' 'LIMIT' n ('HOURS' | 'DAYS' | 'WEEKS' | 'MONTHS' | 'YEARS') -alterMatViewSetRefreshStart -::= 'ALTER' 'MATERIALIZED' 'VIEW' viewName 'SET' 'REFRESH' ( ('START' timestamp)? 'EVERY' interval ) +alterMatViewSetRefresh +::= 'ALTER' 'MATERIALIZED' 'VIEW' viewName 'SET' 'REFRESH' + (('IMMEDIATE' | 'MANUAL') | ('EVERY' interval ('DEFERRED')? ('START' timestamp)? ('TIME' 'ZONE' timezone)?) )? + ('PERIOD' '(' 'LENGTH' length ('TIME' 'ZONE' timezone)? ('DELAY' delay)? ')')? refreshMatView ::= 'REFRESH' 'MATERIALIZED' 'VIEW' viewName ('FULL' | 'INCREMENTAL' | ('INTERVAL' 'FROM' fromTimestamp 'TO' toTimestamp)) diff --git a/static/images/docs/diagrams/alterMatViewSetRefresh.svg b/static/images/docs/diagrams/alterMatViewSetRefresh.svg new file mode 100644 index 00000000..ed880ee4 --- /dev/null +++ b/static/images/docs/diagrams/alterMatViewSetRefresh.svg @@ -0,0 +1,107 @@ + + + + + + + + + ALTER + + + MATERIALIZED + + + VIEW + + + viewName + + SET + + + REFRESH + + + IMMEDIATE + + + MANUAL + + + EVERY + + + interval + + DEFERRED + + + START + + + timestamp + + TIME + + + ZONE + + + timezone + + PERIOD + + + ( + + + LENGTH + + + length + + TIME + + + ZONE + + + timezone + + DELAY + + + delay + + ) + + + + \ No newline at end of file diff --git a/static/images/docs/diagrams/alterMatViewSetRefreshStart.svg b/static/images/docs/diagrams/alterMatViewSetRefreshStart.svg deleted file mode 100644 index fa3437a9..00000000 --- a/static/images/docs/diagrams/alterMatViewSetRefreshStart.svg +++ /dev/null @@ -1,63 +0,0 @@ - - - - - - - - - ALTER - - - MATERIALIZED - - - VIEW - - - viewName - - SET - - - REFRESH - - - START - - - timestamp - - EVERY - - - interval - - - \ No newline at end of file diff --git a/static/images/docs/diagrams/createMatViewDef.svg b/static/images/docs/diagrams/createMatViewDef.svg index d475a103..4db3b9c6 100644 --- a/static/images/docs/diagrams/createMatViewDef.svg +++ b/static/images/docs/diagrams/createMatViewDef.svg @@ -1,4 +1,4 @@ - + \ No newline at end of file