Skip to content

9.0.0 release roundup #207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion documentation/concept/replication.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,8 @@ timeline
Currently : AWS S3
: Azure Blob Store
: NFS Filesystem
: Google Cloud Storage
Next-up : HDFS
Later on : Google Cloud Storage
```

Something missing? Want to see it sooner? [Contact us](/enterprise/contact)!
Expand All @@ -109,6 +109,12 @@ is:
An example of a replication object store configuration using NFS is:
`replication.object.store=fs::root=/mnt/nfs_replication/final;atomic_write_dir=/mnt/nfs_replication/scratch;`

An example of a replication object store configuration using GCS is:

`replication.object.store=gcs::bucket=<bucket name here>;root=/;credential=<base64 encoded key>;`

For `GCS`, you can also use `credential_path` to set a key-file location.

See the [Replication setup guide](/docs/operations/replication) for direct
examples.

Expand Down
28 changes: 28 additions & 0 deletions documentation/operations/replication.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,34 @@ With your values, skip to the
[Setup database replication](/docs/operations/replication/#setup-database-replication)
section.

### Google GCP GCS

First, create a new Google Cloud Storage (GCS) bucket, most likely with `Public access: Not public`.

Then create a new service account, and give it read-write permissions for the bucket. The simplest
role is `Storage Admin`, but you may set up more granular permissions as needed.

Create a new private key for this user and download it in `JSON` format. Then encode this key as `Base64`.

If you are on Linux, you can `cat` the file and pass it to `base64`:

```
cat <key>.json | base64
```

Then construct the connection string:

```
replication.object.store=gcs::bucket=<bucket name here>;root=/;credential=<base64 encoded key>;
```

If you do not want to put the credentials directly in the connection string, you can swap the `credential`
key for `credential_path`, and give it a path to the key-file.

With your values, continue to the
[Setup database replication](/docs/operations/replication/#setup-database-replication)
section.

## Setup database replication

Set the following changes in their respective `server.conf` files:
Expand Down
210 changes: 210 additions & 0 deletions documentation/reference/function/row-generator.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,3 +94,213 @@ use the same seed in long_sequence.
| ------------------ |
| 0.8251337821991485 |
| 0.2714941145110299 |

## generate_series

Rather than generating a fixed number of entries, `generate_series` can instead be used
to generate entries within a bounded range.

This function supports `LONG` and `DOUBLE` generation. There is also a `TIMESTAMP` generating version,
which can be found [here](/documentation/reference/function/timestamp-generator.md)

The `start` and `end` values are interchangeable, and a negative `step` value can be used to
obtain the series in reverse order.

The series is inclusive on both ends.

**Arguments:**

`generate_series(start_long, end_long, step_long)` - generates a series of longs.

`generate_series(start_double, end_double, step_double)` - generates a series of doubles.


**Return value:**

Return value type is `times---
title: Row generator
sidebar_label: Row generator
description: Row generator function reference documentation.
---

The `long_sequence()` function may be used as a row generator to create table
data for testing. Basic usage of this function involves providing the number of
iterations required. Deterministic pseudo-random behavior can be achieved by
providing seed values when calling the function.

This function is commonly used in combination with
[random generator functions](/docs/reference/function/random-value-generator/)
to produce mock data.

## long_sequence

- `long_sequence(iterations)` - generates rows
- `long_sequence(iterations, seed1, seed2)` - generates rows deterministically

**Arguments:**

-`iterations`: is a `long` representing the number of rows to generate. -`seed1`
and `seed2` are `long64` representing both parts of a `long128` seed.

### Row generation

The `long_sequence()` function can be used to generate very large datasets for
testing e.g. billions of rows.

`long_sequence(iterations)` is used to:

- Generate a number of rows defined by `iterations`.
- Generate a column `x:long` of monotonically increasing long integers starting
from 1, which can be accessed for queries.

### Random number seed

When `long_sequence` is used conjointly with
[random generators](/docs/reference/function/random-value-generator/), these
values are usually generated at random. The function supports a seed to be
passed in order to produce deterministic results.

:::note

Deterministic procedural generation makes it easy to test on vasts amounts of
data without actually moving large files around across machines. Using the same
seed on any machine at any time will consistently produce the same results for
all random functions.

:::

**Examples:**

```questdb-sql title="Generating multiple rows"
SELECT x, rnd_double()
FROM long_sequence(5);
```

| x | rnd_double |
| --- | ------------ |
| 1 | 0.3279246687 |
| 2 | 0.8341038236 |
| 3 | 0.1023834675 |
| 4 | 0.9130602021 |
| 5 | 0.718276777 |

```questdb-sql title="Accessing row_number using the x column"
SELECT x, x*x
FROM long_sequence(5);
```

| x | x\*x |
| --- | ---- |
| 1 | 1 |
| 2 | 4 |
| 3 | 9 |
| 4 | 16 |
| 5 | 25 |

```questdb-sql title="Using with a seed"
SELECT rnd_double()
FROM long_sequence(2,128349234,4327897);
```

:::note

The results below will be the same on any machine at any time as long as they
use the same seed in long_sequence.

:::

| rnd_double |
| ------------------ |
| 0.8251337821991485 |
| 0.2714941145110299 |

## generate_series

Rather than generating a fixed number of entries, `generate_series` can instead be used
to generate entries within a bounded range.

This function supports `LONG` and `DOUBLE` generation. There is also a `TIMESTAMP` generating version,
which can be found [here](/documentation/reference/function/timestamp-generator.md)

The `start` and `end` values are interchangeable, and a negative `step` value can be used to
obtain the series in reverse order.

The series is inclusive on both ends.

The final argument is optional, and defaults to `1`.

As a pseudo-table, the function can be called in isolation (`generate_series()`), or as
part of a select (`SELECT * FROM generate_series()`).

**Arguments:**

`generate_series(start_long, end_long, step_long)` - generates a series of longs.

`generate_series(start_double, end_double, step_double)` - generates a series of doubles.


**Return value:**

Return value type is `long` or `double`.

**Examples:**

```questdb-sql title="fwd long generation" demo
generate_series(-3, 3, 1);
-- or
generate_series(-3, 3);
```

| generate_series |
| --------------- |
| -3 |
| -2 |
| -1 |
| 0 |
| 1 |
| 2 |
| 3 |

```questdb-sql title="bwd long generation" demo
generate_series(3, -3, -1);
```

| generate_series |
| --------------- |
| 3 |
| 2 |
| 1 |
| 0 |
| -1 |
| -2 |
| -3 |

```questdb-sql title="fwd double generation" demo
generate_series(-3d, 3d, 1d);
-- or
generate_series(-3d, 3d);
```

| generate_series |
| --------------- |
| -3.0 |
| -2.0 |
| -1.0 |
| 0.0 |
| 1.0 |
| 2.0 |
| 3.0 |

```questdb-sql title="bwd double generation" demo
generate_series(-3d, 3d, -1d);
```

| generate_series |
| --------------- |
| 3.0 |
| 2.0 |
| 1.0 |
| 0.0 |
| -1.0 |
| -2.0 |
| -3.0 |
93 changes: 93 additions & 0 deletions documentation/reference/function/timestamp-generator.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ create data for testing. Pseudo-random steps can be achieved by providing a
`step` argument. A `seed` value may be provided to a random function if the
randomly-generated `step` should be deterministic.


## timestamp_sequence

- `timestamp_sequence(startTimestamp, step)` generates a sequence of `timestamp`
Expand Down Expand Up @@ -64,3 +65,95 @@ FROM long_sequence(5);
| 3 | 2019-10-17T00:00:00.600000Z |
| 4 | 2019-10-17T00:00:00.900000Z |
| 5 | 2019-10-17T00:00:01.300000Z |


## generate_series

Rather than generating a fixed number of timestamps, you can instead generate timestamps in a range,
using `generate_series`.

The step can be provided in either microseconds, or in a period string, similar to `SAMPLE BY`.

The `start` and `end` values are interchangeable, and a negative `step` value can be used to
obtain the series in reverse order.

The series is inclusive on both ends.

**Arguments:**

There are two timestamp-generating variants of `generate_series`:

`generate_series(start, end, step_period)` - generate a series of timestamps between `start` and `end`
in periodic steps.
`generate_series(start, end, step_micros)` - generates a series of timestamps between `start` and `end`,
in microsecond steps.


**Return value:**

Return value type is `timestamp`.

**Examples:**

```questdb-sql title="fwd series with period" demo
generate_series('2025-01-01', '2025-02-01', '5d');
```

| generate_series |
| --------------------------- |
| 2025-01-01T00:00:00.000000Z |
| 2025-01-06T00:00:00.000000Z |
| 2025-01-11T00:00:00.000000Z |
| 2025-01-16T00:00:00.000000Z |
| 2025-01-21T00:00:00.000000Z |
| 2025-01-26T00:00:00.000000Z |
| 2025-01-31T00:00:00.000000Z |

```questdb-sql title="bwd series with period" demo
generate_series('2025-01-01', '2025-02-01', '-5d');
```

| generate_series |
| --------------------------- |
| 2025-02-01T00:00:00.000000Z |
| 2025-01-27T00:00:00.000000Z |
| 2025-01-22T00:00:00.000000Z |
| 2025-01-17T00:00:00.000000Z |
| 2025-01-12T00:00:00.000000Z |
| 2025-01-07T00:00:00.000000Z |
| 2025-01-02T00:00:00.000000Z |

```questdb-sql title="fwd series with micro step demo
generate_series(
'2025-01-01T00:00:00Z'::timestamp,
'2025-01-01T00:05:00Z'::timestamp,
60_000_000
);
```

| generate_series |
| --------------------------- |
| 2025-01-01T00:00:00.000000Z |
| 2025-01-01T00:01:00.000000Z |
| 2025-01-01T00:02:00.000000Z |
| 2025-01-01T00:03:00.000000Z |
| 2025-01-01T00:04:00.000000Z |
| 2025-01-01T00:05:00.000000Z |


```questdb-sql title="vwd series with micro step demo
generate_series(
'2025-01-01T00:00:00Z'::timestamp,
'2025-01-01T00:05:00Z'::timestamp,
-60_000_000
);
```

| generate_series |
| --------------------------- |
| 2025-01-01T00:05:00.000000Z |
| 2025-01-01T00:04:00.000000Z |
| 2025-01-01T00:03:00.000000Z |
| 2025-01-01T00:02:00.000000Z |
| 2025-01-01T00:01:00.000000Z |
| 2025-01-01T00:00:00.000000Z |