Skip to content

add zstd compression support #591

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: 1.0
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions configuration/buffer-section.md
Original file line number Diff line number Diff line change
Expand Up @@ -411,9 +411,9 @@ Following are the configuration parameters for buffer plugin and its chunks:
fd resources when output destination has a problem. This parameter

mitigates such situations.
* `compress` \[enum: `text`/`gzip`\]
* `compress` \[ enum: `text` / `gzip` / `zstd`\(since v1.19.0\) \]
* Default: `text`
* If `gzip` is set, Fluentd compresses data records before writing to
* If `gzip`/`zstd` is set, Fluentd compresses data records before writing to

buffer chunks.

Expand Down
15 changes: 14 additions & 1 deletion output/file.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,20 @@ The suffix for output result.

### `compress`

Compresses flushed files using `gzip`. No compression is performed by default.
| type | default |
| :--- | :--- |
| enum | text |

Compresses flushed files.

No compression is performed by default.

Supported values:

* `text`
* `gz`
* `gzip`
* `zstd` (since v1.19.0)

### `recompress`

Expand Down
55 changes: 36 additions & 19 deletions output/forward.md
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,42 @@ Enable client-side DNS round robin. Uniform randomly pick an IP address to send

Ignores DNS resolution and errors at startup time.

### `compress`

| type | default | version |
| :--- | :--- | :--- |
| enum | text | 0.14.7 |

Since v0.14.7, Fluentd supports transparent data compression. You can use this feature to reduce the transferred payload size.

Example:

```text
<match debug.**>
@type forward
compress gzip
<server>
host 192.168.1.2
port 24224
</server>
</match>
```

You do not need any configuration in the receiving server. Data compression is auto-detected and handled transparently by the destination node.

Supported values:

* `text`
* `gzip`
* `zstd` (Experimental) (since v1.19.0)

{% hint style='info' %}
`zstd` is an experimental feature supported since v1.19.0.
Please make sure that the destination server also supports this feature before using it.

* [in_forward plugin](../input/forward.md) supports it since v1.19.0.
{% endhint %}

### `tls_version`

| type | default | available | version |
Expand Down Expand Up @@ -604,25 +640,6 @@ If you want to connect to a server that requires [password authentication](../in

Note that, as to the option `self_hostname`, you need to set the name of the server on which your `out_forward` instance is running. In the current implementation, it is considered invalid if your `in_forward` and `out_forward` share the same `hostname`.

### How to enable `gzip` compression?

Since v0.14.7, Fluentd supports transparent data compression. You can use this feature to reduce the transferred payload size.

To enable this feature, set the `compress` option as follows:

```text
<match debug.**>
@type forward
compress gzip
<server>
host 192.168.1.2
port 24224
</server>
</match>
```

You do not need any configuration in the receiving server. Data compression is auto-detected and handled transparently by the destination node.

### What is a Phi accrual failure detector?

Fluentd implements an adaptive failure detection mechanism called "Phi accrual failure detector". Here is how it works:
Expand Down