Skip to content

Commit a9d611d

Browse files
author
vvitkovskiy
committed
initial commit
0 parents  commit a9d611d

File tree

151 files changed

+12989
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

151 files changed

+12989
-0
lines changed

‎.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
file.d
2+
*.pprof
3+
*.test

‎.nojekyll

Whitespace-only changes.

‎AUTHORS

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
This is the official list of file.d authors for copyright purposes.
2+
Names should be added to this file as one of:
3+
* Organization's name
4+
* Individual's name <submission email address>
5+
6+
Authors:
7+
* OZON.ru
8+
* Vitkovskii Vladimir <[email protected]>

‎Dockerfile

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
FROM ubuntu:19.04
2+
3+
WORKDIR /file.d
4+
5+
COPY ./file.d .
6+
7+
CMD ./file.d

‎Insanedocfile

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
extractors:
2+
config-params: '"config-params" /json:\"([a-z_]+)\"/ #2 /default:\"([^"]+)\"/ /(required):\"true\"/ /options:\"([^"]+)\"/'
3+
fn-list: '"fn-list" #4 /Plugin\)\s(.+)\s{/'
4+
decorators:
5+
config-params: '_ _ /*`%s`* / /*`default=%s`* / /*`%s`* / /*`options=%s`* /'
6+
fn-list: '_ _ /`%s`/'
7+
templates:
8+
- template: plugin/*/*/README.idoc.md
9+
files: ["*.go"]
10+
- template: plugin/*/README.idoc.md
11+
files: []
12+
- template: plugin/README.idoc.md
13+
files: []
14+
- template: README.idoc.md
15+
files: []
16+
- template: _sidebar.idoc.md
17+
disable_footer: true
18+
files: []

‎LICENSE

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
Copyright (c) 2020 OZON.ru. All rights reserved.
2+
3+
Redistribution and use in source and binary forms, with or without
4+
modification, are permitted provided that the following conditions are met:
5+
* Redistributions of source code must retain the above copyright
6+
notice, this list of conditions and the following disclaimer.
7+
* Redistributions in binary form must reproduce the above copyright
8+
notice, this list of conditions and the following disclaimer in the
9+
documentation and/or other materials provided with the distribution.
10+
* Neither the name of the OZON.ru nor the
11+
names of its contributors may be used to endorse or promote products
12+
derived from this software without specific prior written permission.
13+
14+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
15+
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
16+
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
17+
DISCLAIMED. IN NO EVENT SHALL COPYRIGHT HOLDER BE LIABLE FOR ANY
18+
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
19+
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
20+
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
21+
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
22+
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
23+
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎Makefile

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
VERSION ?= v0.1.2
2+
3+
.PHONY: test
4+
test:
5+
go test ./fd/ -v -count 1
6+
go test ./pipeline/ -v -count 1
7+
go test ./plugin/... -v -count 1
8+
9+
.PHONY: test-e2e
10+
test-e2e:
11+
go test ./cmd/ -v -count 1
12+
13+
.PHONY: bench-file
14+
bench-file:
15+
go test -bench LightJsonReadPar ./plugin/input/file -v -count 1 -run -benchmem -benchtime 1x
16+
17+
.PHONY: profile-file
18+
profile-file:
19+
go test -bench LightJsonReadPar ./plugin/input/file -v -count 1 -run -benchmem -benchtime 1x -cpuprofile cpu.pprof -memprofile mem.pprof -mutexprofile mutex.pprof
20+
21+
.PHONY: push-linux-amd64
22+
push-linux-amd64:
23+
GOOS=linux GOARCH=amd64 go build -v -o file.d ./cmd/file.d.go
24+
docker build -t docker.pkg.github.com/ozonru/file.d/file.d-linux-amd64:${VERSION} .
25+
docker push docker.pkg.github.com/ozonru/file.d/file.d-linux-amd64:${VERSION}
26+
27+
.PHONY: push-images
28+
push-images: push-linux-amd64

‎README.idoc.md

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
![file.d](/static/file.d.png)
2+
3+
# Overview
4+
`file.d` is a blazing fast tool for building data pipelines: read, process, and output events. Primarily developed to read from files, but also supports numerous input/action/output plugins.
5+
6+
> ⚠ Although we use it in production, `it still isn't v1.0.0`. Please, test your pipelines carefully on dev/stage environments.
7+
8+
## Motivation
9+
Well, we already have several similar tools: vector, filebeat, logstash, fluend-d, fluent-bit, etc.
10+
11+
Performance tests state that best ones achieve a throughput of roughly 100MB/sec.
12+
Guys, it's 2020 now. HDDs and NICs can handle the throughput of a **few GB/sec** and CPUs processes **dozens of GB/sec**. Are you sure **100MB/sec** is what we deserve? Are you sure it is fast?
13+
14+
## Main features
15+
* Fast: more than 10x faster compared to similar tools
16+
* Predictable: it uses pooling, so memory consumption is limited
17+
* Reliable: doesn't lose data due to commitment mechanism
18+
* Container / cloud / kubernetes native
19+
* Simply configurable with YAML
20+
* Prometheus-friendly: transform your events into metrics on any pipeline stage
21+
* Well-tested and used in production to collect logs from Kubernetes cluster with 3000+ total CPU cores
22+
23+
## Performance
24+
On MacBook Pro 2017 with two physical cores `file.d` can achieve the following throughput:
25+
* 1.7GB/s in `files > devnull` case
26+
* 1.0GB/s in `files > json decode > devnull` case
27+
28+
TBD: throughput on production servers.
29+
30+
## Plugins
31+
32+
**Input**: @global-contents-table-plugin-input|links
33+
34+
**Action**: @global-contents-table-plugin-action|links
35+
36+
**Output**: @global-contents-table-plugin-output|links
37+
38+
## What's next
39+
* [Quick start](/docs/quick-start.md)
40+
* [Installation](/docs/installation.md)
41+
* [Examples](/docs/examples.md)
42+
* [Configuring](/docs/configuring.md)
43+
* [Architecture](/docs/architecture.md)
44+
* [Contributing](/docs/contributing.md)
45+
* [License](/docs/license.md)

‎README.md

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
![file.d](/static/file.d.png)
2+
# Overview
3+
`file.d` is a blazing fast tool for building data pipelines: read, process, and output events. Primarily developed to read from files, but also supports numerous input/action/output plugins.
4+
5+
> ⚠ Although we use it in production, `it still isn't v1.0.0`. Please, test your pipelines carefully on dev/stage environments.
6+
7+
## Motivation
8+
Well, we already have several similar tools: vector, filebeat, logstash, fluend-d, fluent-bit, etc.
9+
10+
Performance tests state that best ones achieve a throughput of roughly 100MB/sec.
11+
Guys, it's 2020 now. HDDs and NICs can handle the throughput of a **few GB/sec** and CPUs processes **dozens of GB/sec**. Are you sure **100MB/sec** is what we deserve? Are you sure it is fast?
12+
13+
## Main features
14+
* Fast: more than 10x faster compared to similar tools
15+
* Predictable: it uses pooling, so memory consumption is limited
16+
* Reliable: doesn't lose data due to commitment mechanism
17+
* Container / cloud / kubernetes native
18+
* Simply configurable with YAML
19+
* Prometheus-friendly: transform your events into metrics on any pipeline stage
20+
* Well-tested and used in production to collect logs from Kubernetes cluster with 3000+ total CPU cores
21+
22+
## Performance
23+
On MacBook Pro 2017 with two physical cores `file.d` can achieve the following throughput:
24+
* 1.7GB/s in `files > devnull` case
25+
* 1.0GB/s in `files > json decode > devnull` case
26+
27+
TBD: throughput on production servers.
28+
29+
## Plugins
30+
31+
**Input**: [fake](plugin/input/fake/README.md), [file](plugin/input/file/README.md), [http](plugin/input/http/README.md), [kafka](plugin/input/kafka/README.md)
32+
33+
**Action**: [discard](plugin/action/discard/README.md), [flatten](plugin/action/flatten/README.md), [join](plugin/action/join/README.md), [json_decode](plugin/action/json_decode/README.md), [k8s](plugin/action/k8s/README.md), [keep_fields](plugin/action/keep_fields/README.md), [modify](plugin/action/modify/README.md), [parse_es](plugin/action/parse_es/README.md), [remove_fields](plugin/action/remove_fields/README.md), [rename](plugin/action/rename/README.md), [throttle](plugin/action/throttle/README.md)
34+
35+
**Output**: [devnull](plugin/output/devnull/README.md), [elasticsearch](plugin/output/elasticsearch/README.md), [gelf](plugin/output/gelf/README.md), [kafka](plugin/output/kafka/README.md), [stdout](plugin/output/stdout/README.md)
36+
37+
## What's next
38+
* [Quick start](/docs/quick-start.md)
39+
* [Installation](/docs/installation.md)
40+
* [Examples](/docs/examples.md)
41+
* [Configuring](/docs/configuring.md)
42+
* [Architecture](/docs/architecture.md)
43+
* [Contributing](/docs/contributing.md)
44+
* [License](/docs/license.md)
45+
46+
<br>*Generated using [__insane-doc__](https://github.com/vitkovskii/insane-doc)*

‎_coverpage.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# <img src="/file.d/static/file.d.png" width="30%">
2+
3+
> A blazing fast tool for building data pipelines: read, process, and output events.
4+
5+
[GitHub](https://github.com/ozonru/file.d)
6+
[Get started](/README.md)
7+
8+

‎_navbar.md

Whitespace-only changes.

‎_sidebar.idoc.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
- **Getting started**
2+
- [Overview](/README.md)
3+
- [Quick start](/docs/quick-start.md)
4+
- [Installation](/docs/installation.md)
5+
- [Examples](/docs/examples.md)
6+
- [Configuring](/docs/configuring.md)
7+
8+
- **Documentation**
9+
- [Architecture](/docs/architecture.md)
10+
- [Benchmarks](/docs/benchmarks.md)
11+
- [Guarantees](/docs/guarantees.md)
12+
- [Optimization tips](/docs/optimization-tips.md)
13+
14+
- **Plugins**
15+
- Input
16+
@global-contents-table-plugin-input|links-list
17+
- Action
18+
@global-contents-table-plugin-action|links-list
19+
- Output
20+
@global-contents-table-plugin-output|links-list
21+
22+
- **Other**
23+
- [Contributing](/docs/contributing.md)
24+
- [License](/docs/license.md)

‎_sidebar.md

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
- **Getting started**
2+
- [Overview](/README.md)
3+
- [Quick start](/docs/quick-start.md)
4+
- [Installation](/docs/installation.md)
5+
- [Examples](/docs/examples.md)
6+
- [Configuring](/docs/configuring.md)
7+
8+
- **Documentation**
9+
- [Architecture](/docs/architecture.md)
10+
- [Benchmarks](/docs/benchmarks.md)
11+
- [Guarantees](/docs/guarantees.md)
12+
- [Optimization tips](/docs/optimization-tips.md)
13+
14+
- **Plugins**
15+
- Input
16+
- [fake](plugin/input/fake/README.md)
17+
- [file](plugin/input/file/README.md)
18+
- [http](plugin/input/http/README.md)
19+
- [kafka](plugin/input/kafka/README.md)
20+
21+
- Action
22+
- [discard](plugin/action/discard/README.md)
23+
- [flatten](plugin/action/flatten/README.md)
24+
- [join](plugin/action/join/README.md)
25+
- [json_decode](plugin/action/json_decode/README.md)
26+
- [k8s](plugin/action/k8s/README.md)
27+
- [keep_fields](plugin/action/keep_fields/README.md)
28+
- [modify](plugin/action/modify/README.md)
29+
- [parse_es](plugin/action/parse_es/README.md)
30+
- [remove_fields](plugin/action/remove_fields/README.md)
31+
- [rename](plugin/action/rename/README.md)
32+
- [throttle](plugin/action/throttle/README.md)
33+
34+
- Output
35+
- [devnull](plugin/output/devnull/README.md)
36+
- [elasticsearch](plugin/output/elasticsearch/README.md)
37+
- [gelf](plugin/output/gelf/README.md)
38+
- [kafka](plugin/output/kafka/README.md)
39+
- [stdout](plugin/output/stdout/README.md)
40+
41+
42+
- **Other**
43+
- [Contributing](/docs/contributing.md)
44+
- [License](/docs/license.md)

0 commit comments

Comments
 (0)