New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to improve the performance of filebeat collection? #42898

Open

huoxinyang opened this issue Feb 26, 2025 · 1 comment

Labels

needs_team

huoxinyang commented Feb 26, 2025

Currently, about 3,000 logs are collected per second, with each log being 7 KB in size.
Here is my configuration file:

type: log
enabled: true
paths:
- /host/var/lib/kubelet/pods/de3fc800-c051-457b-8e8d-9209852b0316/volumes/kubernetes.io~empty-dir/log/worker.log
scan_frequency: 5s
fields_under_root: true

fields:
```
index: worker-online01

topic: worker-online01


docker_container: k8s_worker_worker-deploy-68ff7648d5-2d7g5_iov-online01_de3fc800-c051-457b-8e8d-9209852b0316_0

k8s_container_name: worker

k8s_node_name: 100.70.17.100

k8s_pod: worker-deploy-68ff7648d5-2d7g5

k8s_pod_namespace: worker-online01
```
tail_files: false
close_inactive: 2h
close_eof: false
close_removed: true
clean_removed: true
close_renamed: true
harvester_buffer_size: 2097152
backoff: 0.5s
max_backoff: 5s
max_bytes: 4194304
clean_inactive: 73h
ignore_older: 72h
harvester_limit: 1024

The text was updated successfully, but these errors were encountered:

botelastic bot added the needs_team label

botelastic bot commented Feb 26, 2025

This issue doesn't have a Team:<team> label.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment