tail: New keep_file_handle and fstat_interval_nsec options for tail input #11151
Conversation
david-garcia-garcia
requested review from
cosmo0920 and
edsiper
as code owners
david-garcia-garcia
had a problem deploying
to
pr
GitHub Actions
Failure
david-garcia-garcia
had a problem deploying
to
pr
GitHub Actions
Failure
david-garcia-garcia
had a problem deploying
to
pr
GitHub Actions
Failure
WalkthroughAdds configurable file-handle management and stat-polling interval to in_tail: new options Changes
Sequence Diagram(s)sequenceDiagram
participant Cfg as Config
participant Core as TailCore
participant File as TailFile
participant FS as Filesystem
rect rgba(45,125,255,0.06)
Cfg->>Core: set keep_file_handle & fstat_interval
end
alt keep_file_handle = true
Core->>File: flb_tail_file_stat(file)
File->>FS: fstat(fd)
FS-->>File: stat result
File-->>Core: size/ino
Core->>File: lseek/update offset
else keep_file_handle = false
Core->>File: flb_tail_file_ensure_open_handle()
File->>FS: open() + lseek() + stat(opened fd)
FS-->>File: stat result
File-->>Core: size/ino
Core->>File: flb_tail_file_close_handle_during_tail()
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes
Suggested reviewers
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
fluent-bit/plugins/in_tail/tail_fs_inotify.c
Lines 135 to 139 in 0bc15c8
Restore external visibility for flb_tail_file_name
This change makes flb_tail_file_name static inside tail_file.c and removes the declaration from the header, but callers in other compilation units (tail_fs_inotify.c and tail_fs_stat.c) still invoke it. Because the symbol now has internal linkage, those references no longer resolve and the build fails with an undefined symbol for flb_tail_file_name. Either keep the function exported or replace the cross-file uses.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/in_tail/tail_file.c (1)
2071-2170: Restore external linkage for flb_tail_file_name().Other units call this symbol; making it static breaks linking. Change to non‑static (paired with header re‑export).
- static char *flb_tail_file_name(struct flb_tail_file *file) + char *flb_tail_file_name(struct flb_tail_file *file)
🧹 Nitpick comments (6)
tests/runtime/in_tail.c (2)
1388-1462: Reduce test flakiness: wait for outputs instead of fixed sleeps.Replace the three 500 ms sleeps with wait_num_with_timeout(5000, &num) loops to make the test robust on slow CI.
- /* waiting to flush */ - flb_time_msleep(500); + /* wait up to 5s for at least one output */ + wait_num_with_timeout(5000, &num); ... - /* waiting to flush */ - flb_time_msleep(500); + wait_num_with_timeout(5000, &num); ... - /* waiting to flush */ - flb_time_msleep(500); + wait_num_with_timeout(5000, &num);
1464-1539: Apply the same non‑flaky wait strategy to the “keep_file_handle=true” test.Mirror the wait_num_with_timeout change here to avoid time‑sensitive failures.
plugins/in_tail/tail.c (1)
622-628: Consider parsing fstat_interval via existing time utility.You advertise “s/ms/us/ns”; instead of manual parsing in tail_config.c, you could use flb_utils_time_split() for consistent semantics with refresh_interval. Not a blocker.
plugins/in_tail/tail_config.c (1)
194-233: Leverage flb_utils_time_split for fstat_interval.You can simplify and unify parsing by using flb_utils_time_split(tmp, &sec, &nsec) like refresh_interval, then compute ctx->fstat_interval_nsec. This removes unit parsing code and edge cases.
- tmp = flb_input_get_property("fstat_interval", ins); - if (tmp) { - char *end = NULL; - double val = strtod(tmp, &end); - uint64_t mult = 1000000000ULL; /* default: seconds */ - ... - ctx->fstat_interval_nsec = (uint64_t) nsec_d; - ... - } + tmp = flb_input_get_property("fstat_interval", ins); + if (tmp) { + int sec; long nsec; + if (flb_utils_time_split(tmp, &sec, &nsec) == 0 && (sec > 0 || nsec > 0)) { + ctx->fstat_interval_nsec = (uint64_t)sec * 1000000000ULL + (uint64_t)nsec; + if (ctx->fstat_interval_nsec <= 1000000ULL) { + flb_plg_warn(ctx->ins, "very low fstat_interval (%" PRIu64 " ns) may cause high CPU usage", + ctx->fstat_interval_nsec); + } + } + else { + flb_plg_error(ctx->ins, "invalid 'fstat_interval' value (%s)", tmp); + flb_tail_config_destroy(ctx); + return NULL; + } + }plugins/in_tail/tail_fs_stat.c (1)
168-190: Avoid redundant real‑name lookup.You call flb_tail_file_name(file) but then rely on flb_tail_file_is_rotated(), which does the lookup again. Remove the extra allocation.
- name = flb_tail_file_name(file); - if (!name) { - flb_plg_debug(ctx->ins, "could not resolve %s, removing", file->name); - flb_tail_file_remove(file); - continue; - } ... - if (flb_tail_file_is_rotated(ctx, file) == FLB_TRUE) { + if (flb_tail_file_is_rotated(ctx, file) == FLB_TRUE) { flb_tail_file_rotated(file); } - flb_free(name);plugins/in_tail/tail_file.c (1)
1635-1666: Optional: set pending_bytes on truncation.After truncation branch, pending_bytes may carry stale value; set it explicitly from current st/offset.
if (size_delta < 0) { ... - /* Update offset in the database file */ + /* Sync pending_bytes after truncation */ + file->pending_bytes = 0; + /* Update offset in the database file */
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (8)
plugins/in_tail/tail.c(5 hunks)plugins/in_tail/tail_config.c(2 hunks)plugins/in_tail/tail_config.h(1 hunks)plugins/in_tail/tail_file.c(23 hunks)plugins/in_tail/tail_file.h(1 hunks)plugins/in_tail/tail_fs_inotify.c(3 hunks)plugins/in_tail/tail_fs_stat.c(5 hunks)tests/runtime/in_tail.c(3 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-10-23T07:43:16.216Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11059
File: plugins/in_tail/tail_file.c:1618-1640
Timestamp: 2025-10-23T07:43:16.216Z
Learning: In plugins/in_tail/tail_file.c, when truncate_long_lines is enabled and the buffer is full, the early truncation path uses `lines > 0` as the validation pattern to confirm whether process_content successfully processed content. This is intentional to track occurrences of line processing rather than byte consumption, and consuming bytes based on `processed_bytes > 0` would be overkill for this validation purpose.
Applied to files:
plugins/in_tail/tail_fs_inotify.cplugins/in_tail/tail_fs_stat.cplugins/in_tail/tail_file.c
🧬 Code graph analysis (7)
plugins/in_tail/tail_file.h (1)
plugins/in_tail/tail_file.c (4)
flb_tail_file_ensure_open_handle(62-90)flb_tail_file_stat(93-101)flb_tail_file_close_handle(104-110)flb_tail_file_close_handle_during_tail(113-118)
plugins/in_tail/tail_fs_inotify.c (1)
plugins/in_tail/tail_file.c (1)
flb_tail_file_stat(93-101)
plugins/in_tail/tail.c (1)
plugins/in_tail/tail_file.c (1)
flb_tail_file_stat(93-101)
tests/runtime/in_tail.c (1)
src/flb_lib.c (2)
flb_input_set(305-335)flb_output_set(520-551)
plugins/in_tail/tail_config.c (1)
src/flb_input.c (1)
flb_input_get_property(776-780)
plugins/in_tail/tail_fs_stat.c (2)
plugins/in_tail/tail_file.c (5)
flb_tail_file_stat(93-101)flb_tail_file_name(2079-2170)flb_tail_file_remove(1519-1594)flb_tail_file_is_rotated(1948-2019)flb_tail_file_rotated(2197-2273)src/flb_input.c (1)
flb_input_set_collector_time(1685-1704)
plugins/in_tail/tail_file.c (2)
include/fluent-bit/flb_mem.h (1)
flb_free(126-128)src/flb_compression.c (1)
flb_decompression_context_create(151-219)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (28)
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_ARROW=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_COVERAGE=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, clang, clang++)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-24.04, clang-14)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, clang, clang++)
- GitHub Check: pr-compile-centos-7
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, gcc, g++)
- GitHub Check: pr-compile-without-cxx (3.31.6)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-22.04, clang-12)
- GitHub Check: PR - fuzzing test
🔇 Additional comments (13)
plugins/in_tail/tail_config.h (1)
110-113: LGTM: new config fields look correct and well‑placed.No structural risks; types are appropriate for ns granularity and boolean.
tests/runtime/in_tail.c (1)
1562-1564: Good fix: use input fd for flb_input_set.Corrects misuse of ctx->o_ffd.
plugins/in_tail/tail.c (3)
81-89: Stat wrapper usage is correct.Switching to flb_tail_file_stat() centralizes status retrieval and respects closed FDs.
327-335: Gating link-rotation checks by keep_file_handle is appropriate.Rotation detection relies on an open FD; skipping it when handles aren’t kept avoids invalid paths.
730-736: Option name and default read well.keep_file_handle default true maintains current behavior; help text is clear.
plugins/in_tail/tail_fs_inotify.c (2)
218-233: Stat wrapper adoption looks good.Consistent with new helpers; error handling remains correct.
259-273: Truncation handling respects keep_file_handle.Seeking vs resetting offset matches the configured policy.
plugins/in_tail/tail_config.c (1)
109-111: Sane defaults.Default keep_file_handle=true and 250ms poll are reasonable.
plugins/in_tail/tail_fs_stat.c (3)
102-107: Stat wrapper adoption is correct.Aligns with closed‑FD behavior and unified error handling.
127-154: Truncation handling respects keep_file_handle.Seeking vs offset reset mirrors inotify path; DB offset update preserved.
205-213: Nice: timer now uses configured fstat_interval.Converts ns to (sec, nsec) for collector; good.
plugins/in_tail/tail_file.c (2)
61-90: Open‑handle helper looks good.Opens on demand and seeks to current offset; returns FLB_TAIL_ERROR consistently.
1891-1943: Closing on non‑kept handles is correctly placed.You close after processing, EOF, and errors; matches policy.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (1)
plugins/in_tail/tail_file.c (1)
1277-1294: Fix memory leak: file->name is duplicated twice.This issue was previously identified. Line 1281 allocates
file->namewithflb_strdup(path), and then line 1307'sflb_tail_file_name_dup()allocates it again (see line 2174) without freeing the first allocation.Apply the fix suggested in the previous review:
file->name = flb_strdup(path); if (!file->name) { flb_errno(); goto error; } file->name_len = strlen(file->name); /* Open file handle first - flb_tail_file_name_dup() needs it open */ ret = flb_tail_file_ensure_open_handle(file); if (ret != 0) { flb_free(file->name); file->name = NULL; goto error; } + /* Avoid leaking the temporary name; flb_tail_file_name_dup() will set it again */ + flb_free(file->name); + file->name = NULL; + file->name_len = 0; + /* * Duplicate string into 'file' structure, the called function * take cares to resolve real-name of the file in case we are * running in a non-Linux system. */ ret = flb_tail_file_name_dup(path, file);
🧹 Nitpick comments (1)
plugins/in_tail/tail_file.c (1)
1911-1945: Consider consolidating redundant close calls.Lines 1911, 1921, 1936, and 1942 all call
flb_tail_file_close_handle_during_tail(). While the function is idempotent and safe to call multiple times, the calls at lines 1936 and 1942 are in the same error path and could be consolidated.Example refactor:
else { /* error */ flb_errno(); flb_plg_error(ctx->ins, "error reading %s", file->name); - - /* Close file handle if keep_file_handle is false */ - flb_tail_file_close_handle_during_tail(file); - - return FLB_TAIL_ERROR; - } - - /* Close file handle if keep_file_handle is false */ - flb_tail_file_close_handle_during_tail(file); + flb_tail_file_close_handle_during_tail(file); + return FLB_TAIL_ERROR; + } + return FLB_TAIL_ERROR;
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
plugins/in_tail/tail_file.c(23 hunks)plugins/in_tail/tail_file.h(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- plugins/in_tail/tail_file.h
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-10-23T07:43:16.216Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11059
File: plugins/in_tail/tail_file.c:1618-1640
Timestamp: 2025-10-23T07:43:16.216Z
Learning: In plugins/in_tail/tail_file.c, when truncate_long_lines is enabled and the buffer is full, the early truncation path uses `lines > 0` as the validation pattern to confirm whether process_content successfully processed content. This is intentional to track occurrences of line processing rather than byte consumption, and consuming bytes based on `processed_bytes > 0` would be overkill for this validation purpose.
Applied to files:
plugins/in_tail/tail_file.c
🧬 Code graph analysis (1)
plugins/in_tail/tail_file.c (2)
include/fluent-bit/flb_mem.h (1)
flb_free(126-128)src/flb_compression.c (1)
flb_decompression_context_create(151-219)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (28)
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-24.04, clang-14)
- GitHub Check: pr-compile-without-cxx (3.31.6)
- GitHub Check: pr-compile-centos-7
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_ARROW=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_COMPILER_STRICT_POINTER_TYPES=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_COVERAGE=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, gcc, g++)
- GitHub Check: PR - fuzzing test
🔇 Additional comments (7)
plugins/in_tail/tail_file.c (7)
61-119: LGTM: Clean abstraction for file handle management.The new helper functions provide a clean abstraction for managing file handles with proper error handling and conditional behavior based on the
keep_file_handleconfiguration.
1313-1322: LGTM: Decompression context initialization.The decompression context creation is correctly positioned after the file handle is opened and the name is duplicated.
1637-1649: LGTM: Correct truncation handling for both handle modes.The logic correctly handles file truncation: when
keep_file_handleis true, it seeks the open handle; when false, it updates the offset knowing thatflb_tail_file_ensure_open_handle()will seek correctly on the next open.
2043-2049: LGTM: Rotation detection correctly gated.Rotation detection is appropriately conditional on
keep_file_handlebeing enabled, as the detection mechanism requires an open file descriptor.
2207-2214: LGTM: Good defensive check.The defensive check ensures
flb_tail_file_rotated()is only called whenkeep_file_handleis enabled, preventing misuse.
2074-2078: LGTM: Clarified function requirements.The updated comment clearly documents that this function requires an open file descriptor and should only be called when
keep_file_handleis enabled.
2282-2336: LGTM: Consistent use of stat abstraction.The purge logic correctly uses the
flb_tail_file_stat()wrapper for consistent file status checking.
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/in_tail/tail_fs_stat.c (1)
164-184: Critical: Rotation detection should be conditional onkeep_file_handle.The rotation detection logic calls
flb_tail_file_name(file)which discovers the file name from an open file descriptor. Whenkeep_file_handleisFLB_FALSE,file->fdis-1, causingflb_tail_file_nameto fail and returnNULL. This results in all files being incorrectly removed with "could not resolve" errors.The AI summary states: "Rotation checks now run only when keep_file_handle is true," but the code doesn't implement this conditional.
Wrap the rotation detection in a
keep_file_handlecheck:file->pending_bytes = (st.st_size - file->offset); tail_signal_pending(ctx); } else { file->pending_bytes = 0; } - /* Discover the current file name for the open file descriptor */ - name = flb_tail_file_name(file); - if (!name) { - flb_plg_debug(ctx->ins, "could not resolve %s, removing", file->name); - flb_tail_file_remove(file); - continue; - } - - /* - * Check if file still exists. This method requires explicity that the - * user is using an absolute path, otherwise we will be rotating the - * wrong file. - * - * flb_tail_target_file_name_cmp is a deeper compare than - * flb_tail_file_name_cmp. If applicable, it compares to the underlying - * real_name of the file. - */ - if (flb_tail_file_is_rotated(ctx, file) == FLB_TRUE) { - flb_tail_file_rotated(file); - } - flb_free(name); + /* Rotation detection only works when keeping file handles open */ + if (ctx->keep_file_handle == FLB_TRUE) { + /* Discover the current file name for the open file descriptor */ + name = flb_tail_file_name(file); + if (!name) { + flb_plg_debug(ctx->ins, "could not resolve %s, removing", file->name); + flb_tail_file_remove(file); + continue; + } + + /* + * Check if file still exists. This method requires explicity that the + * user is using an absolute path, otherwise we will be rotating the + * wrong file. + * + * flb_tail_target_file_name_cmp is a deeper compare than + * flb_tail_file_name_cmp. If applicable, it compares to the underlying + * real_name of the file. + */ + if (flb_tail_file_is_rotated(ctx, file) == FLB_TRUE) { + flb_tail_file_rotated(file); + } + flb_free(name); + } }
🧹 Nitpick comments (1)
plugins/in_tail/tail_fs_stat.c (1)
198-203: Consider using explicit time_t cast for clarity.The interval calculation logic is correct. The cast to
(int)for seconds works but is slightly unusual sinceflb_input_set_collector_timeexpectstime_t. While unlikely to cause issues in practice (would require extremely large intervals), usingtime_tdirectly would be clearer./* Set a manual timer to collect events using configured interval */ /* Convert nanoseconds to seconds and nanoseconds for the API */ ret = flb_input_set_collector_time(in, tail_fs_event, - (int)(ctx->fstat_interval_nsec / 1000000000L), + (time_t)(ctx->fstat_interval_nsec / 1000000000L), (long)(ctx->fstat_interval_nsec % 1000000000L), config);
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
plugins/in_tail/tail_fs_stat.c(4 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-10-23T07:43:16.216Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11059
File: plugins/in_tail/tail_file.c:1618-1640
Timestamp: 2025-10-23T07:43:16.216Z
Learning: In plugins/in_tail/tail_file.c, when truncate_long_lines is enabled and the buffer is full, the early truncation path uses `lines > 0` as the validation pattern to confirm whether process_content successfully processed content. This is intentional to track occurrences of line processing rather than byte consumption, and consuming bytes based on `processed_bytes > 0` would be overkill for this validation purpose.
Applied to files:
plugins/in_tail/tail_fs_stat.c
🧬 Code graph analysis (1)
plugins/in_tail/tail_fs_stat.c (3)
plugins/in_tail/tail_file.c (1)
flb_tail_file_stat(93-101)src/flb_input.c (1)
flb_input_set_collector_time(1685-1704)plugins/in_tail/tail_fs_inotify.c (1)
tail_fs_event(166-311)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (31)
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_COMPILER_STRICT_POINTER_TYPES=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_ARROW=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_COVERAGE=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, gcc, g++)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-22.04, clang-12)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, clang, clang++)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-24.04, clang-14)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-24.04, clang-14)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-22.04, clang-12)
- GitHub Check: PR - fuzzing test
- GitHub Check: pr-compile-without-cxx (3.31.6)
- GitHub Check: pr-compile-centos-7
🔇 Additional comments (2)
plugins/in_tail/tail_fs_stat.c (2)
65-65: LGTM!Correct usage of the
flb_tail_file_statwrapper, which handles both open (fstat) and closed (stat) file descriptor scenarios based on thekeep_file_handleconfiguration.
102-104: LGTM!The stat wrapper usage is correct, and the error message improvement (adding "check:" prefix) helps distinguish this stat error context from others in the logs.
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (1)
plugins/in_tail/tail_fs_stat.c (1)
129-154: Avoid aborting the entire stat check loop onlseekfailure after truncationIf
lseek(file->fd, 0, SEEK_SET)fails, returning-1abortstail_fs_checkand prevents remaining files from being processed. A single bad FD should not break the whole stat-based backend; we should log, drop the problematic file, and continue.Suggested change:
- /* If keeping handle open, it's already open but at wrong offset - seek to beginning */ - if (ctx->keep_file_handle == FLB_TRUE) { - offset = lseek(file->fd, 0, SEEK_SET); - if (offset == -1) { - flb_errno(); - return -1; - } - file->offset = offset; - } + /* If keeping handle open, it's already open but at wrong offset - seek to beginning */ + if (ctx->keep_file_handle == FLB_TRUE) { + offset = lseek(file->fd, 0, SEEK_SET); + if (offset == -1) { + flb_errno(); + flb_plg_error(ctx->ins, + "tail_fs_check: failed to seek '%s' after truncation, removing", + file->name); + flb_tail_file_remove(file); + continue; + } + file->offset = offset; + } else { /* If not keeping handle open, just update offset - handle will be opened/seeks correctly later */ file->offset = 0; }
🧹 Nitpick comments (1)
plugins/in_tail/tail_fs_stat.c (1)
212-217: Interval conversion fromfstat_interval_nseclooks correct; consider guarding extreme/zero valuesThe seconds/nanoseconds split from
ctx->fstat_interval_nsecis arithmetically sound and preserves the previous 250ms default when configured accordingly. If not already validated in config parsing, you may want to ensure this value is non-zero and within a sane range to avoid a too-tight or effectively disabled collector.If
fstat_interval_nsecis not already range-checked intail_config.c, consider adding validation there rather than here.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
plugins/in_tail/tail_fs_stat.c(5 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-10-23T07:43:16.216Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11059
File: plugins/in_tail/tail_file.c:1618-1640
Timestamp: 2025-10-23T07:43:16.216Z
Learning: In plugins/in_tail/tail_file.c, when truncate_long_lines is enabled and the buffer is full, the early truncation path uses `lines > 0` as the validation pattern to confirm whether process_content successfully processed content. This is intentional to track occurrences of line processing rather than byte consumption, and consuming bytes based on `processed_bytes > 0` would be overkill for this validation purpose.
Applied to files:
plugins/in_tail/tail_fs_stat.c
🧬 Code graph analysis (1)
plugins/in_tail/tail_fs_stat.c (3)
plugins/in_tail/tail_file.c (1)
flb_tail_file_stat(93-101)src/flb_input.c (1)
flb_input_set_collector_time(1685-1704)plugins/in_tail/tail_fs_inotify.c (1)
tail_fs_event(166-311)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (31)
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_COMPILER_STRICT_POINTER_TYPES=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_ARROW=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_COVERAGE=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, clang, clang++)
- GitHub Check: PR - fuzzing test
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-24.04, clang-14)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-22.04, clang-12)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-22.04, clang-12)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-24.04, clang-14)
- GitHub Check: pr-compile-without-cxx (3.31.6)
- GitHub Check: pr-compile-centos-7
🔇 Additional comments (3)
plugins/in_tail/tail_fs_stat.c (3)
65-69: Stat wrapper use intail_fs_eventis correct for both handle modesSwitching to
flb_tail_file_stat(file, &st)lets this path work whether the FD is open or closed, and the existing error handling (flb_errno(); continue;) remains appropriate.
102-107: Stat wrapper and error handling intail_fs_checklook consistentUsing
flb_tail_file_stathere aligns with the inotify backend, and removing the file on repeatedstatfailure is a reasonable recovery strategy; the updated log message is clearer.
164-176: Rotation detection skip whenkeep_file_handleis false matches documented behaviorThe early
continuecleanly disables rotation detection in stat mode when handles are not kept open, which aligns with the new option’s documented trade-off and avoids repeatedflb_tail_file_name()work.
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
force-pushed
the
11147c
branch
from
f27bd7b to
8feb31e
Compare
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/in_tail/tail_file.c (1)
1679-1715: Ensure fds are closed on all early-return paths inflb_tail_file_chunkforkeep_file_handle == FLB_FALSE
flb_tail_file_chunk()correctly opens viaflb_tail_file_ensure_open_handle()and closes on most exit paths viaflb_tail_file_close_handle_during_tail(). However, two early returns in the “buffer full + truncate_long_lines” branch do not close the fd:
- When
lines < 0afterprocess_content()(Line 1694+).- When
lines > 0and youreturn adjust_counters(ctx, file);(Line 1701+).In
keep_file_handle == FLB_FALSEmode, these leave the fd open across iterations, undermining the “do not keep file handles open” guarantee and potentially blocking deletion on SMB/SAMBA for that file.Recommended fix:
if (ctx->truncate_long_lines == FLB_TRUE) { lines = process_content(file, &processed_bytes); if (lines < 0) { flb_plg_debug(ctx->ins, "inode=%"PRIu64" file=%s process content ERROR", file->inode, file->name); - return FLB_TAIL_ERROR; + flb_tail_file_close_handle_during_tail(file); + return FLB_TAIL_ERROR; } if (lines > 0) { file->stream_offset += processed_bytes; file->last_processed_bytes = 0; consume_bytes(file->buf_data, processed_bytes, file->buf_len); file->buf_len -= processed_bytes; file->buf_data[file->buf_len] = '\0'; ... - return adjust_counters(ctx, file); + ret = adjust_counters(ctx, file); + flb_tail_file_close_handle_during_tail(file); + return ret; } }This makes the “open once per chunk, close on every exit” rule consistent across all paths for
keep_file_handle == FLB_FALSE.
♻️ Duplicate comments (1)
plugins/in_tail/tail_fs_stat.c (1)
128-155: Avoid abortingtail_fs_checkwhenlseekfails on a single fileOn truncation with
keep_file_handle == FLB_TRUE, a failinglseek(file->fd, 0, SEEK_SET)logs viaflb_errno()and returns-1, which abortstail_fs_checkand prevents remaining files from being checked in this cycle. A single bad file should not block monitoring of all others; it’s safer to log, drop this file, and continue the loop. This matches the earlier suggested behavior.Suggested change:
- if (ctx->keep_file_handle == FLB_TRUE) { - offset = lseek(file->fd, 0, SEEK_SET); - if (offset == -1) { - flb_errno(); - return -1; - } - file->offset = offset; - } + if (ctx->keep_file_handle == FLB_TRUE) { + offset = lseek(file->fd, 0, SEEK_SET); + if (offset == -1) { + flb_errno(); + flb_plg_error(ctx->ins, + "tail_fs_check: failed to seek after truncation, " + "dropping file %s", file->name); + flb_tail_file_remove(file); + continue; + } + file->offset = offset; + }This preserves the current file’s failure behavior but allows the rest of
ctx->files_eventto be processed.
🧹 Nitpick comments (10)
plugins/in_tail/tail_fs_inotify.c (1)
218-224: Good use offlb_tail_file_statto support closed FDsSwitching to
flb_tail_file_stat()here lets truncation/progress checks work whenfile->fd == -1(e.g.keep_file_handle=false) without changing other call sites. Only minor nit: the"fstat error"log inin_tail_progress_check_callbackis now slightly misleading since the helper may call eitherstatorfstat, but functionally this is fine.Also applies to: 337-343
tests/runtime/in_tail.c (2)
1388-1642: New keep_file_handle tests cover basic and truncation scenariosThese three tests:
- exercise both
keep_file_handle=falseandkeep_file_handle=truewith multiple writes, and- explicitly validate truncation behavior (
ftruncate+ subsequent writes) forkeep_file_handle=false.They are consistent with existing test style (sleep-based flushing and global counters) and should give decent coverage of the new option without introducing new patterns.
If you later want to harden them against slow CI, consider reusing
wait_num_with_timeout()instead of fixed 500ms sleeps to reduce flakiness risk.
1666-1668: Correctly using input fd inflb_test_path_commaSwitching
flb_input_setto usectx->i_ffdinstead ofctx->o_ffdaligns with the API and the rest of the file’s patterns; this fixes the descriptor mix-up in this test.You might want to audit nearby tests that still pass
ctx->o_ffdintoflb_input_set(e.g.,flb_test_path_key,flb_test_exclude_path) and bring them in line with this fix in a follow-up.plugins/in_tail/tail.c (1)
622-628: Config surface forfstat_intervalandkeep_file_handlelooks coherentThe new
fstat_interval(STR) andkeep_file_handle(BOOL) entries:
- are documented clearly (units and intended use),
- map cleanly onto
struct flb_tail_configfields, and- align with the tests and parsing in
tail_config.c.Note that
fstat_interval’s default is effectively implemented viactx->fstat_interval_nsecinitialization rather than the config_map default, which is fine but slightly redundant.Also applies to: 729-736
plugins/in_tail/tail_config.c (1)
194-233: fstat_interval parsing is robust and matches documented unitsThis block:
- reads the raw
fstat_intervalproperty,- supports
s,ms,us, andnssuffixes as well as plain/fractional seconds,- validates positive values, converts to nanoseconds, and warns for very small intervals.
The error handling (logging +
flb_tail_config_destroy+NULLreturn) is consistent with other config validations.If you ever want to harden this further, trimming leading/trailing whitespace before
strtod/ suffix checks would make the parser a bit more forgiving of config formatting.plugins/in_tail/tail_fs_stat.c (2)
164-174: Rotation detection guard forkeep_file_handle == FLB_FALSEis consistent with designSkipping rotation detection when
keep_file_handleis false aligns with the documented behavior (rotation interpreted via truncation) and avoids repeated open/close cycles fromflb_tail_file_name(). The earlycontinuekeeps the check loop simple and avoids unnecessary syscalls for the SMB/SAMBA use case.If you later want rotation heuristics even without persistent handles, consider a lightweight, path-based strategy behind a separate option, but current behavior is fine for this PR scope.
210-215:fstat_interval_nsecconversion to collector interval looks correctThe conversion from
ctx->fstat_interval_nsecto(seconds, nanoseconds)forflb_input_set_collector_time()is arithmetically correct and keeps behavior compatible with the existing API while making the interval configurable.If
fstat_interval_nseccan be configured freely, consider validating against0(or extremely small values) at config parse time to avoid tight polling loops.plugins/in_tail/tail_file.c (3)
61-118: Handle helper APIs (ensure_open_handle/stat/close) are sound and cohesiveThe new helpers cleanly separate concerns:
flb_tail_file_ensure_open_handle()opens and seeks based onfile->offsetonly whenfd == -1.flb_tail_file_stat()transparently choosesstat()vsfstat()depending on open state.flb_tail_file_close_handle()and_close_handle_during_tail()encapsulate closing with/withoutkeep_file_handle.This greatly simplifies the rest of the tailing code and matches the intended semantics for SMB/SAMBA when
keep_file_handleis false.You might consider adding a small debug log on
lseekfailure inflb_tail_file_ensure_open_handle()to complementflb_errno()for easier tracing of resume-position problems.
1553-1555: Usingflb_tail_file_close_handle()in remove centralizes close behaviorSwitching to
flb_tail_file_close_handle()inflb_tail_file_remove()unifies close semantics and keeps thefdinvariant (-1when closed) consistent everywhere, which is helpful now that descriptors may be reopened/closed multiple times.The outer
if (file->fd != -1)check is now redundant with the helper’s own guard and could be dropped in a later cleanup.
1752-1755: Handle-closing on error/EOF paths inflb_tail_file_chunkis well-integratedOn error conditions (buffer resize failure, decompression failure, read error) and on normal EOF, you now call
flb_tail_file_close_handle_during_tail(file)before returning. Forkeep_file_handle == FLB_FALSEthis ensures descriptors are not left open after each polling cycle, whilekeep_file_handle == FLB_TRUEkeeps the previous behavior via a no-op. Combined with the suggested early-return fixes, this will fully honor the new configuration.Once the early-return paths are fixed, it might be worth factoring a
goto close_and_returnpattern to reduce duplication of the close+return logic.Also applies to: 1804-1805, 1843-1844, 1876-1877, 1893-1901, 1906-1908, 1921-1924, 1927-1929
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (8)
plugins/in_tail/tail.c(4 hunks)plugins/in_tail/tail_config.c(2 hunks)plugins/in_tail/tail_config.h(1 hunks)plugins/in_tail/tail_file.c(19 hunks)plugins/in_tail/tail_file.h(1 hunks)plugins/in_tail/tail_fs_inotify.c(3 hunks)plugins/in_tail/tail_fs_stat.c(5 hunks)tests/runtime/in_tail.c(3 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- plugins/in_tail/tail_file.h
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-10-23T07:43:16.216Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11059
File: plugins/in_tail/tail_file.c:1618-1640
Timestamp: 2025-10-23T07:43:16.216Z
Learning: In plugins/in_tail/tail_file.c, when truncate_long_lines is enabled and the buffer is full, the early truncation path uses `lines > 0` as the validation pattern to confirm whether process_content successfully processed content. This is intentional to track occurrences of line processing rather than byte consumption, and consuming bytes based on `processed_bytes > 0` would be overkill for this validation purpose.
Applied to files:
plugins/in_tail/tail_fs_inotify.cplugins/in_tail/tail_fs_stat.cplugins/in_tail/tail_file.ctests/runtime/in_tail.c
🧬 Code graph analysis (5)
plugins/in_tail/tail_fs_inotify.c (1)
plugins/in_tail/tail_file.c (1)
flb_tail_file_stat(93-101)
plugins/in_tail/tail_fs_stat.c (3)
plugins/in_tail/tail_file.c (1)
flb_tail_file_stat(93-101)src/flb_input.c (1)
flb_input_set_collector_time(1685-1704)plugins/in_tail/tail_fs_inotify.c (1)
tail_fs_event(166-311)
plugins/in_tail/tail.c (1)
plugins/in_tail/tail_file.c (1)
flb_tail_file_stat(93-101)
plugins/in_tail/tail_config.c (1)
src/flb_input.c (1)
flb_input_get_property(776-780)
tests/runtime/in_tail.c (1)
src/flb_lib.c (2)
flb_input_set(305-335)flb_output_set(520-551)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (31)
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_COVERAGE=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_COMPILER_STRICT_POINTER_TYPES=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_ARROW=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, clang, clang++)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-24.04, clang-14)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, gcc, g++)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-22.04, clang-12)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, clang, clang++)
- GitHub Check: PR - fuzzing test
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-22.04, clang-12)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-24.04, clang-14)
- GitHub Check: pr-compile-without-cxx (3.31.6)
- GitHub Check: pr-compile-centos-7
🔇 Additional comments (12)
plugins/in_tail/tail_config.h (1)
110-113: New config fields are well-placed and consistent with usageAdding
fstat_interval_nsecandkeep_file_handlehere is coherent with the rest of the config struct and matches the new config_map entries and initializers; no structural or ABI concerns within the plugin tree.plugins/in_tail/tail_fs_inotify.c (1)
259-272: Truncation handling matches keep_file_handle semanticsThe new branch correctly:
- re-seeks the existing FD and updates
file->offsetwhenkeep_file_handle==FLB_TRUE, and- resets
file->offsetto0whenkeep_file_handle==FLB_FALSE, leaving open/seek to later logic.Combined with the immediate
in_tail_collect_event()call, this should honor truncations in both modes without relying on a persistent handle.If you have an SMB/SAMBA environment handy, it’s worth running a real-world tail scenario with
keep_file_handle=falseand truncation to confirm behavior matches these tests under load.tests/runtime/in_tail.c (1)
2917-2919: Registering new keep_file_handle tests in TEST_LIST is consistentThe three new entries are correctly wired into
TEST_LISTand follow the existing naming convention, so they’ll be picked up by the runtime test harness.plugins/in_tail/tail.c (1)
81-89: Stat wrapper integration keeps collect paths working with closed handlesUsing
flb_tail_file_stat()in bothin_tail_collect_pendingandin_tail_collect_eventis the right abstraction now thatfile->fdcan legitimately be-1whenkeep_file_handle=false; it preserves previous behavior for persistent handles while making stat-based flows work for closed ones.Also applies to: 346-351
plugins/in_tail/tail_config.c (1)
109-111: Defaults for keep_file_handle and fstat_interval_nsec are sensibleInitializing
keep_file_handletoFLB_TRUEandfstat_interval_nsecto 250ms matches the config documentation and provides a safe, backward-compatible default for existing users.plugins/in_tail/tail_fs_stat.c (1)
64-79: Centralizing stat logic viaflb_tail_file_statlooks correctUsing
flb_tail_file_stat(file, &st)in bothtail_fs_eventandtail_fs_checkcorrectly abstracts over open/closed descriptors and keeps behavior consistent with the inotify backend. Error handling (logging viaflb_errno()or debug+remove) remains appropriate and localized.Also applies to: 102-107
plugins/in_tail/tail_file.c (6)
1452-1457: Closing the fd after append whenkeep_file_handle == FLB_FALSEmatches the new option’s intentClosing the file handle immediately after initialization for
keep_file_handle == FLB_FALSEensures the steady-state behavior does not keep descriptors open unnecessarily, while still allowingset_file_position()and initial DB offset setup to use an open fd.
1610-1653:adjust_counterstruncation logic is consistent with keep_file_handle semantics
adjust_counters()now:
- Uses
flb_tail_file_stat()to abstract stat/fstat.- Detects truncation via
size_delta < 0and:
- Seeks to 0 when
keep_file_handle == FLB_TRUE, updatingfile->offset.- Resets
file->offsetto 0 without touching the (closed) fd whenkeep_file_handle == FLB_FALSE.- Protects against negative
pending_byteswhen sizes briefly disagree.This matches the design of the new keep-handle option and avoids negative counters.
2014-2035:flb_tail_file_to_eventcorrectly adoptsflb_tail_file_statand guards rotation checksUsing
flb_tail_file_stat()for the pending-bytes check is consistent with the new helper, and gatingflb_tail_file_is_rotated()behindctx->keep_file_handle == FLB_TRUEis important: rotation detection relies on a stable open handle, which you intentionally do not provide whenkeep_file_handleis false.
2057-2190: Name resolution split into internal and public helpers is clean and keep_file_handle-awareThe refactor to:
flb_tail_file_name_internal(struct flb_tail_file *file)(OS-specific fd→path),- and the public
flb_tail_file_name(struct flb_tail_file *file)wrappergives you:
- A single place dealing with
/proc/.../fd,F_GETPATH, WindowsGetFinalPathNameByHandleA, etc.- Correct behavior when
file->fd == -1: the wrapper temporarily opens viaflb_tail_file_ensure_open_handle()and then closes again if it had to open.This works for both modes:
- For
keep_file_handle == FLB_TRUE,fd_was_openedremains false so the wrapper leaves the original open handle untouched.- For
keep_file_handle == FLB_FALSE, occasional name resolution still works without leaving descriptors open.
flb_tail_file_name_dup()also correctly uses the wrapper and cleans up on failure.
2293-2327: Purge-deleted-file logic now respects handle state viaflb_tail_file_stat
check_purge_deleted_file()now usesflb_tail_file_stat()so it works regardless of whether the fd is currently open. On stat failure you log and remove the file, and onst_nlink == 0you delete DB state (if present) and drop the file. This is aligned with the new handle semantics and avoids unnecessary fd opens for purge checks.
2347-2353: Rotated-file purge usesflb_tail_file_statbut preserves existing behaviorIn
flb_tail_file_purge(), swapping directstat()/fstat()calls forflb_tail_file_stat()keeps behavior the same forkeep_file_handle == FLB_TRUEand still allows stat-based logging for rotated files if the descriptor has already been closed. The subsequent removal and counting logic is unchanged.
david-garcia-garcia
force-pushed
the
11147c
branch
from
8feb31e to
1c8e51a
Compare
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
…nput Signed-off-by: David Garcia <[email protected]> Signed-off-by: deivid.garcia.garcia <[email protected]>
david-garcia-garcia
force-pushed
the
11147c
branch
from
1c8e51a to
d3ddfb1
Compare
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
david-garcia-garcia
temporarily deployed
to
pr
GitHub Actions
Inactive
|
Can you link the docs PR as well? |
|
@patrick-stephens added link to PR for docs in the PR description. |
2 participants
Adds new configuration options keep_file_handle and fstat_interval_nsec for tail input.
The purpose of this is to prevent file handles from being kept open while files are tailed. The motivation is to support tailing files that have a SMB/SAMBA underlying storage. SMB/SAMBA implementation will prevent tailed files from being deleted if a file handle is open.
Tailing files from SAMBA/SMB shares was fixed in #10405, but a pre existing file locking issue related to this storage backend was overlooked.
Why fstat_interval_nsec and not only keep_file_handle?
Because cloud storage is very sensitive to IOPS (depending on the storage you might be even billed by this metric), having control on how often fstat is checked has an impact.
NOTICE: when keep_file_handle is set to false, log rotation detection does not work as it requires an active handle to figure out where the original file was rotated to.
I tried to honor existing implementation as much as possible so as to not intrudce regresions when keep_file_handle=true (default).
Fixes #11147
Links of interest
https://learn.microsoft.com/en-us/rest/api/storageservices/managing-file-locks
https://www.samba.org/samba/docs/4.5/man-html/smb.conf.5.html
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
PR for docs: fluent/fluent-bit-docs#2180
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Summary by CodeRabbit
New Features
Refactor
Tests
✏️ Tip: You can customize this high-level summary in your review settings.