log: add system_config option to force stacktrace level (#5008)

**Which issue(s) this PR fixes**: 
None.

**What this PR does / why we need it**: 

**What this PR does**

Add a new system_config option: `forced_stacktrace_level`.

* Option: `system/log/forced_stacktrace_level`
* Type: `enum`
  * `none`, `trace`, `debug`, `info`, `warn`, `error`, `fatal`
* Default: `none`

By default, the behavior of Fluentd does not change.
So, this does not affect existing users.

If you set this option and change the value, the log levels of
stacktraces are forced to that value.

Stacktraces that do not meet the log level are discarded in advance.
After the log level is changed, it is judged again for logging or
discarded finally.
For example, by default (`log_level info`), ...

* All `trace`/`debug` stacktraces are not logged regardless of this
option.
* If you set `forced_stacktrace_level debug`, then all stacktraces are
not logged.

This is because we should avoid logging logs that contradict
`log_level`, and to avoid unexpectedly logging logs that are initially
`trace`/`debug` level.

Setting example:

```xml
<system>
  <log>
    forced_stacktrace_level info
  </log>
</system>
```

**why we need it**

To make it easier to exclude stacktraces from monitoring Fluentd's own
logs.
(You don't need this feature if using
[@FLUENT_LOG](https://docs.fluentd.org/deployment/logging#capture-fluentd-logs)
to transfer. For some reason, there are users who need to monitor
Fluentd's log files directly. This feature is for those uses.)

For example, there could be a system that raises an alert when an
error-level log is detected.
You may want to exclude some error-level logs from the detection.
However, if stacktraces are logged at error level, you need to exclude
those stacktraces as well.
With the current specification, that is troublesome.

Ideally, it would be preferable to exclude all stacktraces from the
detection.
It is sufficient to have access to stacktraces when investigating the
details.

An easy way to improve the current situation is this feature.
If you are detecting error logs as in the previous example, you can
easily exclude all stacktraces from the detection by forcing the
stacktrace level to `info` with this feature.

**Docs Changes**:
* fluent/fluentd-docs-gitbook#589

**Release Note**: 
System configuration: Add `forced_stacktrace_level` to force the log
level of stacktraces.

**Sample**:

```xml
<system>
  <log>
    forced_stacktrace_level info
  </log>
</system>

<source>
  @type sample
  @id test
  tag test.fail
</source>

<match test.fail>
  @type null
  never_flush
</match>
```

Without `forced_stacktrace_level info`:

```
2025-06-23 18:16:45 +0900 [info]: #0 fluent/log.rb:371:info: fluentd worker is now running worker=0
2025-06-23 18:16:46 +0900 [warn]: #0 fluent/log.rb:392:warn: emit transaction failed: error_class=RuntimeError error="failed to flush" location="/home/daipom/work/fluentd/fluentd/lib/fluent/plugin/out_null.rb:54:in `process'" tag="test.fail"
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/out_null.rb:54:in `process'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/output.rb:865:in `emit_sync'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/event_router.rb:115:in `emit_stream'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/event_router.rb:106:in `emit'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:115:in `block in emit'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:115:in `times'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:115:in `emit'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:100:in `run'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
```

With `forced_stacktrace_level info`:

```
2025-06-23 18:13:59 +0900 [warn]: #0 fluent/log.rb:392:warn: emit transaction failed: error_class=RuntimeError error="failed to flush" location="/home/daipom/work/fluentd/fluentd/lib/fluent/plugin/out_null.rb:54:in `process'" tag="test.fail"
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/out_null.rb:54:in `process'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/output.rb:865:in `emit_sync'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/event_router.rb:115:in `emit_stream'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/event_router.rb:106:in `emit'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:115:in `block in emit'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:115:in `times'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:115:in `emit'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:100:in `run'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
```

Signed-off-by: Daijiro Fukuda <[email protected]>

Author: daipom

lib/fluent/log.rb

@@ -138,6 +138,7 @@ def initialize(logger, opts={})
       @optional_attrs = nil
 
       @suppress_repeated_stacktrace = opts[:suppress_repeated_stacktrace]
+      @forced_stacktrace_level = nil
       @ignore_repeated_log_interval = opts[:ignore_repeated_log_interval]
       @ignore_same_log_interval = opts[:ignore_same_log_interval]
 
@@ -240,6 +241,14 @@ def reopen!
       nil
     end
 
+    def force_stacktrace_level?
+      not @forced_stacktrace_level.nil?
+    end
+
+    def force_stacktrace_level(level)
+      @forced_stacktrace_level = level
+    end
+
     def enable_debug(b=true)
       @debug_mode = b
       self
@@ -500,6 +509,16 @@ def suppress_stacktrace?(backtrace)
     def dump_stacktrace(type, backtrace, level)
       return if @level > level
 
+      dump_stacktrace_internal(
+        type,
+        backtrace,
+        force_stacktrace_level? ? @forced_stacktrace_level : level,
+      )
+    end
+
+    def dump_stacktrace_internal(type, backtrace, level)
+      return if @level > level
+
       time = Time.now
 
       if @format == :text

lib/fluent/supervisor.rb

@@ -908,6 +908,7 @@ def setup_global_logger(supervisor: false)
         ignore_repeated_log_interval: system_config.ignore_repeated_log_interval,
         ignore_same_log_interval: system_config.ignore_same_log_interval,
       )
+      $log.force_stacktrace_level(system_config.log.forced_stacktrace_level) if system_config.force_stacktrace_level?
       $log.enable_color(false) if actual_log_path
       $log.enable_debug if system_config.log_level <= Fluent::Log::LEVEL_DEBUG
 

lib/fluent/system_config.rb

@@ -79,6 +79,7 @@ class SystemConfig
         end
       end
       config_param :rotate_size, :size, default: nil
+      config_param :forced_stacktrace_level, :enum, list: [:none, :trace, :debug, :info, :warn, :error, :fatal], default: :none
     end
 
     config_section :counter_server, multi: false do
@@ -118,6 +119,10 @@ class SystemConfig
       config_param :compress, :enum, list: [:text, :gzip], default: nil
     end
 
+    def force_stacktrace_level?
+      @log.forced_stacktrace_level != :none
+    end
+
     def self.create(conf, strict_config_value=false)
       systems = conf.elements(name: 'system')
       return SystemConfig.new if systems.empty?
@@ -162,6 +167,7 @@ def configure(conf, strict_config_value=false)
       end
 
       @log_level = Log.str_to_level(@log_level.to_s) if @log_level
+      @log[:forced_stacktrace_level] = Log.str_to_level(@log.forced_stacktrace_level.to_s) if force_stacktrace_level?
     end
 
     def dup

test/config/test_system_config.rb

@@ -82,6 +82,7 @@ def parse_text(text)
       assert_nil(sc.log.path)
       assert_equal(:text, sc.log.format)
       assert_equal('%Y-%m-%d %H:%M:%S %z', sc.log.time_format)
+      assert_equal(:none, sc.log.forced_stacktrace_level)
     end
 
     data(
@@ -124,6 +125,7 @@ def parse_text(text)
               path /tmp/fluentd.log
               format json
               time_format %Y
+              forced_stacktrace_level info
             </log>
           </system>
       EOS
@@ -133,6 +135,7 @@ def parse_text(text)
       assert_equal('/tmp/fluentd.log', sc.log.path)
       assert_equal(:json, sc.log.format)
       assert_equal('%Y', sc.log.time_format)
+      assert_equal(Fluent::Log::LEVEL_INFO, sc.log.forced_stacktrace_level)
     end
 
     # info is removed because info level can't be specified via command line

test/test_log.rb

@@ -386,6 +386,85 @@ def test_different_log_level
     end
   end
 
+  sub_test_case "force_stacktrace_level" do
+    data(
+      none: [ nil, ["trace", "debug", "info", "warn", "error", "fatal"] ],
+      trace: [ Fluent::Log::LEVEL_TRACE, ["trace", "trace", "trace", "trace", "trace", "trace"] ],
+      debug: [ Fluent::Log::LEVEL_DEBUG, ["debug", "debug", "debug", "debug", "debug", "debug"] ],
+      info: [ Fluent::Log::LEVEL_INFO, ["info", "info", "info", "info", "info", "info"] ],
+      warn: [ Fluent::Log::LEVEL_WARN, ["warn", "warn", "warn", "warn", "warn", "warn"] ],
+      error: [ Fluent::Log::LEVEL_ERROR, ["error", "error", "error", "error", "error", "error"] ],
+      fatal: [ Fluent::Log::LEVEL_FATAL, ["fatal", "fatal", "fatal", "fatal", "fatal", "fatal"] ],
+    )
+    test "level should be forced" do |(level, expected)|
+      backtrace = ["backtrace"]
+      logger = Fluent::Log.new(
+        ServerEngine::DaemonLogger.new(
+          @log_device,
+          log_level: ServerEngine::DaemonLogger::TRACE,
+        )
+      )
+      logger.force_stacktrace_level(level) unless level.nil?
+
+      logger.trace_backtrace(backtrace)
+      logger.debug_backtrace(backtrace)
+      logger.info_backtrace(backtrace)
+      logger.warn_backtrace(backtrace)
+      logger.error_backtrace(backtrace)
+      logger.fatal_backtrace(backtrace)
+
+      assert do
+        expected == logger.out.logs.map { |log| log.match(/ \[([a-z]+)\]: backtrace$/)[1] }
+      end
+    end
+
+    test "stacktraces that do not meet log_level initially should be discarded" do
+      logger = Fluent::Log.new(
+        ServerEngine::DaemonLogger.new(
+          @log_device,
+          log_level: ServerEngine::DaemonLogger::INFO,
+        )
+      )
+      logger.force_stacktrace_level(Fluent::Log::LEVEL_INFO)
+
+      logger.trace_backtrace(["trace"])
+      logger.debug_backtrace(["debug"])
+      logger.info_backtrace(["info"])
+      logger.warn_backtrace(["warn"])
+      logger.error_backtrace(["error"])
+      logger.fatal_backtrace(["fatal"])
+
+      assert_equal(
+        [
+          "  #{@timestamp_str} [info]: info\n",
+          "  #{@timestamp_str} [info]: warn\n",
+          "  #{@timestamp_str} [info]: error\n",
+          "  #{@timestamp_str} [info]: fatal\n",
+        ],
+        logger.out.logs,
+      )
+    end
+
+    test "stacktraces that do not meet log_level finally should be discarded" do
+      logger = Fluent::Log.new(
+        ServerEngine::DaemonLogger.new(
+          @log_device,
+          log_level: ServerEngine::DaemonLogger::INFO,
+        )
+      )
+      logger.force_stacktrace_level(Fluent::Log::LEVEL_DEBUG)
+
+      logger.trace_backtrace(["trace"])
+      logger.debug_backtrace(["debug"])
+      logger.info_backtrace(["info"])
+      logger.warn_backtrace(["warn"])
+      logger.error_backtrace(["error"])
+      logger.fatal_backtrace(["fatal"])
+
+      assert_equal([], logger.out.logs)
+    end
+  end
+
   sub_test_case "ignore_repeated_log_interval" do
     def test_same_message
       message = "This is test"

test/test_supervisor.rb

@@ -715,6 +715,7 @@ def test_init_for_logger(supervisor)
           <log>
             format json
             time_format %FT%T.%L%z
+            forced_stacktrace_level info
           </log>
         </system>
       EOC
@@ -728,6 +729,8 @@ def test_init_for_logger(supervisor)
       assert_equal false, $log.suppress_repeated_stacktrace
       assert_equal 10, $log.ignore_repeated_log_interval
       assert_equal 20, $log.ignore_same_log_interval
+      assert_equal Fluent::Log::LEVEL_INFO, $log.instance_variable_get(:@forced_stacktrace_level)
+      assert_true $log.force_stacktrace_level?
     end
 
     data(