filter_lua: add new function prototype to support groups and metadata access

Prior to this change, the Lua filter in Fluent Bit supported processing of individual log
records with a function signature that allowed modifying the timestamp and the record only:

  function append_tag(tag, timestamp, record)
      ...
      return 1, timestamp, record
  end

This commit introduces an optional extended prototype that allows access to group metadata and
record metadata, making the function more powerful and better suited for structured formats
like OpenTelemetry Logs.

New function signature
----------------------

The new supported Lua function prototype is:

  function cb_metadata(tag, timestamp, group, metadata, record)
      ...

      return 1, timestamp, metadata, record
  end

Arguments:

- tag: the input tag of the log record.
- timestamp: the timestamp of the log record.
- group: a read-only table that contains group-level metadata (e.g., OpenTelemetry resource or
         scope info). This will be an empty table if the log is not part of a group.
- metadata: a table representing the record-specific metadata. You may modify this if needed.
- record: the actual log record table, same as in the original signature.

Return Values:

The function must return exactly 4 values, in the following order:

- Return Code:
  - 1: Record was modified.
  - 0: Record was not modified.
  - -1: Record should be dropped.
- Timestamp: The updated timestamp.
- Metadata Table: A new or modified metadata table.
- Record Table: A new or modified log record.

How Fluent Bit Chooses the Function Signature ?
-----------------------------------------------

At load time, the Lua filter analyzes the function signature by checking the number of
parameters. If the Lua function accepts:

- 3 arguments: it assumes the classic mode (tag, timestamp, record)
- 5 arguments: it uses the metadata-aware mode (tag, timestamp, group, metadata, record)

This ensures backward compatibility with existing Lua scripts.

Support for Returning Arrays (Multiple Records)
-----------------------------------------------

As with the original Lua callback design, the function may optionally return multiple records
as arrays.

When using the metadata-aware prototype, you must return:

  return 1, timestamp, {metadata_1, metadata_2, ...}, {record_1, record_2, ...}

Example:

  function cb_metadata(tag, ts, group, metadata, record)
      -- first record with its metadata
      m1 = {foo = "meta1"}
      r1 = {msg = "first log", old_record = record}

      -- second record with its metadata
      m2 = {foo = "meta2"}
      r2 = {msg = "second log", old_record = record}

      return 1, ts, {m1, m2}, {r1, r2}
  end

note: The metadata and record arrays must be the same length.

Final comments:

- Group metadata is read-only and should not be modified.
- If you don’t need group or metadata support, you can continue using the 3-argument prototype.
- Mixed return types (single record vs array) are supported, but must follow the proper structure.

Signed-off-by: Eduardo Silva <[email protected]>

Author: edsiper

plugins/filter_lua/CMakeLists.txt

@@ -8,6 +8,3 @@ else()
   FLB_PLUGIN(filter_lua "${src}" "m")
 endif()
 
-if(FLB_FILTER_LUA_USE_MPACK)
-    add_definitions(-DFLB_FILTER_LUA_USE_MPACK)
-endif()