How to use this package to AWS Lambda?

[cjnoname]

We’re interested in using this package within an AWS Lambda environment, but we’re unsure whether it’s compatible or how to properly deploy it.

Is there any documentation or guidance on how to get started with Lambda support for this package?

Thanks in advance!

[felixmagnus]

We're interested in this as well.

Currently, our lambdas error as soon as the '@confluentinc/kafka-javascript' package is imported.

This is the error:

INIT_REPORT Init Duration: 1652.67 ms	Phase: init	Status: error	Error Type: Runtime.Unknown

To me, it reads like the runtime of the lambda is incompatible with the package.

This is our lambda runtime configuration:

• Memory 1024 MB
• Runtime: Node.js 22.x
• Architecture: arm64

I can find Node 22 and arm64 on here, https://docs.confluent.io/kafka-clients/javascript/current/overview.html but I'm not sure if that matches lambdas architecture...

[milindl]

I have two preliminary suggestions:

1. is it possible to increase the verbosity of the available logging? The error is quite uninformative
2. Seeing this document https://docs.aws.amazon.com/lambda/latest/dg/nodejs-package.html#nodejs-package-create-dependencies it seems like you have to make a zip file with the installed node modules that's sent to Lambda. While installing, our package checks the current architecture to determine which prebuilt binary to fetch.
   So, for instance, you create your zip file on a macOS system with Node 18, it won't work on linux with Node 22. If this is the case, then you could use a docker image with the right OS, node version, and architecture to create the zip file for submission. I've seen similar errors earlier in different contexts.

[cjnoname]

Hi @milindl , thanks for the response. I hope there’s a guideline in the documentation that shows how to deploy to Lambda—either including the steps to create a layer or providing an official layer for people to use.

[felixmagnus]

Hi @milindl

is it possible to increase the verbosity of the available logging? The error is quite uninformative

Good suggestion, thanks! I increased the verbosity to the maximum possible for lambda. See attached logs at the end.

[...] So, for instance, you create your zip file on a macOS system with Node 18, it won't work on linux with Node 22. [...]

That's also a good hint, thanks a lot. All our lambdas are built in github actions, using the following runner configuration:

• OS: Ubuntu 24.04.2 LTS
• image: ubuntu-24.04 - 20250527.1.0
• Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20250527.1/images/ubuntu/Ubuntu2404-Readme.md
• Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20250527.1

We use the same node version at buildtime & at runtime (node 22).

I tried both possible lambda runtime architectures (arm64 and x86_64) and both do currently not work (and produce the same error).

I also tried both available bundling options for the lambda (CJS and ESM, ESM being our default) and they produce different errors currently.

Lambda logs (JSON format) for the ESM bundling (same for arm64 and x86_64):

{
    "time": "2025-06-02T08:33:13.231Z",
    "type": "platform.initStart",
    "record": {
        "initializationType": "on-demand",
        "phase": "invoke",
        "runtimeVersion": "nodejs:22.v43",
        "runtimeVersionArn": "arn:aws:lambda:eu-central-1::runtime:e4b97d4055c2afa813f5bda7d8e92ef971a7605f10be17ef7ae967743a21ba7d",
        "functionName": "<FUNCTION_NAME>",
        "functionVersion": "$LATEST"
    }
}
file:///var/task/index.mjs:431265
if (fileName !== __filename) {
^
ReferenceError: __filename is not defined
(Use `node --trace-uncaught ...` to show where the exception was thrown)
Node.js v22.15.0
{
    "time": "2025-06-02T08:33:16.034Z",
    "type": "platform.initRuntimeDone",
    "record": {
        "initializationType": "on-demand",
        "phase": "invoke",
        "status": "error",
        "errorType": "Runtime.ExitError"
    }
}
{
    "time": "2025-06-02T08:33:16.034Z",
    "type": "platform.initReport",
    "record": {
        "initializationType": "on-demand",
        "phase": "invoke",
        "status": "error",
        "errorType": "Runtime.ExitError",
        "metrics": {
            "durationMs": 2803.685
        }
    }
}
{
    "time": "2025-06-02T08:33:16.034Z",
    "type": "platform.start",
    "record": {
        "requestId": "2ea4f82b-5fda-483c-8ed8-427d5908c82a",
        "functionArn": "<FUNCTION_ARN>",
        "version": "$LATEST",
        "tracing": {
            "spanId": "69765a8e17d79988",
            "type": "X-Amzn-Trace-Id",
            "value": "Root=1-683d61c9-3a9685c768d1be1f76d7404a;Parent=27d52784c5996cbf;Sampled=1"
        }
    }
}
{
    "time": "2025-06-02T08:33:16.035Z",
    "type": "platform.runtimeDone",
    "record": {
        "requestId": "2ea4f82b-5fda-483c-8ed8-427d5908c82a",
        "status": "error",
        "errorType": "Runtime.ExitError",
        "metrics": {
            "durationMs": 0,
            "producedBytes": 0
        }
    }
}
{
    "time": "2025-06-02T08:33:16.035Z",
    "type": "platform.report",
    "record": {
        "requestId": "2ea4f82b-5fda-483c-8ed8-427d5908c82a",
        "metrics": {
            "durationMs": 2805.027,
            "billedDurationMs": 2806,
            "memorySizeMB": 1024,
            "maxMemoryUsedMB": 202
        },
        "tracing": {
            "spanId": "69765a8e17d79988",
            "type": "X-Amzn-Trace-Id",
            "value": "Root=1-683d61c9-3a9685c768d1be1f76d7404a;Parent=27d52784c5996cbf;Sampled=1"
        },
        "status": "error",
        "errorType": "Runtime.ExitError"
    }
}
{
    "time": "2025-06-02T08:33:16.063Z",
    "type": "platform.initStart",
    "record": {
        "initializationType": "on-demand",
        "phase": "init",
        "runtimeVersion": "nodejs:22.v43",
        "runtimeVersionArn": "arn:aws:lambda:eu-central-1::runtime:e4b97d4055c2afa813f5bda7d8e92ef971a7605f10be17ef7ae967743a21ba7d",
        "functionName": "FUNCTION_NAME",
        "functionVersion": "$LATEST"
    }
}
file:///var/task/index.mjs:431265
if (fileName !== __filename) {
^
ReferenceError: __filename is not defined
(Use `node --trace-uncaught ...` to show where the exception was thrown)
Node.js v22.15.0
{
    "time": "2025-06-02T08:33:17.668Z",
    "type": "platform.initRuntimeDone",
    "record": {
        "initializationType": "on-demand",
        "phase": "init",
        "status": "error",
        "errorType": "Runtime.ExitError"
    }
}
{
    "time": "2025-06-02T08:33:17.668Z",
    "type": "platform.initReport",
    "record": {
        "initializationType": "on-demand",
        "phase": "init",
        "status": "error",
        "errorType": "Runtime.ExitError",
        "metrics": {
            "durationMs": 1605.514
        }
    }
}

When looking at the bundled lambda's index.mjs, this is the code trace that produces the above error:

require_librdkafka -> require_bindings -> getFileName. Those are the relevant code parts:

// node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/librdkafka.js
var require_librdkafka = __commonJS({
  "node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/librdkafka.js"(exports, module) {
    var kafka = require_bindings()("confluent-kafka-javascript");
    module.exports = kafka;
  }
});

// node_modules/.pnpm/bindings@1.5.0/node_modules/bindings/bindings.js
var require_bindings = __commonJS({
  "node_modules/.pnpm/bindings@1.5.0/node_modules/bindings/bindings.js"(exports, module) {
    var fs = __require("fs");
    var path = __require("path");
    var fileURLToPath2 = require_file_uri_to_path();
    var join = path.join;
    var dirname = path.dirname;
    var exists = fs.accessSync && function(path2) {
      try {
        fs.accessSync(path2);
      } catch (e) {
        return false;
      }
      return true;
    } || fs.existsSync || path.existsSync;
    var defaults = {
      arrow: process.env.NODE_BINDINGS_ARROW || " \u2192 ",
      compiled: process.env.NODE_BINDINGS_COMPILED_DIR || "compiled",
      platform: process.platform,
      arch: process.arch,
      nodePreGyp: "node-v" + process.versions.modules + "-" + process.platform + "-" + process.arch,
      version: process.versions.node,
      bindings: "bindings.node",
      try: [
        // node-gyp's linked version in the "build" dir
        ["module_root", "build", "bindings"],
        // node-waf and gyp_addon (a.k.a node-gyp)
        ["module_root", "build", "Debug", "bindings"],
        ["module_root", "build", "Release", "bindings"],
        // Debug files, for development (legacy behavior, remove for node v0.9)
        ["module_root", "out", "Debug", "bindings"],
        ["module_root", "Debug", "bindings"],
        // Release files, but manually compiled (legacy behavior, remove for node v0.9)
        ["module_root", "out", "Release", "bindings"],
        ["module_root", "Release", "bindings"],
        // Legacy from node-waf, node <= 0.4.x
        ["module_root", "build", "default", "bindings"],
        // Production "Release" buildtype binary (meh...)
        ["module_root", "compiled", "version", "platform", "arch", "bindings"],
        // node-qbs builds
        ["module_root", "addon-build", "release", "install-root", "bindings"],
        ["module_root", "addon-build", "debug", "install-root", "bindings"],
        ["module_root", "addon-build", "default", "install-root", "bindings"],
        // node-pre-gyp path ./lib/binding/{node_abi}-{platform}-{arch}
        ["module_root", "lib", "binding", "nodePreGyp", "bindings"]
      ]
    };
    function bindings(opts) {
      if (typeof opts == "string") {
        opts = { bindings: opts };
      } else if (!opts) {
        opts = {};
      }
      Object.keys(defaults).map(function(i2) {
        if (!(i2 in opts)) opts[i2] = defaults[i2];
      });
      if (!opts.module_root) {
        opts.module_root = exports.getRoot(exports.getFileName());
      }
      if (path.extname(opts.bindings) != ".node") {
        opts.bindings += ".node";
      }
      var requireFunc = typeof __webpack_require__ === "function" ? __non_webpack_require__ : __require;
      var tries = [], i = 0, l = opts.try.length, n, b, err2;
      for (; i < l; i++) {
        n = join.apply(
          null,
          opts.try[i].map(function(p) {
            return opts[p] || p;
          })
        );
        tries.push(n);
        try {
          b = opts.path ? requireFunc.resolve(n) : requireFunc(n);
          if (!opts.path) {
            b.path = n;
          }
          return b;
        } catch (e) {
          if (e.code !== "MODULE_NOT_FOUND" && e.code !== "QUALIFIED_PATH_RESOLUTION_FAILED" && !/not find/i.test(e.message)) {
            throw e;
          }
        }
      }
      err2 = new Error(
        "Could not locate the bindings file. Tried:\n" + tries.map(function(a) {
          return opts.arrow + a;
        }).join("\n")
      );
      err2.tries = tries;
      throw err2;
    }
    module.exports = exports = bindings;
    exports.getFileName = function getFileName(calling_file) {
      var origPST = Error.prepareStackTrace, origSTL = Error.stackTraceLimit, dummy = {}, fileName;
      Error.stackTraceLimit = 10;
      Error.prepareStackTrace = function(e, st) {
        for (var i = 0, l = st.length; i < l; i++) {
          fileName = st[i].getFileName();
          if (fileName !== __filename) {
            if (calling_file) {
              if (fileName !== calling_file) {
                return;
              }
            } else {
              return;
            }
          }
        }
      };
      Error.captureStackTrace(dummy);
      dummy.stack;
      Error.prepareStackTrace = origPST;
      Error.stackTraceLimit = origSTL;
      var fileSchema = "file://";
      if (fileName.indexOf(fileSchema) === 0) {
        fileName = fileURLToPath2(fileName);
      }
      return fileName;
    };
    exports.getRoot = function getRoot(file) {
      var dir = dirname(file), prev;
      while (true) {
        if (dir === ".") {
          dir = process.cwd();
        }
        if (exists(join(dir, "package.json")) || exists(join(dir, "node_modules"))) {
          return dir;
        }
        if (prev === dir) {
          throw new Error(
            'Could not find module root given file: "' + file + '". Do you have a `package.json` file? '
          );
        }
        prev = dir;
        dir = join(dir, "..");
      }
    };
  }
});

Lambda logs (JSON format) for the CJS bundling (same for arm64 and x86_64):

{
    "time": "2025-06-02T14:37:23.727Z",
    "type": "platform.initStart",
    "record": {
        "initializationType": "on-demand",
        "phase": "init",
        "runtimeVersion": "nodejs:22.v43",
        "runtimeVersionArn": "arn:aws:lambda:eu-central-1::runtime:fd2e05b324f99edd3c6e17800b2535deb79bcce74b7506d595a94870b3d9bd2e",
        "functionName": "<FUNCTION_NAME>",
        "functionVersion": "$LATEST"
    }
}
{
    "timestamp": "2025-06-02T14:37:25.373Z",
    "level": "ERROR",
    "message": {
        "errorType": "Error",
        "errorMessage": "Could not find module root given file: \"/var/task/index.js\". Do you have a `package.json` file? ",
        "stackTrace": [
            "Error: Could not find module root given file: \"/var/task/index.js\". Do you have a `package.json` file? ",
            "    at Function.getRoot (/var/task/index.js:427074:17)",
            "    at bindings (/var/task/index.js:427001:37)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/librdkafka.js (/var/task/index.js:427088:35)",
            "    at __require (/var/task/index.js:9:50)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/lib/client.js (/var/task/index.js:427562:17)",
            "    at __require (/var/task/index.js:9:50)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/lib/kafka-consumer.js (/var/task/index.js:428182:18)",
            "    at __require (/var/task/index.js:9:50)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/lib/rdkafka.js (/var/task/index.js:429503:25)",
            "    at __require (/var/task/index.js:9:50)"
        ]
    }
}
{
    "time": "2025-06-02T14:37:25.469Z",
    "type": "platform.initRuntimeDone",
    "record": {
        "initializationType": "on-demand",
        "phase": "init",
        "status": "error",
        "errorType": "Runtime.Unknown"
    }
}
{
    "time": "2025-06-02T14:37:25.469Z",
    "type": "platform.initReport",
    "record": {
        "initializationType": "on-demand",
        "phase": "init",
        "status": "error",
        "errorType": "Runtime.Unknown",
        "metrics": {
            "durationMs": 1742.111
        }
    }
}
{
    "time": "2025-06-02T14:37:25.491Z",
    "type": "platform.initStart",
    "record": {
        "initializationType": "on-demand",
        "phase": "invoke",
        "runtimeVersion": "nodejs:22.v43",
        "runtimeVersionArn": "arn:aws:lambda:eu-central-1::runtime:fd2e05b324f99edd3c6e17800b2535deb79bcce74b7506d595a94870b3d9bd2e",
        "functionName": "<FUNCTION_NAME>",
        "functionVersion": "$LATEST"
    }
}
{
    "timestamp": "2025-06-02T14:37:27.903Z",
    "level": "ERROR",
    "message": {
        "errorType": "Error",
        "errorMessage": "Could not find module root given file: \"/var/task/index.js\". Do you have a `package.json` file? ",
        "stackTrace": [
            "Error: Could not find module root given file: \"/var/task/index.js\". Do you have a `package.json` file? ",
            "    at Function.getRoot (/var/task/index.js:427074:17)",
            "    at bindings (/var/task/index.js:427001:37)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/librdkafka.js (/var/task/index.js:427088:35)",
            "    at __require (/var/task/index.js:9:50)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/lib/client.js (/var/task/index.js:427562:17)",
            "    at __require (/var/task/index.js:9:50)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/lib/kafka-consumer.js (/var/task/index.js:428182:18)",
            "    at __require (/var/task/index.js:9:50)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/lib/rdkafka.js (/var/task/index.js:429503:25)",
            "    at __require (/var/task/index.js:9:50)"
        ]
    }
}
{
    "time": "2025-06-02T14:37:28.072Z",
    "type": "platform.initRuntimeDone",
    "record": {
        "initializationType": "on-demand",
        "phase": "invoke",
        "status": "error",
        "errorType": "Runtime.Unknown"
    }
}
{
    "time": "2025-06-02T14:37:28.073Z",
    "type": "platform.initReport",
    "record": {
        "initializationType": "on-demand",
        "phase": "invoke",
        "status": "error",
        "errorType": "Runtime.Unknown",
        "metrics": {
            "durationMs": 2581.309
        }
    }
}
{
    "time": "2025-06-02T14:37:28.073Z",
    "type": "platform.start",
    "record": {
        "requestId": "ab5fdd36-8d16-486c-8be2-7efb0c2ecca1",
        "functionArn": "<FUNCTION_ARN>",
        "version": "$LATEST",
        "tracing": {
            "spanId": "169d7dcf1f466182",
            "type": "X-Amzn-Trace-Id",
            "value": "Root=1-683db723-6a1d131b6e9d9ab108adf796;Parent=321f6f8ae8e37911;Sampled=1"
        }
    }
}
{
    "time": "2025-06-02T14:37:28.073Z",
    "type": "platform.runtimeDone",
    "record": {
        "requestId": "ab5fdd36-8d16-486c-8be2-7efb0c2ecca1",
        "status": "error",
        "errorType": "Runtime.Unknown",
        "metrics": {
            "durationMs": 0,
            "producedBytes": 0
        }
    }
}
{
    "time": "2025-06-02T14:37:28.073Z",
    "type": "platform.report",
    "record": {
        "requestId": "ab5fdd36-8d16-486c-8be2-7efb0c2ecca1",
        "metrics": {
            "durationMs": 2582.303,
            "billedDurationMs": 2583,
            "memorySizeMB": 1024,
            "maxMemoryUsedMB": 163
        },
        "tracing": {
            "spanId": "169d7dcf1f466182",
            "type": "X-Amzn-Trace-Id",
            "value": "Root=1-683db723-6a1d131b6e9d9ab108adf796;Parent=321f6f8ae8e37911;Sampled=1"
        },
        "status": "error",
        "errorType": "Runtime.Unknown"
    }
}
{
    "time": "2025-06-02T14:37:28.103Z",
    "type": "platform.initStart",
    "record": {
        "initializationType": "on-demand",
        "phase": "init",
        "runtimeVersion": "nodejs:22.v43",
        "runtimeVersionArn": "arn:aws:lambda:eu-central-1::runtime:fd2e05b324f99edd3c6e17800b2535deb79bcce74b7506d595a94870b3d9bd2e",
        "functionName": "<FUNCTION_NAME>",
        "functionVersion": "$LATEST"
    }
}
{
    "timestamp": "2025-06-02T14:37:29.507Z",
    "level": "ERROR",
    "message": {
        "errorType": "Error",
        "errorMessage": "Could not find module root given file: \"/var/task/index.js\". Do you have a `package.json` file? ",
        "stackTrace": [
            "Error: Could not find module root given file: \"/var/task/index.js\". Do you have a `package.json` file? ",
            "    at Function.getRoot (/var/task/index.js:427074:17)",
            "    at bindings (/var/task/index.js:427001:37)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/librdkafka.js (/var/task/index.js:427088:35)",
            "    at __require (/var/task/index.js:9:50)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/lib/client.js (/var/task/index.js:427562:17)",
            "    at __require (/var/task/index.js:9:50)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/lib/kafka-consumer.js (/var/task/index.js:428182:18)",
            "    at __require (/var/task/index.js:9:50)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/lib/rdkafka.js (/var/task/index.js:429503:25)",
            "    at __require (/var/task/index.js:9:50)"
        ]
    }
}
{
    "time": "2025-06-02T14:37:29.598Z",
    "type": "platform.initRuntimeDone",
    "record": {
        "initializationType": "on-demand",
        "phase": "init",
        "status": "error",
        "errorType": "Runtime.Unknown"
    }
}
{
    "time": "2025-06-02T14:37:29.598Z",
    "type": "platform.initReport",
    "record": {
        "initializationType": "on-demand",
        "phase": "init",
        "status": "error",
        "errorType": "Runtime.Unknown",
        "metrics": {
            "durationMs": 1494.505
        }
    }
}

For additional context, we use pnpm as a package manager.

The CJS error message "Could not find module root given file: \"/var/task/index.js\". Do you have a package.json file? " is (maybe?) understandable, since in our lambdas, we do not have a package.json (only the bundled index.js file for the function).

Hope that helps 👍

Would be great if you could find a solution for lambdas bundled with ESM. The architecture (arm64 or x86_64) is not that important to us.