pino-pretty logger doesn't work with Next 13 app directory

[scmx]

Summary

Hi! I've been using the pino json logging library with Next.js for a long time. And the pino-pretty extension for development.

When upgrading to Next 13 app folder I noticed that logging using pino with pino-pretty no longer works.

I've opened a pino-pretty issue about this. pinojs/pino-pretty#411 with a

Minimal reproduce repo https://github.com/scmx/next13-app-pino-pretty-worker-has-exited-minimal-reproduce

Error 1

error - node_modules/thread-stream/index.js (195:31) @ Worker.onWorkerExit error - uncaughtException: Error: the worker thread exited at Worker.onWorkerExit (webpack-internal:///(sc_server)/./node_modules/thread-stream/index.js:163:34) at Worker.emit (node:events:513:28) at Worker.[kOnExit] (node:internal/worker:287:10) at Worker..onexit (node:internal/worker:202:20) at Worker.callbackTrampoline (node:internal/async_hooks:130:17) null

Error 2

error - uncaughtException: Error: Cannot find module 'next13-app-pino-pretty/.next/server/app/lib/worker.js' at Function.Module._resolveFilename (node:internal/modules/cjs/loader:985:15) at Function.Module._load (node:internal/modules/cjs/loader:833:27) at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) at MessagePort. (node:internal/main/worker_thread:197:24) at MessagePort.[nodejs.internal.kHybridDispatch] (node:internal/event_target:736:20) at MessagePort.exports.emitMessage (node:internal/per_context/messageport:23:28) { code: 'MODULE_NOT_FOUND', requireStack: [] }

It seems to be something regarding SWC bundling that causes pino-pretty to no longer work.

mcollina commented Mar 10, 2023

This is due to https://getpino.io/#/docs/bundling.
I don't exactly know how to pass those options to Next.js (also with swc). If you figure it out send a PR!

Additional information

No response

Example

https://github.com/scmx/next13-app-pino-pretty-worker-has-exited-minimal-reproduce

[samyakb-mtechzilla]

I am also getting this error since Nextjs v13.1.1. Its for @logtail/pino package. This error is probably for all the pino packages that use transport

[zurgl]

@scmx Maybe this topic could help you
issue#1908

[scmx]

I worked around this by piping to pino-pretty instead.

npx next dev | pino-pretty

For example:

"dev": "next dev | pino-pretty -m message -i time -c"

[jokoso]

@scmx, do you know of a work-around for other transports, or did you not have to?

I'm facing this issue when setting up a pino/file transport:

const transport = pino.transport({
  target: 'pino/file',
  options: { destination: 'path/to/server.log' },
})
const logger = pino(transport)

Getting

- error uncaughtException: Error: Cannot find module './lib/worker.js'

and

- error node_modules/.pnpm/[email protected]/node_modules/thread-stream/index.js (195:31) @ Worker.onWorkerExit

[AaronLayton]

I just came across this as well - it seems my Pino logger will work from the app/page.tsx but not from an app/api/my-api/route.ts file

[yarinsa]

I followed the solution of using pipe like @scmx suggested. Hopefully it will be fixed.
I would like to have msg format of [LEVEL][MODULE][SCOPE] which I cant achieve using pino-prettyrc but only trough a programmatic usage

[francoisjacques]

Interesting. My nextjs app is using app router and I managed to get pino w/pino-pretty to work thru next-logger. I'm installing it from within instrumentation.ts

  if (process.env.NEXT_RUNTIME === "nodejs") {
    require("next-logger");
    // eslint-disable-next-line @typescript-eslint/no-var-requires
    const logger = require("next-logger/lib/logger");

And in next-logger.config.js (it has to be a CommonJS module)

// initPino is typed as `(config = {}) => pino.Logger` and is a helper to constructs our pino config, including the transport
const { initPino } = require("utils/logger");
module.exports = {
  logger: initPino 
};

Hope it can help anyone.

[yarinsa]

Interesting. My nextjs app is using app router and I managed to get pino w/pino-pretty to work thru next-logger. I'm installing it from within instrumentation.ts

  if (process.env.NEXT_RUNTIME === "nodejs") {

    require("next-logger");

    // eslint-disable-next-line @typescript-eslint/no-var-requires

    const logger = require("next-logger/lib/logger");

And in next-logger.config.js (it has to be a CommonJS module)

// initPino is typed as `(config = {}) => pino.Logger` and is a helper to constructs our pino config, including the transport

const { initPino } = require("utils/logger");

module.exports = {

  logger: initPino 

};

Hope it can help anyone.

Where do you use Pini pretty here?

[pierscin]

I made it work by adding pino to serverComponentsExternalPackages in next.config.mjs. I'm working only on local machine, but it might be worth trying.

https://nextjs.org/docs/app/api-reference/next-config-js/serverComponentsExternalPackages

[JunlinPan-Crypto]

It works. Saw same workaround in #54289 (comment)

[tmaihoff]

Adding pino to serverComponentsExternalPackages does not work for me.
Using nextjs 14 with app router, and still getting the MODULE_NOT_FOUND error

[teddybee]

Try out without pino-pretty(don't even include the package).
I have a successful build with the code below.

[therealonenak]

I am using

"next": "14.1.0"

here is logger

const pino = require('pino');

let logger = pino({
    transport: {
        target: 'pino-pretty',
        options: {
            colorize: true
        }
    }
});

module.exports = { logger };

here is next.config.mjs

/** @type {import('next').NextConfig} */
const nextConfig = {
    experimental:{
        serverComponentsExternalPackages: ['sequelize', 'pino', 'pino-pretty']
    }
};

export default nextConfig;

And it is working fine.

[maelp]

This works for my, but I then get this warning:
⚠ Invalid next.config.mjs options detected:
⚠ Unrecognized key(s) in object: 'serverComponentsExternalPackages' at "experimental"
⚠ See more info here: https://nextjs.org/docs/messages/invalid-next-config
⚠ experimental.serverComponentsExternalPackages has been moved to serverExternalPackages. Please update your next.config.mjs file accordingly.

although it removes the error in the NextJS devtool

[ruuuruiya]

// in next.config.js
/** @type {import('next').NextConfig} */
const nextConfig = {
  experimental: {
    appDir: true,
  },
  webpack: (config) => {
    config.externals.push("pino-pretty", "lokijs", "encoding");
    return config;
  },
};

[dariodjuric]

I was able to have it running like this:

import pino from "pino";
import { IS_PRODUCTION, LOG_LEVEL } from "./environment";
import pinoPretty from "pino-pretty";

const logger = pino(
  {
    level: LOG_LEVEL,
  },
  !IS_PRODUCTION
    ? pinoPretty({
      colorize: true,
    })
    : undefined,
);

[magnusriga]

I am using the recommended setup above (#46987 (comment)), but getting the same Cannot find module error:

Error: Cannot find module '/home/........../.next/server/chunks/lib/worker.js'
    at Function._resolveFilename (node:internal/modules/cjs/loader:1219:15)
    at Function._load (node:internal/modules/cjs/loader:1045:27)
    at TracingChannel.traceSync (node:diagnostics_channel:315:14)
    at wrapModuleLoad (node:internal/modules/cjs/loader:215:24)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:158:5)
    at MessagePort.<anonymous> (node:internal/main/worker_thread:185:26)
    at [nodejs.internal.kHybridDispatch] (node:internal/event_target:816:20)
    at MessagePort.<anonymous> (node:internal/per_context/messageport:23:28) {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Node.js v22.5.1

> Build error occurred
Error: Jest worker encountered 4 child process exceptions, exceeding retry limit
    at ChildProcessWorker.initialize (/home/........./node_modules/.pnpm/[email protected]_@[email protected][email protected][email protected][email protected][email protected][email protected]/node_modules/next/dist/compiled/jest-worker/index.js:1:11580)
    at ChildProcessWorker._onExit (/home/......../node_modules/.pnpm/[email protected]_@[email protected][email protected][email protected][email protected][email protected][email protected]/node_modules/next/dist/compiled/jest-worker/index.js:1:12545)
    at ChildProcess.emit (node:events:520:28)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
  type: 'WorkerError'
}

Removing the pino transport makes the error go away.

Anyone experiencing the same? Is there a fix?

[vktrl]

Check out pino bundling doc. It looks like worker.js is part of thread-stream-worker, try marking that external.

[magnusriga]

@vktrl Thank you, that pointed me in the right direction. I finally figured it out. Below is the full setup.

[magnusriga]

Finally, after a lot of trial-and-error I figured out how to make pino work in a Next.js monorepo (Turborepo), with transports, pino-pretty, and functions for messageFormat, customPrettifier, and so on (i.e. non-streamable types).

EDIT:

• The below works build-time, but at runtime the application cannot find the pino module. Which I guess makes sense, since we are asking webpack to exclude it from the build.
• As such, as of now, it is not a valid solution.
• I tried use the pino-webpack-plugin, but now getting a new runtime error: TypeError: Cannot read properties of null (reading 'fn') is pino is in config.externals, or No template for dependency: CommonJsRequireDependency if pino is not added to config.externals.
• Trying to figure out what's going on and if there is a fix.

The setup:

@workspace/apps/appname/next.config.mjs

• A lot of guides talk about adding pino and pino-pretty to serverExternalPackages (or experimental.serverComponentsExternalPackages if on v14). That does not work if also using config.externals.push in the webpack callback.
• So, comment out serverExternalPackages and stick to only using config.externals.push, since the latter allows including node:crypto (among others) and the former did not.
• If you include both, it seems config.externals.push overwrites serverExternalPackages, and thus it is not enough to include pino and pino-pretty in serverExternalPackages. Here is the relevant code:

  webpack(
    config,
    // { buildId, dev, isServer, defaultLoaders, nextRuntime, webpack },
  ) {
    config.resolve.fallback = {
      ...config.resolve.fallback,

      fs: false, // the solution
    };

    // NOTE: Must use this instead of serverExternalPackages | serverComponentsExternalPackages
    // because node:crypto for some reason only works below.
    // Also, when the below is defined, it overrides the serverExternalPackages,
    // so pino and pino-pretty must be included here.
    config.externals.push({
      'node:fs': 'commonjs fs',
      'node:crypto': 'commonjs crypto',
      argon2: 'argon2',
      pino: 'pino',
      'thread-stream': 'thread-stream',
      'pino-worker': 'pino-worker',
      'pino-file': 'pino-file',
      'pino-pretty': 'pino-pretty',
    });
    return config;
  },

@workspace/logger/pino.ts

• The important part here, is to add the right path to target. I could not make a relative path work, so I landed on using the package from node_modules this way: @<workspace>/logger/transports/pino-pretty-transport.
• If you are not using a monorepo, I guess it will be easier to just do a relative import within the local project.

import pino from 'pino';

const transport = pino.transport({
  targets: [
    {
      target: 'pino/file',
      options: { destination: `./logs/pino.log` },
      level: 'warn',
    },
    {
      target: `@<workspace>/logger/transports/pino-pretty-transport`,
      options: {
        colorize: true,
      },
    },
    {
      target: '@logtail/pino',
      options: { sourceToken: process.env.LOGTAIL_SOURCE_TOKEN },
    },
  ],
  options: {
    colorize: true,
  },
});

export const logger = pino(
  {
    level: process.env.NODE_ENV === 'production' ? 'warn' : 'trace',
    // redact,
   // other options
  },
  transport,
);

@workspace/logger/ transports/pino-pretty-transport.mjs

• The pino-pretty transport file had to be mjs, I could not make it work as .ts.

import pretty from 'pino-pretty';

function getPrettyStream(options) {
  return pretty({
    ...options,
    colorize: true,
    translateTime: 'SYS:standard',
    // ignore: 'pid,hostname',
    // levelFirst: true,
    // messageKey: 'msg',
    // timestampKey: 'time',
    // levelKey: 'level',

    // messageFormat: (log, messageKey, levelLabel, { colors }) => {
    //   // `colors` is a Colorette object with colors enabled based on `colorize` option.
    //   const req = log['req'];
    //   const reqUrl = req instanceof Request ? req.url : null;

    //   return `This is a ${colors.red('colorized')}, custom message: ${log[messageKey]}, ${levelLabel} ${log['pid'] ? ` - ${log['pid']}` : ''} - ${reqUrl ? ` - url: ${reqUrl}` : ''}${log[''] ? ` - ${log['pid']}` : ''}`;
    // },

    customPrettifiers: {
      // The argument for this function will be the same
      // string that's at the start of the log-line by default:
      timestamp: (timestamp) => `🕰 ${timestamp}`,

      // The argument for the level-prettifier may vary depending
      // on if the levelKey option is used or not.
      // By default this will be the same numerics as the Pino default:
      // level: (logLevel) => `LEVEL: ${logLevel}`,

      // level provides additional data in `extras`:
      // * label => derived level label string
      // * labelColorized => derived level label string with colorette colors applied based on customColors and whether colors are supported
      // level: (logLevel, key, log, { label, labelColorized, colors }) =>
      //   `LEVEL: ${logLevel} LABEL: ${label} COLORIZED LABEL: ${labelColorized}`,

      hostname: (hostname) => `MY HOST: ${hostname}`,

      pid: (pid) => `${pid}`,

      name: (name, key, log, { colors }) => `${colors.blue(String(name))}`,

      caller: (caller, key, log, { colors }) =>
        `${colors.greenBright(String(caller))}`,

      myCustomLogProp: (value, key, log, { colors }) =>
        `My Prop -> ${colors.bold(String(value))} <--`,
    },
  });
}

export default (options) => {
  const prettyStream = getPrettyStream(options);
  return prettyStream;
};

@workspace/logger/ transports/package.json

• Make sure to export the right files:

{
  "exports": {
    "./pino": "./src/pino.ts",
    "./transports/*": "./src/transports/*.mjs"
  },
}

@workspace/apps/appname/anyfile.ts

• To use the logger:

import { logger } from '@workspace/logger/pino';

logger.error(new Error('foobar'));

I think that is it. It fully works on my end with that setup.

[AnastasiiaRoch]

Just this line in the next.config will also work:

webpack(config) {
        ...

        // resolve not found .next/worker.js issue
        config.externals.push({ 'thread-stream': 'commonjs thread-stream', pino: 'commonjs pino' });

        return config;
    },

[rishabhsxn]

@AnastasiiaRoch Thank you so much for sharing!
This worked for me.

For anyone trying to understand why this works:
The issue was occurring because Next.js was trying to bundle Pino and its dependencies (including thread-stream) incorrectly for server components.

By adding this configuration we are telling Nextjs to treat these modules as External dependencies, not include in the bundle and use CommonJS version.
The logger will now be directly loaded from node_modules in the server.

[magnusriga]

That setting will allow you to build the application without error, but importing the pino module will fail at runtime, as the pino module is no longer accessible.

It makes sense that pino cannot be found, as the module is excluded from the webpack bundle, and that bundle is stored server-side after building.

AFAIK, there is no separate node_modules outside the bundle on the server. Just make a standalone build and peak into the built directory to see what it looks like post build.

I might be wrong about the above, but that is my current understanding.

PS: The bundle built with next build should not be confused with the React Server Component Payload sent to the client, that is a different bundle altogether.

Other threads on this topic:

pinojs/pino#1964 (comment)

pinojs/pino#2028 (comment)

[maelp]

Any update on this? Still not working for me, does a weird error for a missing "path" variable?

[maelp]