Next.JS 14.2.23 - Importing @aws-sdk/crc64-nvme-crt" as required but fails to fetch S3 objects.

[arealesramirez]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

I have a project in Next.JS 14.2.23 facing the following error using NodeJS v20.10.0

Error: Please check whether you have installed the "@aws-sdk/crc64-nvme-crt" package explicitly. 
You must also register the package by calling [require("@aws-sdk/crc64-nvme-crt");] or an ESM equivalent such as [import "@aws-sdk/crc64-nvme-crt";]. 
For more information please go to https://github.com/aws/aws-sdk-js-v3#functionality-requiring-aws-common-runtime-crt
    at selectChecksumAlgorithmFunction (/Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:209:15)
    at validateChecksumFromResponse (/Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:411:35)
    at /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:466:11
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:174:20
    at async /Users/andres/Repos/my-project/node_modules/@smithy/middleware-serde/dist-cjs/index.js:33:24
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:483:18
    at async /Users/andres/Repos/my-project/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:109:22
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:136:14
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
    at async getFileStreamFromS3 (webpack-internal:///(rsc)/./services/s3Service.ts:57:22)
    at async GET (webpack-internal:///(rsc)/./app/api/submissions/[submissionId]/file/route.ts:34:28)
    at async /Users/andres/Repos/my-project/node_modules/next/dist/compiled/next-server/app-route.runtime.dev.js:6:57228
    at async eT.execute (/Users/andres/Repos/my-project/node_modules/next/dist/compiled/next-server/app-route.runtime.dev.js:6:46851)
    at async eT.handle (/Users/andres/Repos/my-project/node_modules/next/dist/compiled/next-server/app-route.runtime.dev.js:6:58760)
    at async doRender (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1366:42)
    at async cacheEntry.responseCache.get.routeKind (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1576:40)
    at async DevServer.renderToResponseWithComponentsImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1496:28)
    at async DevServer.renderPageComponent (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1924:24)
    at async DevServer.renderToResponseImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1962:32)
    at async DevServer.pipeImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:922:25)
    at async NextNodeServer.handleCatchallRenderRequest (/Users/andres/Repos/my-project/node_modules/next/dist/server/next-server.js:272:17)
    at async DevServer.handleRequestImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:818:17)
    at async /Users/andres/Repos/my-project/node_modules/next/dist/server/dev/next-dev-server.js:339:20
    at async Span.traceAsyncFn (/Users/andres/Repos/my-project/node_modules/next/dist/trace/trace.js:154:20)
    at async DevServer.handleRequest (/Users/andres/Repos/my-project/node_modules/next/dist/server/dev/next-dev-server.js:336:24)
    at async invokeRender (/Users/andres/Repos/my-project/node_modules/next/dist/server/lib/router-server.js:173:21)
    at async handleRequest (/Users/andres/Repos/my-project/node_modules/next/dist/server/lib/router-server.js:350:24)
    at async requestHandlerImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/lib/router-server.js:374:13)
    at async Server.requestListener (/Users/andres/Repos/my-project/node_modules/next/dist/server/lib/start-server.js:141:13) {
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}

This error occurs when I trigger the getFileStreamFromS3 function, which is a straightforward function that fetches an object from s3

import {
  GetObjectCommand,
  PutObjectCommand,
  PutObjectCommandOutput,
  S3Client,
} from "@aws-sdk/client-s3";
import "@ungap/with-resolvers";
import { pdf } from "@/utils/pdf-to-img";
const s3Client = new S3Client({
  region: process.env.AWS_REGION,
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  },
});


export const getFileStreamFromS3 = async (
  key: string
): Promise<ReadableStream> => {
  // await import("@aws-sdk/crc64-nvme-crt");

  const command = new GetObjectCommand({
    Bucket: process.env.AWS_S3_BUCKET_NAME!,
    Key: key,
    ChecksumMode: "ENABLED",
  });

  const response = await s3Client.send(command);

  if (!response.Body) {
    throw new Error("No body returned from S3");
  }

  return response.Body as ReadableStream;
};

// + other functions

According to the error message, I need to install and import @aws-sdk/crc64-nvme-crt, which I did

import "@aws-sdk/crc64-nvme-crt";

However, the error keeps showing up.

Here are the package versions I'm using:

        "@aws-sdk/client-s3": "^3.705.0",
        "@aws-sdk/crc64-nvme-crt": "^3.730.0",

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/client-s3@3.705.0, @aws-sdk/crc64-nvme-crt": "^3.730.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.10.0

Reproduction Steps

Trigger the following function

export const getFileStreamFromS3 = async (
  key: string
): Promise<ReadableStream> => {
  // await import("@aws-sdk/crc64-nvme-crt");

  const command = new GetObjectCommand({
    Bucket: process.env.AWS_S3_BUCKET_NAME!,
    Key: key,
    ChecksumMode: "ENABLED",
  });

  const response = await s3Client.send(command);

  if (!response.Body) {
    throw new Error("No body returned from S3");
  }

  return response.Body as ReadableStream;
};

And it fails after triggering

  const response = await s3Client.send(command);

Observed Behavior

Here is the error thrown

Error: Please check whether you have installed the "@aws-sdk/crc64-nvme-crt" package explicitly. 
You must also register the package by calling [require("@aws-sdk/crc64-nvme-crt");] or an ESM equivalent such as [import "@aws-sdk/crc64-nvme-crt";]. 
For more information please go to https://github.com/aws/aws-sdk-js-v3#functionality-requiring-aws-common-runtime-crt
    at selectChecksumAlgorithmFunction (/Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:209:15)
    at validateChecksumFromResponse (/Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:411:35)
    at /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:466:11
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:174:20
    at async /Users/andres/Repos/my-project/node_modules/@smithy/middleware-serde/dist-cjs/index.js:33:24
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:483:18
    at async /Users/andres/Repos/my-project/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:109:22
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:136:14
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
    at async getFileStreamFromS3 (webpack-internal:///(rsc)/./services/s3Service.ts:57:22)
    at async GET (webpack-internal:///(rsc)/./app/api/submissions/[submissionId]/file/route.ts:34:28)
    at async /Users/andres/Repos/my-project/node_modules/next/dist/compiled/next-server/app-route.runtime.dev.js:6:57228
    at async eT.execute (/Users/andres/Repos/my-project/node_modules/next/dist/compiled/next-server/app-route.runtime.dev.js:6:46851)
    at async eT.handle (/Users/andres/Repos/my-project/node_modules/next/dist/compiled/next-server/app-route.runtime.dev.js:6:58760)
    at async doRender (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1366:42)
    at async cacheEntry.responseCache.get.routeKind (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1576:40)
    at async DevServer.renderToResponseWithComponentsImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1496:28)
    at async DevServer.renderPageComponent (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1924:24)
    at async DevServer.renderToResponseImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1962:32)
    at async DevServer.pipeImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:922:25)
    at async NextNodeServer.handleCatchallRenderRequest (/Users/andres/Repos/my-project/node_modules/next/dist/server/next-server.js:272:17)
    at async DevServer.handleRequestImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:818:17)
    at async /Users/andres/Repos/my-project/node_modules/next/dist/server/dev/next-dev-server.js:339:20
    at async Span.traceAsyncFn (/Users/andres/Repos/my-project/node_modules/next/dist/trace/trace.js:154:20)
    at async DevServer.handleRequest (/Users/andres/Repos/my-project/node_modules/next/dist/server/dev/next-dev-server.js:336:24)
    at async invokeRender (/Users/andres/Repos/my-project/node_modules/next/dist/server/lib/router-server.js:173:21)
    at async handleRequest (/Users/andres/Repos/my-project/node_modules/next/dist/server/lib/router-server.js:350:24)
    at async requestHandlerImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/lib/router-server.js:374:13)
    at async Server.requestListener (/Users/andres/Repos/my-project/node_modules/next/dist/server/lib/start-server.js:141:13) {
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}

Expected Behavior

I expected to fetch the file/object from S3

Possible Solution

No response

Additional Information/Context

No response

[trivikr]

This issue happens as the S3 objects were uploaded with CRC64NVME checksums - either by another JS SDK application which had the required additional @aws-sdk/crc64-nvme-crt dependency, or using another SDK.

The S3 backend returns the CRC64NVME checksum, and the JS SDK fails since the optional dependency is not available.

[trivikr]

We're discussing internally on how the behavior should be.

Ideally, the JS SDK should skip validating the checksum for get operations if the dependency is not available. It's because the application did not explicitly request checksum to be validate using CRC64-NVME algorithm.

The fix is likely to be released on Tuesday Jan 21st or Wednesday Jan 22nd.

For a quick fix, you can pin the SDK version to <3.729.0 in your package.json.
I'll create a minimal repro and provide another workaround which required adding a middleware in the following comment.

[arealesramirez]

I found out the following:

The project is in a monorepo structure in the following way:
monorepo
-> packages
-> nextjsapp
-> lambda

The two packages, nextjsapp and lambda use @aws-sdk/client-s3 library.

For the NextJS app, the version in package.json was: "@aws-sdk/client-s3": "^3.705.0"
For a Lambda, the version found out in package.json was: "@aws-sdk/client-s3": "^3.723.0",

Now, at the moment of installing the packages, the package installed was "@aws-sdk/client-s3": "3.730.0". Which means that's the version that is used in both packages.

This leads me to the error described above.

After I forced setting the package version to

"@aws-sdk/client-s3": "3.705.0"

in both packages of the monorepo,

deleting the packages (node_modules folder)
deleting the package.lock.json

and reinstalling packages

That forces to install the version "@aws-sdk/client-s3": "3.705.0".

At version 3.705.0, I didn't see any errors

[arealesramirez]

We're discussing internally on how the behavior should be.

Ideally, the JS SDK should skip validating the checksum for get operations if the dependency is not available. It's because the application did not explicitly request checksum to be validate using CRC64-NVME algorithm.

The fix is likely to be released on Tuesday Jan 21st or Wednesday Jan 22nd.

For a quick fix, you can pin the SDK version to <3.729.0 in your package.json. I'll create a minimal repro and provide another workaround which required adding a middleware in the following comment.

That makes sense, as my solution was to use version 3.705.0

[arealesramirez]

After the fix is released, will it still require installing and importing "@aws-sdk/crc64-nvme-crt"?

Could it just come as a dependency of the @aws-sdk/client-s3 fix version?

[trivikr]

After the fix is released, will it still require installing and importing "@aws-sdk/crc64-nvme-crt"?

We're still discussing internally. As explained in #6822 (comment) I don't think we'll require dependency to be explicitly installed.

Ideally, the JS SDK should skip validating the checksum for get operations if the dependency is not available. It's because the application did not explicitly request checksum to be validate using CRC64-NVME algorithm.

[trivikr]

Could it just come as a dependency of the @aws-sdk/client-s3 fix version?

No. The AWS SDK for JavaScript customers are sensitive to install sizes, and the aws-crt dependency is written in C which has a large install size. We cannot install it by default and need to request customers to explicitly install it.

[trivikr]

Minimal repro

PutObject

This can be another JS SDK application which has optional dependency, or another AWS SDK which includes CRC64NVME checksum by default. Running this file will register CRC64NVME checksum with S3 backend.

// putObject.mjs
import { S3, ChecksumAlgorithm } from "@aws-sdk/client-s3"; // >=v3.729.0
import "@aws-sdk/crc64-nvme-crt";

const client = new S3();

const Bucket = "test-flexible-checksums-v2"; // Change to your bucket name.
const Key = "foo";
const Body = "bar";

await client.putObject({ Bucket, Key, Body, ChecksumAlgorithm: ChecksumAlgorithm.CRC64NVME });

GetObject

This is a JS SDK application which performs data integrity checks by default.
Since S3 backend returns a CRC64NVME checksum,

// putObject.mjs
import { S3 } from "@aws-sdk/client-s3"; // >=v3.729.0

const client = new S3();

const Bucket = "test-flexible-checksums-v2";
const Key = "foo";

// ChecksumMode not required to be set in >=v3.729.0
await client.getObject({ Bucket, Key, ChecksumMode: "ENABLED" });

This throws error in >=3.729.0

/Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:209
        throw new Error(
              ^

Error: Please check whether you have installed the "@aws-sdk/crc64-nvme-crt" package explicitly. 
You must also register the package by calling [require("@aws-sdk/crc64-nvme-crt");] or an ESM equivalent such as [import "@aws-sdk/crc64-nvme-crt";]. 
For more information please go to https://github.com/aws/aws-sdk-js-v3#functionality-requiring-aws-common-runtime-crt
    at selectChecksumAlgorithmFunction (/Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:209:15)
    at validateChecksumFromResponse (/Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:411:35)
    at /Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:466:11
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async /Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:174:20
    at async /Users/trivikr/workspace/test/node_modules/@smithy/middleware-serde/dist-cjs/index.js:33:24
    at async /Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:483:18
    at async /Users/trivikr/workspace/test/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38
    at async /Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:109:22
    at async /Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:136:14 {
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}

Node.js v22.13.0

Verified that error is not thrown in @aws-sdk/client-s3@<3.729.0

[trivikr]

Code workarounds if you can't pin your SDK dependency to <3.729.0

Recommended: Manually install @aws-sdk/crc64-nvme-crt

Manually install and import @aws-sdk/crc64-nvme-crt, if your application and production environment allows it.
This will provide SDK with optional dependency to compute and validate CRC64-NVME checksum returned in getObject response.

// putObject.mjs
import { S3 } from "@aws-sdk/client-s3"; // >=v3.729.0
import "@aws-sdk/crc64-nvme-crt"; // <-- here

const client = new S3();

const Bucket = "test-flexible-checksums-v2";
const Key = "foo";

await client.getObject({ Bucket, Key });

Remove x-amz-checksum-crc64nvme header using a middleware

This will just remove the crc64nvme checksum from the response.
If the GetObject API call responds with any other checksum (say CRC32), then it'll be validated.

// putObject.mjs
import { S3 } from "@aws-sdk/client-s3"; // >=v3.729.0

const client = new S3();

// ToDo: Remove workaround once https://github.com/aws/aws-sdk-js-v3/issues/6822 is fixed.
client.middlewareStack.addRelativeTo(
  (next) => async (args) => {
    const result = await next(args);
    delete result.response.headers["x-amz-checksum-crc64nvme"];
    return result;
  },
  {
    relation: "after",
    toMiddleware: "flexibleChecksumsResponseMiddleware",
  }
);

const Bucket = "test-flexible-checksums-v2";
const Key = "foo";

await client.getObject({ Bucket, Key });

Disable response checksum validation on response

This is not recommended, as it'll disable checksum validation on all responses

// putObject.mjs
import { S3 } from "@aws-sdk/client-s3"; // >=v3.729.0

const client = new S3({
  // ToDo: Remove workaround once https://github.com/aws/aws-sdk-js-v3/issues/6822 is fixed.
  responseChecksumValidation: "WHEN_REQUIRED",
});

const Bucket = "test-flexible-checksums-v2";
const Key = "foo";

await client.getObject({ Bucket, Key });