[Bug]: typescript cannot compile since v5.11.1

[DavideViolante]

MQTTjs Version

5.11.1 and above (also 5.13.1)

Broker

Aedes

Environment

NodeJS

Description

Simply upgrading from v5.11.0...v5.11.1 5.11.0 to 5.11.1 the build of my typescript project returns these errors:

> tsc

node_modules/mqtt/build/lib/client.d.ts:3:31 - error TS7016: Could not find a declaration file for module 'readable-stream'. '/home/dave/job/myproject/mqtt-broker/node_modules/mqtt/node_modules/readable-stream/lib/ours/index.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/readable-stream` if it exists or add a new declaration (.d.ts) file containing `declare module 'readable-stream';`

3 import { DuplexOptions } from 'readable-stream';
                                ~~~~~~~~~~~~~~~~~

node_modules/mqtt/build/lib/client.d.ts:5:31 - error TS7016: Could not find a declaration file for module 'ws'. '/home/dave/job/myproject/mqtt-broker/node_modules/ws/index.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/ws` if it exists or add a new declaration (.d.ts) file containing `declare module 'ws';`

5 import { ClientOptions } from 'ws';
                                ~~~~

node_modules/mqtt/build/lib/store.d.ts:1:26 - error TS7016: Could not find a declaration file for module 'readable-stream'. '/home/dave/job/myproject/mqtt-broker/node_modules/mqtt/node_modules/readable-stream/lib/ours/index.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/readable-stream` if it exists or add a new declaration (.d.ts) file containing `declare module 'readable-stream';`

1 import { Readable } from 'readable-stream';
                           ~~~~~~~~~~~~~~~~~

package.json

...
"dependencies": {
    "@influxdata/influxdb-client": "^1.35.0",
    "aedes": "^0.51.3",
    "async-mutex": "^0.5.0",
    "axios": "^1.9.0",
    "date-fns": "~3.6.0",
    "date-fns-tz": "~3.2.0",
    "firebase": "^11.10.0",
    "firebase-admin": "^13.4.0",
    "log4js": "^6.9.1",
    "mqtt": "^5.11.1",
    "node-cache": "^5.1.2",
    "redis": "^4.7.0"
  },
  "devDependencies": {
    "@commitlint/cli": "^19.8.1",
    "@commitlint/config-conventional": "^19.8.1",
    "@eslint/js": "^9.28.0",
    "@types/jest": "^29.5.14",
    "@types/node": "^22.15.29",
    "eslint": "^9.28.0",
    "husky": "^9.1.7",
    "jest": "^29.7.0",
    "ts-jest": "^29.3.4",
    "typescript": "^5.8.3",
    "typescript-eslint": "^8.35.1"
  }

Minimal Reproduction

cannot share job code, but no changes were made since last version. Let me know if you need more details specifically.
#2006 (comment)

Debug logs

error on compilation, no running. Found it via github action failing to build the project for the dependabot PR updating this lib.

[robertsLando]

Seems related to: #1991, try to check there

[DavideViolante]

I checked but cannot find a proper solution honestly. cc @SmashingQuasar

Note: with "skipLibCheck": true in tsconfig it works, but this is not a solution

Edit

The reproduction repo zip posted in #1991 by @hjdhjd actually returns the error (just change the package.json and set mqtt to 5.11.1 or higher).

[17:54:17] dave@DaveXPS:~/Downloads/homebridge-plugin-utils-1.15.3$ npm run build

> homebridge-plugin-utils@1.15.3 build
> npm run clean && tsc && shx cp dist/featureoptions.js{,.map} dist/ui


> homebridge-plugin-utils@1.15.3 clean
> shx rm -rf dist && npm run build-ui


> homebridge-plugin-utils@1.15.3 build-ui
> shx mkdir -p dist/ui && shx cp ui/**.mjs dist/ui

node_modules/mqtt/build/lib/client.d.ts:5:36 - error TS7016: Could not find a declaration file for module 'ws'. '/home/dave/Downloads/homebridge-plugin-utils-1.15.3/node_modules/ws/index.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/ws` if it exists or add a new declaration (.d.ts) file containing `declare module 'ws';`

5 import type { ClientOptions } from 'ws';
                                     ~~~~


Found 1 error in node_modules/mqtt/build/lib/client.d.ts:5

[robertsLando]

OK could you submit a PR to restore those modules to production deps please? I

[robertsLando]

@SmashingQuasar any clue why this still happens?

[robertsLando]

I have opened this PR in case this doesn't get solved: #2008 (comment)

[SmashingQuasar]

@SmashingQuasar any clue why this still happens?

Hey, I will have a look at this during the week-end. I am overwhelmed by work during the week these days.

[SmashingQuasar]

Alright I decided to have a look right away.

Why does it no longer work?

import { Readable } from 'readable-stream';

If you have a look at the @types/readable-stream library, you can clearly find the Readable class being declared:

In that case, it means the readable-stream module is supposed to export a Readable class entity in any valid Javascript syntax, which can include the antiquated function-based approach from ES5 and inferior.

This is where problems begin.

If you look at the compiled source code of the readable-stream library (it's not safe for work, let me warn you), you will notice that it uses a syntax that comes from ES5 and before:

node_modules/readable-stream/lib/internal/streams/readable.js

Readable.prototype.destroy = destroyImpl.destroy
Readable.prototype._undestroy = destroyImpl.undestroy
Readable.prototype._destroy = function (err, cb) {
  cb(err)
}

This class technically exists in their code.
Now what really matters is not this part, but it is the entrypoint of the package which for readable-stream is defined as:

{
  "main": "lib/ours/index.js",
}

You can find it at line 25 of the file node_modules/readable-stream/package.json.
This is the file that the V8 engine will load when you do import { x } from readable-stream`.

Here are the exports of this file:

As you can see, this file does not have any Readable export.

So this is exactly what TypeScript is telling you: You are trying to import a class that does not exist within the target module, so TypeScript will default to any which TypeScript is upset about.

Why did it use to work?

The reason is actually both simple and terrifying.
The library @types/readable-stream is not representative of readable-stream.
When @types/readable-stream exists as a hard dependency, then there is a hidden mechanism in TypeScript to fallback to @types/<module_name> when it cannot find the entity within <module_name>.
So TypeScript looks at import { Readable } from 'readable-stream'; and concludes: It can't find this type. 🤔
Then it decides: Let's see if it is resolved by a typing library and looks in @types/readable-stream (which is a hard dependency so it is installed with the library). It finds the Readable declaration and concludes: Alright, that was a typing library hack, I'm good now.

Remember that when you install a library, you only install the hard dependencies (listed in dependencies of the package.json file) of the library. Because you were declaring @types/readable-stream as a hard dependency, it was being installed with your package as well, as a hard dependency, which is the reason for my original PR.
With the 5.11 change, your package declares @types/readable-stream as a development dependency (listed in devDependencies of the package.json file). So @types/readable-stream is no longer installed alongside your package.

This change did not create the issue it only revealed the issue.

The reason is exactly the same of ws so I won't detail it here as it will be the same conclusion.

How can we move forward?

So that's really up to you @robertsLando. I can't open a PR as there is an important decision that needs to be made. I can, however, offer several solutions.

Solution 1 (Recommended)

Ideally, you want to remove the readable-stream and ws packages from your hard dependencies, as they are clearly not properly bundled and you have been using a buggy library for a long time which may or may not have caused invisible issues before.
I can help you with that by opening a PR with a revised code if you want me to, though this may take me some time.

Solution 2

A quick and awful fix is to simply put @types/readable-stream back in the dependencies section of the package.json file.
This will go back to the previous status quo, hiding the issue and all possible failures.
If you are very stressed about the stability of the library right now, I would say you could do that while we, you or I figure out the removal of those two buggy dependencies (ws and readable-stream).

Solution 3

Alright so this one is, in my opinion, the best for the entire Javascript community, but it's annoying.
Someone - and I don't know who - needs to fix ws and readable-stream. I can open a PR on both packages to fix them, but here are my concerns:

• Considering the immense technical debt of those libraries, it might be a rabbit hole.
• The maintainers may or may not be friendly. Not all maintainers are as nice as you are @robertsLando.
• The approval process will most likely take a long time and go through many discussions which can result in this problem being addressed in several months.

Again, completely opened to fix bugs where they exist, I'm just skeptical about this last solution.

How can we make sure this does not happen again?

So here I am not going to make a lot of friends, but the best ways to ensure this does not happen again are:

• Limit the number of dependencies you install.
• Make an in-depth of review of the dependencies you really need to install to make sure they are well maintained, packaged and typed.
• Avoid @types/ substitutes at all costs for this very reason. All libraries should export their typing nowadays. DefinitelyTyped (@types organisation) is just a "hack" to port old libraries, it's not the solution in the long run.
• Use stricter TypeScript rules to get blocked on this type of situations.
• Use stricted ESlint configuration to enforce even more rigorous typing which will likely make this situation impossible.

Sorry for the long post but I was under the impression that everyone was puzzled and confused about this issue. Since I understand it entirely I thought that you might be interested in the full explanation.
Also, sorry for the response delay. Don't hesitate to @ me when needed. I can even give you my e-mail or my Discord if you need to reach me quicker, always happy to help.

Let me know how you want to proceed, I'll help you the way I can.

[SmashingQuasar]

Just as a minor addition to the explanation regarding why it used to work:

In Javascript, classes exist only as a prototype, which itself is just an instance of Object with some specific behaviour.
Historically, classes where simply created as functions with the following syntax:

function MyClass()
{
}

MyClass.prototype.myMethod = function () {
}

Because of Javascript being prototype-based, there is no higher order class referential.
What this means, is that as long as you are not using instanceof (and even then there are caveats), codes like this one will work:

class Foo
{
	doSomething() {
		console.debug("I am the real one.");
	}
	doSomethingElse() {}
}

function processAFooObject(myFooInstance) // In TypeScript, we would write myFooInstance: Foo
{
	myFooInstance.doSomething();
}

const evilCopy = {
	doSomething: function() {
			console.debug("I am a defaced method.")
	}
};

processAFooObject(evilCopy); // This will work perfectly.
// > I am a defaced method.

Indeed in TypeScript, you would be blocked if you had the following signature:

function processAFooObject(myFooInstance: Foo)
{
	myFooInstance.doSomething();
}

But what would happen if you wrote this?

import { Foo } from 'improperly-bundled-library';

function processAFooObject(myFooInstance) // In TypeScript, we would write myFooInstance: Foo
{
	myFooInstance.doSomething();
}

Let's pretend that improperly-bundled-library has no Foo member, but there is a @types/improperly-bundled-library installed that does.

TypeScript would silently go look at @types/improperly-bundled-library and be satisfied.
When you compile the code, TypeScript would not complain due to this silent behaviour.
At runtime, we fallback to raw Javascript, so no type whatsoever, so it would never fail. However, your code would absolutely not be what you wanted it to be.

In conclusion:

• Inference is evil.
• any is evil.
• @types/ have a purpose but invite care and vigilance.
• TypeScript typeguards can validate types at runtime and can be extremely performant if approached in a functional way like done with @vitruvius-labs/ts-predicate.

I could go more in-depth but I think this is enough to understand what's going on in details for those interested.

[SmashingQuasar]

I opened a PR to address this issue.
I already fixed the readable-stream dependency which was trivial since this dependency has the same signature as node:stream. It might have been integrated within Node.js itself.
I see that you declare supporting Node.js 16 in your package.json file but only test for Node.js 18 and 20.
Node.js 16 is an awkward version as this is an LTS with a very short support duration (not so "L"TS).

For ws, I can either use the native Node implementation (which is in fact the undici library integrated in the stdlib) or use undici separately. I think I'll go with the latter, as the ws implementation of Node.js only works for Node 22 and above.

[DavideViolante]

Thank you so much for the deep and long explanation, very appreciated! 🙏
I would proceed with solution 2 (#2008) and when it's ready the solution 1 (ideal) will replace the solution 2.

[robertsLando]

@SmashingQuasar thanks a lot for taking time for this in depth explanation! 🙏🏼

Unfortunately readable-stream cannot be dropped AFAIK because this would break mqttjs in browsers, maybe we could drop ws module but I should investigate. In that case the issue with browser doens't exists as we are using Websocket for browser and ws for nodejs.

I know the maintainer of readable-stream and maybe he could fix the issue there (or better accept incoming PR). cc @mcollina

So IMO the best thing to do is:

1. Merge fix: move back @types/ws from dev to deps #2008 so at least we fix the actual version
2. Open a PR to fix readable-stream. I will ping @mcollina to ask for some priority on that
3. Open a PR here to drop ws module in favor of native nodejs implementation (need to check compatibility of this one)
4. Add some more strict checks on eslint and tsconfig to avoid this happening again.

@SmashingQuasar WDYT? Can you help with this?

About nodejs version, I have planned to drop support for nodejs < 20 in next v6 release. Reason why I waited last time is that I forgot to drop it's support in version 5 and then I would like to avoid 2 majors very close (many people have been on version 4 for so long and 2 major can scary a bit and slow the adoption)