Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Socket.IO Adapter Close causes premature HTTP Server close() call and blocks application shutdown #13910

Open
laino opened this issue Aug 25, 2024 · 3 comments
Labels
needs triage This issue has not been looked into

Comments

@laino
Copy link

laino commented Aug 25, 2024

On Nest application shutdown, close() is called on the socket.io server here.

Internally socket.io will now disconnect all sockets handled by itself, then calls close() on the underlying http server here.

This is a problem because now we are blocking until all connections on the underlying http server are closed, but before the nest application has any chance to run logic that would disconnect other long-running connections. That means that if there's a long running connection that wasn't handled by that socket.io adapter, we now block forever.

For instance this disconnect logic is never reached.

@laino laino added the needs triage This issue has not been looked into label Aug 25, 2024
@laino
Copy link
Author

laino commented Aug 25, 2024

As a workaround you can hack in your own logic that disconnects clients and prevents new connections before initiating application shutdown.

@laino
Copy link
Author

laino commented Aug 26, 2024

Here's a workaround:

// FIXME: workaround for https://github.com/nestjs/nest/issues/13910
if (forceCloseConnections) {
    const server = (app as NestExpressApplication).getHttpServer();

    server.close = (close => (cb) => {
        close(cb);
        server.closeAllConnections();
        return server;
    })(server.close.bind(server));
}

@laino
Copy link
Author

laino commented Aug 26, 2024

I think a fix for this is to never call SocketIOServer.close(), and instead track connections ourselves in the websocket adapter, destroy them on close, and also destroy any new incoming connections that may happen after the adapter is stopped, but before nest closes the actual http server.

Another way would be to close the http server early in the shutdown process without waiting for the callback, progress through other shutdown logic, then finally wait for the callback:

  1. Call httpServer.close(closeCallback), this will synchronously make it not accept new connections
  2. Perform nest application shutdown
  3. if forceCloseConnections is set, call destroy() on all remaining sockets
  4. wait for closeCallback

With the latter way it is up to each component what it wants to do with in-flight requests during shutdown, but they won't have to worry about new requests coming in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs triage This issue has not been looked into
Projects
None yet
Development

No branches or pull requests

2 participants
@laino and others