Bug report

• I confirm this is a bug with Supabase, not with my own application.
• I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

The before_user_created auth hook returns an "Invalid payload sent to hook" error when attempting to reject user account creation with a properly formatted HTTP 400 response. Instead of processing the rejection with the custom error message, Supabase returns an unexpected failure error.

To Reproduce

Steps to reproduce the behavior:

1. Set up a before_user_created auth hook in your Supabase project dashboard
2. Configure it to point to a custom API endpoint (e.g., using ngrok for local development)
3. Create an endpoint that conditionally rejects signups with this response:

   return response
     .status(400)
     .set('Content-Type', 'application/json')
     .json({
       error: {
         message: 'Signups from this email domain are not allowed.',
         http_code: 400,
       },
     });

4. Attempt to create a user account that should trigger the rejection logic
5. Observe the error response from Supabase

Expected behavior

When the auth hook returns HTTP 400 with the proper error payload format, Supabase should:

• Prevent the account creation
• Return the custom error message specified in the hook response
• Handle the rejection gracefully without throwing an "Invalid payload" error

Screenshots

Error response received:

{
  "code": "unexpected_failure",
  "message": "Invalid payload sent to hook"
}

System information

• OS: [Various - issue occurs on server side]
• Browser: N/A (affects API calls)
• Version of supabase-js: NA
• Version of Node.js: NA

Additional context

• The API endpoint is successfully receiving requests from Supabase (confirmed via server logs)
• The API responds with the exact JSON format as documented
• When returning HTTP 200 with an empty body, account creation proceeds without issues
• This suggests the hook connectivity is working, but error handling for HTTP 400 responses is broken
• The same issue has been reported by other users (@Entropei)
• Hook URL is accessible and properly configured in the Supabase dashboard

The problem specifically occurs when trying to reject user creation - the success path works fine, but the rejection path fails with an unhelpful error message that doesn't match the documentation.