Issue obtaining body and updating it as part of proxy

[coupster74]

Checks

• I understand project setup issues should be asked on StackOverflow or in GitHub Discussions.
• I updated to latest http-proxy-middleware.

Describe the bug (be clear and concise)

I am proxying azure search queries, and would like to obtain the complete request body, modify it, and forward it on. I've tried a number of things, and the results are consistently where either the body is empty, or by the time I get the entire body, the request has already been sent through to the server and cannot be changed.

I've reviewed the recipe here: https://github.com/chimurai/http-proxy-middleware/blob/master/recipes/modify-post.md, (which just replaces the body, a unrealistic usecase), but I don't quite understand why we assume we would have the body at all since it is a stream. I've had to add this within the onProxyReq: to obtain the body:

         req.on('data', (chunk) => {
            bodyData += chunk;
          });

here is the code I currently have which obtains the body successfully, but it is too late to modify it.

 onProxyReq: async (proxyReq, req, res) => {
        if (req.method === 'POST' || req.method === 'PUT') {

          // log the search request
          let bodyData = '';
          Logger.log(req.body);  // is always  undefined

          req.on('data', (chunk) => {
            bodyData += chunk;
          });
          req.on('end', async () => {
            const email = req.user ? req.user : 'unknown';
            let searchArgs = {};
            try {
              // bodyData exists
              searchArgs = JSON.parse(bodyData);
              // log this..
              if (searchArgs['search']) {
                const activityLogService = app.get(ActivityLogService);
                await activityLogService.logSearchActivity(searchArgs, email);
              }
            } catch (e) {
              // parse error
              Logger.log('error', e);
            }
          });
        }
      },

what am I missing here?

Step-by-step reproduction instructions

1. ...
2. ...

Expected behavior (be clear and concise)

Within the onProxyReq there would be an approach to obtain, modify and forward on the req.body successfully

How is http-proxy-middleware used in your project?

to proxy requests through to azure search. Specifically
- ensure they are authenticated (via jwt token) with my middleware, 
- add the azure search APIKEY (removing the need to include it from the front end, protecting the key)
- log the search request details
- ideally (the issue here), modify the search based on the user's profile (from the jwt token)

What http-proxy-middleware configuration are you using?

don't understand the question

What OS/version and node/version are you seeing the problem?

System:
    OS: macOS 13.5.2
    CPU: (10) arm64 Apple M1 Max
    Memory: 1.54 GB / 64.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 18.14.0 - ~/.nvm/versions/node/v18.14.0/bin/node
    Yarn: 1.22.19 - ~/.yarn/bin/yarn
    npm: 8.19.4 
  Managers:
    CocoaPods: 1.11.2 - /usr/local/bin/pod
    Homebrew: 4.1.4 - /opt/homebrew/bin/brew
    pip3: 23.0.1 - /opt/homebrew/bin/pip3
    RubyGems: 3.0.3.1 - /usr/bin/gem
  Utilities:
    CMake: 3.26.4 - /opt/homebrew/bin/cmake
    Make: 3.81 - /usr/bin/make
    GCC: 14.0.3 - /usr/bin/gcc
    Git: 2.41.0 - /opt/homebrew/bin/git
    Clang: 14.0.3 - /usr/bin/clang
    Subversion: 1.14.2 - /opt/homebrew/bin/svn
    Curl: 8.1.2 - /usr/bin/curl
  Servers:
    Apache: 2.4.56 - /usr/sbin/apachectl
  Virtualization:
    Docker: 24.0.2 - /opt/homebrew/bin/docker
  SDKs:
    iOS SDK:
      Platforms: DriverKit 22.4, iOS 16.4, macOS 13.3, tvOS 16.4, watchOS 9.4
  IDEs:
    Android Studio: 2021.1 AI-211.7628.21.2111.8193401
    Atom: 1.60.0
    VSCode: 1.82.0 - /opt/homebrew/bin/code
    Vim: 9.0 - /usr/bin/vim
    Xcode: 14.3.1/14E300c - /usr/bin/xcodebuild
  Languages:
    Bash: 3.2.57 - /bin/bash
    Perl: 5.30.3 - /usr/bin/perl
    Protoc: 23.2 - /opt/homebrew/bin/protoc
    Python: 3.8.11 - /opt/homebrew/Caskroom/miniconda/base/bin/python
    Python3: 3.11.4 - /opt/homebrew/bin/python3
    Ruby: 2.6.10 - /usr/bin/ruby
  Databases:
    PostgreSQL: 14.8 - /opt/homebrew/bin/postgres
    SQLite: 3.39.5 - /usr/bin/sqlite3
  Browsers:
    Chrome: 116.0.5845.187
    Safari: 16.6

Additional context (optional)

No response

[hollowtree]

maybe you can use the body-parser before use http-proxy-middleware

http-proxy-middleware/test/e2e/http-proxy-middleware.spec.ts

Line 91 in bd35b7b

bodyParser.urlencoded({ extended: false }),

[coupster74]

All.. regarding this issue, I found most of the guidance difficult to follow.. there was no single example which just worked. I did manage to figure it out though, and recommend it be added as an example in the documentation. For the code example below, I was proxying azure search and needed to do two things: 1) add the apikey (as an authenticated REST call, this ensured only authenticated users could call search, and keep the APIKEY secret from the client); 2) change the parameters of the search (in the body) based on the user's contest.

import * as httpProxy from 'http-proxy-middleware';
import * as bodyParser from 'body-parser';

/* snip */

app.use('/search/proxy', async (req: Request, res, next) => {
   bodyParser.json()(req, res, (err) => {
      if (err) return next(err);
      // get required context here.. (in my case, user context)

      if (req.body) {
          // make body changes here
      }
    
     httpProxy.createProxyMiddleware({
          target: process.env.AZURE_SEARCH,
          changeOrigin: true,
          pathRewrite: {
            '^/search/proxy': '', 
          },
          onError: (err, req, res) => {
            res.writeHead(500, {
              'Content-Type': 'text/plain',
            });
            res.end('Something went wrong...' + err);
          },
          onProxyReq: async (proxyReq, req: Request) => {
            // Update the content length for the proxy request cause i may have changed the body
            proxyReq.setHeader(
              'Content-Length',
              Buffer.byteLength(JSON.stringify(req.body)),
            );
            // add the search API key
            proxyReq.setHeader('api-key', process.env.AZURE_SEARCH_APIKEY);
            // Write out the body data
            //console.log(JSON.stringify(req.body));
            proxyReq.write(JSON.stringify(req.body));
          },
        })(req, res, next);
      } catch (err) {
        console.log(err);
        return next(new InternalServerErrorException());
      }
    });
  });

[theisof]

I had a related issue forwarding post, put and delete requests to a NextJS API. Your answer pointed me in direction of the solution which was to add proxyReq.write(JSON.stringify(req.body)); in the onProxyReq callback. Took me several hours to solve so I will leave my solution here for future generations:

app.use(
  "*",
  createProxyMiddleware({
    target: targetUrl,
    changeOrigin: true,
    onProxyReq: (proxyReq, req) => {
      if (req.method !== "GET" && Object.keys(req.body).length > 0) {
        proxyReq.write(JSON.stringify(req.body));
      }
    },
  })
);

[dschweinbenz]

I think this can be closed as the docs now include

Intercept and manipulate requests
Intercept requests from downstream by defining onProxyReq in createProxyMiddleware.
Currently the only pre-provided request interceptor is fixRequestBody, which is used to fix proxied POST requests when bodyParser is applied before this middleware.

Commit: 09d05d5