** X-Forwarded-For being overwritten or appended by request sent with http headers parameters **

[mohijeet-changejar]

Hi everyone recently we migrated from nginx ingress controller to nginx-gateway-fabric for some of services and one of weird issue or miss configuration we are facing i would really appreciate any help if there is any solution to avoid this or any suggestions.

ISSUE:

We are handling ip whitelisting at application level for our third party's different ips, but it started failing because we were not able to get x-forwared-for in http headers we are receiving these many headers

curl --location "*****"

Headers:
host: ******
x-real-ip: ******
x-forwarded-proto: http
x-forwarded-host: ******
x-forwarded-port: 80
connection: close
user-agent: curl/8.4.0
accept: /

now to check exact headers of http packet being passed by http-nginx-fabric-gateway we went through documentations and find out service kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v1.6.2/examples/http-request-header-filter/headers.yaml

we deployed it and check what are headers being printed here they are

Headers:
header 'Host' is ''
header 'X-Forwarded-For' is ''
header 'X-Real-IP' is ''
header 'X-Forwarded-Proto' is 'http'
header 'X-Forwarded-Host' is ''
header 'X-Forwarded-Port' is '80'
header 'Connection' is 'close'
header 'sec-ch-ua' is '"Chromium";v="136", "Google Chrome";v="136", "Not.A/Brand";v="99"'
header 'sec-ch-ua-mobile' is '?0'
header 'sec-ch-ua-platform' is '"macOS"'
header 'Upgrade-Insecure-Requests' is '1'
header 'Accept' is 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7'
header 'Sec-Fetch-Site' is 'none'
header 'Sec-Fetch-Mode' is 'navigate'
header 'Sec-Fetch-User' is '?1'
header 'Sec-Fetch-Dest' is 'document'
header 'Accept-Encoding' is 'gzip, deflate, br, zstd'
header 'Accept-Language' is 'en-GB,en-US;q=0.9,en;q=0.8'

so these is first problem we face and not able to understand how is it not able to get this headers in all services

Major Concern:

Now in order to debug we tried passing custom header to curl request and we find out nginx-fabric-gateway is appending these custom ips, when we were using nginx-ingress-controller it was not possible to change this x-forwared-for ip as any whitelisted ips can be added now it becomes security issue while using http-nginx-fabric-gateway we tried using nginxproxy in order to resolve this but its also appending ip and in cidr range we can not specify as there are multiple ips which i dont want to whitelist at nginx level. In sort i am looking for a way which either help in stopping this appending ip issue or something that tells nginx to not touch any header pass it as it is....

EX:

curl --location '' \
--header 'x-forwarded-for: spoofing'
--header 'x-real-ip: spoofing'
Headers:
header 'Host' is ''
header 'X-Forwarded-For' is 'spoofing , '
header 'X-Real-IP' is ''
header 'X-Forwarded-Proto' is 'http'
header 'X-Forwarded-Host' is '**'
header 'X-Forwarded-Port' is '80'
header 'Connection' is 'close'
header 'User-Agent' is 'curl/8.4.0'
header 'Accept' is '/'

here is nginx file i got by going inside nignx-ingress vs nginx-fabric-gateway

nignx-ingress:

                    proxy_set_header X-Forwarded-For        $remote_addr;
                    
                    # mitigate HTTPoxy Vulnerability
                    # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
                    proxy_set_header Proxy                  "";

nginx-fabric-gateway

    proxy_http_version 1.1;
    proxy_set_header Host "$gw_api_compliant_host";
    proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
    proxy_set_header X-Real-IP "$remote_addr";
    proxy_set_header X-Forwarded-Proto "$scheme";
    proxy_set_header X-Forwarded-Host "$host";
    proxy_set_header X-Forwarded-Port "$server_port";
    proxy_set_header Upgrade "$http_upgrade";
    proxy_set_header Connection "$connection_upgrade";
    proxy_pass http://default_headers_80$request_uri;

[Shreyank031]

We are experiencing the exact same issue.
We did experiment with SnippetsFilter and NginxProxy, but didn't resolve the issue.
It would be helpful is anyone can guide us to configure it with right way.

Thanks.

[ciarams87]

Hi there,

Thanks for the detailed report and for your interest in the project!

It looks to me like you are migrating from the community Ingress NGINX (kubernetes/ingress-nginx) project to NGINX Gateway Fabric. What you are seeing here is a behavioral difference between the community Ingress NGINX project, and both the NGINX Ingress Controller (nginx/kubernetes-ingress) and the NGINX Gateway Fabric projects.

Ingress NGINX's default behavior is proxy_set_header X-Forwarded-For $remote_addr;, which overwrites any client-provided forwarded headers with just the real client IP.

NGF and NIC use proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;, which appends to existing headers following RFC 7239 standards. This preserves the complete proxy chain, which is the expected behavior for most reverse proxies and load balancers.

This difference is worth keeping in mind when migrating between projects. For your immediate security concern, consider using X-Real-IP for IP whitelisting since it provides consistent behavior regardless of which ingress solution you're using:

1. Use X-Real-IP for IP whitelisting - This contains $remote_addr and is not directly controllable by client-provided headers
2. Update your application logic to rely on X-Real-IP instead of parsing X-Forwarded-For

(As a side note, many security teams are moving away from header-based IP whitelisting toward more robust authentication methods, but that's beyond the scope of this discussion.)

Update: What might actually help you for this use case is the RewriteClientIP setting in NGINXProxy.

[mohijeet-changejar]

Hi, Thanks for replying and making it clear why this is expected behaviour.
Still just asking for any future consideration in case if there are things need to be rewrite at http.server level or http.server.location level what is the proper way for now we tried using SnippetFilter at http.server level but its including that in *.conf file is there any way to overwrite any low level configurations defined in nginx files ?

listen [::]:80;
    server_name ********;
    include /etc/nginx/includes/SnippetsFilter_http.server_default_x-forwarded-for-fix.conf;
    
    location /headers/ {

        proxy_http_version 1.1;
        proxy_set_header Host "$gw_api_compliant_host";
        proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
        proxy_set_header X-Real-IP "$remote_addr";
        proxy_set_header X-Forwarded-Proto "$scheme";
        proxy_set_header X-Forwarded-Host "$host";
        proxy_set_header X-Forwarded-Port "$server_port";
        proxy_set_header Upgrade "$http_upgrade";
        proxy_set_header Connection "$connection_upgrade";
        proxy_pass http://default_headers_80$request_uri;
    }
    location = /headers {
 
       proxy_http_version 1.1;
        proxy_set_header Host "$gw_api_compliant_host";
        proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
        proxy_set_header X-Real-IP "$remote_addr";
        proxy_set_header X-Forwarded-Proto "$scheme";
        proxy_set_header X-Forwarded-Host "$host";
        proxy_set_header X-Forwarded-Port "$server_port";
        proxy_set_header Upgrade "$http_upgrade";
        proxy_set_header Connection "$connection_upgrade";
        proxy_pass http://default_headers_80$request_uri;
                      
     
    }

[ciarams87]

After understanding your use case a little better, I've updated my answer above to include a link to RewriteClientIP settings which may help you here.

Regarding your question about overriding NGF's location level directives using SnippetFilter - you're correct that some server-level directives (like proxy_set_header) get overridden by NGF's location-level directives due to NGINX's directive precedence rules. However, hopefully in this case the rewriteClientIP approach will work for you.

[mohijeet-changejar]

I already gone through this solution here is what i did and i am getting same result i am not sure what i am doing is right or wrong here is detailed information of implementation :

I created nginxProxy kind with this configuration which i thinks suits our use case as per documentations, Here i just used some dummy ip to check weather is it even working or not

➜  ~ kubectl apply -f - <<EOF
apiVersion: gateway.nginx.org/v1alpha1
kind: NginxProxy
metadata:
  name: ngf-proxy-config
spec:
  rewriteClientIP:
    mode: XForwardedFor
    trustedAddresses:
    - type: CIDR
      value: "192.33.21.1/24"
EOF

nginxproxy.gateway.nginx.org/ngf-proxy-config configured

Now i again tried to add custom headers here

curl --location '******' --header 'x-forwarded-for: dummy ip, hi there, 0.0.0.0'

Headers:
header 'Host' is ''
header 'X-Forwarded-For' is 'dummy ip, hi there, 0.0.0.0, 103.214.63.172'
header 'X-Real-IP' is '103.214.63.172'
header 'X-Forwarded-Proto' is 'http'
header 'X-Forwarded-Host' is ''
header 'X-Forwarded-Port' is '80'
header 'Connection' is 'close'
header 'User-Agent' is 'curl/8.4.0'
header 'Accept' is '/'

here as you can see 103.214.63.172 is the correct ip of request being sent and which is not part of trusted ips so technically it should not allow it to modify as per below documentation where it suggests

TrustedAddresses specifies the addresses that are trusted to send correct client IP information. If a request comes from a trusted address, NGINX will rewrite the client IP information, and forward it to the backend in the X-Forwarded-For* and X-Real-IP headers. If the request does not come from a trusted address, NGINX will not rewrite the client IP information. TrustedAddresses only supports CIDR blocks: 192.33.21.1⁄24, fe80::1⁄64. To trust all addresses (not recommended for production), set to 0.0.0.0/0. If no addresses are provided, NGINX will not rewrite the client IP information. Sets NGINX directive set_real_ip_from: https://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from This field is required if mode is set.

https://docs.nginx.com/nginx-gateway-fabric/reference/api/#gateway.nginx.org/v1alpha1.RewriteClientIP

I am also describing both of component gateway class and nginxproxy after configuration for your reference

➜  ~ kubectl describe nginxproxy.gateway.nginx.org/ngf-proxy-config
Name:         ngf-proxy-config
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  gateway.nginx.org/v1alpha1
Kind:         NginxProxy
Metadata:
Creation Timestamp:  2025-05-28T15:34:21Z
Generation:          5
Resource Version:    376693934
Spec:
Ip Family:  dual
Rewrite Client IP:
  Mode:  XForwardedFor
  Trusted Addresses:
    Type:   CIDR
    Value:  192.0.0.0/16
Events:       <none>


➜  ~ kubectl describe gatewayclass nginx-public
Name:         nginx-public
Namespace:    
Labels:       
            app.kubernetes.io/instance=ngf-public
            app.kubernetes.io/managed-by=Helm
            app.kubernetes.io/name=nginx-gateway-fabric
            app.kubernetes.io/version=1.6.2
            helm.sh/chart=nginx-gateway-fabric-1.6.2
            
Annotations:  
            meta.helm.sh/release-name: ngf-public
            meta.helm.sh/release-namespace: ngf-gateway-public
            
API Version:  gateway.networking.k8s.io/v1
Kind:         GatewayClass
Metadata:
Creation Timestamp:  2025-04-25T07:48:28Z
Generation:          9
Resource Version:    376693935
Spec:
Controller Name:  gateway.nginx.org/nginx-gateway-controller-public
Parameters Ref:
  Group:  gateway.nginx.org
  Kind:   NginxProxy
  Name:   ngf-proxy-config
Status:
Conditions:
  Last Transition Time:  2025-05-28T15:47:25Z
  Message:               GatewayClass is accepted
  Observed Generation:   9
  Reason:                Accepted
  Status:                True
  Type:                  Accepted
  Last Transition Time:  2025-05-28T15:47:25Z
  Message:               Gateway API CRD versions are supported
  Observed Generation:   9
  Reason:                SupportedVersion
  Status:                True
  Type:                  SupportedVersion
  Last Transition Time:  2025-05-28T15:47:25Z
  Message:               parametersRef resource is resolved
  Observed Generation:   9
  Reason:                ResolvedRefs
  Status:                True
  Type:                  ResolvedRefs
Events:                    <none>

[salonichf5]

Hey,

Thank you for the interest in our project. I re-read the issue again and couple of pointers.

1. I think you are concerned about these headers being added to your application ? These are browser generated headers being
   added to your application and showing up from there.

header 'sec-ch-ua' is '"Chromium";v="136", "Google Chrome";v="136", "Not.A/Brand";v="99"'
header 'sec-ch-ua-mobile' is '?0'
header 'sec-ch-ua-platform' is '"macOS"'
header 'Upgrade-Insecure-Requests' is '1'
header 'Accept' is 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7'
header 'Sec-Fetch-Site' is 'none'
header 'Sec-Fetch-Mode' is 'navigate'
header 'Sec-Fetch-User' is '?1'
header 'Sec-Fetch-Dest' is 'document'
header 'Accept-Encoding' is 'gzip, deflate, br, zstd'
header 'Accept-Language' is 'en-GB,en-US;q=0.9,en;q=0.8'

2. I think X-forwarded-for header is used to rewrite the client IP based on the trusted address we set. Since, the IP 103.214.63.172 is not in the trustedAddress range and setIPRecursively field is by default false. It chooses the last element in the array for X-forwarded-for header and indicates that value as the X-Real-IP

If the request is sent from that IP, that is what will reflect as real IP in the header unless 103.214.63.172 is part of trustedAddress, then it picks the second last hi there from the array.

X-Forwaded-For is a request header field whose value is used to replace the client address.

3. Understanding what setIPRecursively does could be helpful.
   For example

SetIPRecursively configures whether recursive search is used when selecting the client's address from
the X-Forwarded-For header. It is used in conjunction with TrustedAddresses.
If enabled, NGINX will recurse on the values in X-Forwarded-Header from the end of array
to start of array and select the first untrusted IP.
For example, if X-Forwarded-For is [11.11.11.11, 22.22.22.22, 55.55.55.1],
and TrustedAddresses is set to 55.55.55.1/32, NGINX will rewrite the client IP to 22.22.22.22.
If disabled, NGINX will select the IP at the end of the array.
In the previous example, 55.55.55.1 would be selected.

Follow-up question:

From what I understand, your requirement is to not append any of these headers being passed in X-Forwarded-For for security concerns while IP whitelisting your trusted IP sources?

I am a little confused if you are trying to bypass the IPs of your load balancer/proxies along with preserving headers to get the real client IP for monitoring or concerned about security issues with headers being appended to X-Forwarded-For