send() failed (32: Broken pipe) when performing HTTP POST to Immich

[Matthias-vdE]

When running Immich (https://github.com/immich-app/immich) behind NPM(plus) and enabling Crowdsec/Appsec, it is not possible to upload files to the server via HTTP POST:

2024-09-27T11:33:13.859502771Z 2024/09/27 13:33:13 [error] 32855#32855: *10529 send() failed (32: Broken pipe), client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

2024-09-27T11:33:13.859545312Z 2024/09/27 13:33:13 [error] 32855#32855: *10529 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: broken pipe, client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

2024-09-27T11:33:13.859553917Z 2024/09/27 13:33:13 [error] 32855#32855: *10529 [lua] crowdsec.lua:651: Allow(): AppSec check: broken pipe, client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

2024-09-27T11:33:13.859564759Z 2024/09/27 13:33:13 [alert] 32855#32855: *10529 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied 'MY_IPADDRESS' with 'ban' (by appsec), client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

The issue was initially reported at ZoeyVid/NPMplus#1123.

[LaurenceJJones]

When running Immich (https://github.com/immich-app/immich) behind NPM(plus) and enabling Crowdsec/Appsec, it is not possible to upload files to the server via HTTP POST:

2024-09-27T11:33:13.859502771Z 2024/09/27 13:33:13 [error] 32855#32855: *10529 send() failed (32: Broken pipe), client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

2024-09-27T11:33:13.859545312Z 2024/09/27 13:33:13 [error] 32855#32855: *10529 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: broken pipe, client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

2024-09-27T11:33:13.859553917Z 2024/09/27 13:33:13 [error] 32855#32855: *10529 [lua] crowdsec.lua:651: Allow(): AppSec check: broken pipe, client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

2024-09-27T11:33:13.859564759Z 2024/09/27 13:33:13 [alert] 32855#32855: *10529 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied 'MY_IPADDRESS' with 'ban' (by appsec), client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

The issue was initially reported at ZoeyVid/NPMplus#1123.

Hey 👋🏻

I have setup a similar setup environment and don't see to be experiencing the same issue. I am uploading a 3mb file as per your reddit thread I dont see any errors.

Is CrowdSec / AppSec running locally to NPMPlus (on same host to reduce latency)?

Could you provide the full nginx configuration that is generated by NPMplus as I am using nginx (since its the same code I dont want to spend time configuring NPMPlus since its the same lua code)

server {
    server_name _;

    listen *:80;

    # allow large file uploads
    client_max_body_size 50000M;

    # Set headers
    proxy_set_header Host              $http_host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    # enable websockets: http://nginx.org/en/docs/http/websocket.html
    proxy_http_version 1.1;
    proxy_set_header   Upgrade    $http_upgrade;
    proxy_set_header   Connection "upgrade";
    proxy_redirect     off;

    # set timeout
    proxy_read_timeout 600s;
    proxy_send_timeout 600s;
    send_timeout       600s;

    location / {
        proxy_pass http://127.0.0.1:2283;
    }
}

everything seems to be getting processed:

root@bookworm:/etc/nginx/sites-enabled# cscli metrics show appsec
Appsec Metrics:
╭─────────────────┬───────────┬─────────╮
│ Appsec Engine   │ Processed │ Blocked │
├─────────────────┼───────────┼─────────┤
│ 127.0.0.1:7422/ │ 296       │ -       │
╰─────────────────┴───────────┴─────────

[Matthias-vdE]

Is CrowdSec / AppSec running locally to NPMPlus (on same host to reduce latency)?

Yup, both Crowdsec and NPMPlus are running in docker containers on the same host. Part of the same docker-compose:

services:
  npmplus:
    container_name: npmplus
    image: zoeyvid/npmplus:latest
    restart: always
    network_mode: host
    volumes:
      - "/opt/npm:/data"
    environment:
      - "TZ=Europe/Brussels" 
      - "NGINX_LOG_NOT_FOUND=true"
      - "LOGROTATE=true" 
      - "LOGROTATIONS=7" 
      - "GOA=true"

  crowdsec:
    container_name: crowdsec
    image: crowdsecurity/crowdsec:latest
    restart: always
    network_mode: bridge
    ports:
      - "127.0.0.1:7422:7422"
      - "127.0.0.1:8080:8080"
    environment:
      - "TZ=Europe/Brussels"
      - "COLLECTIONS=ZoeyVid/npmplus"
      - "LEVEL_FATAL=true"
      - "LEVEL_ERROR=true"
      - "LEVEL_WARN=true"
      - "LEVEL_INFO=false"
      - "LEVEL_DEBUG=false"
      - "LEVEL_TRACE=false"
    volumes:
      - "/opt/crowdsec/conf:/etc/crowdsec"
      - "/opt/crowdsec/data:/var/lib/crowdsec/data"
      - "/opt/npm/nginx:/opt/npm/nginx:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  geoipupdate:
    container_name: geoipupdate
    image: maxmindinc/geoipupdate:latest
    restart: always
    network_mode: bridge
    environment:
      - "TZ=Europe/Brussels"
      - "GEOIPUPDATE_EDITION_IDS=GeoLite2-Country GeoLite2-City GeoLite2-ASN"
      - "GEOIPUPDATE_ACCOUNT_ID=my_account_id"
      - "GEOIPUPDATE_LICENSE_KEY=my_license_key"
      - "GEOIPUPDATE_FREQUENCY=24"
    volumes:
      - "/opt/npm/etc/goaccess/geoip:/usr/share/GeoIP"

Immich is running on the same host as well, as a docker container. The config for that one is very standard with little to no customization.

My proxy config in NPMPlus looks like this:

Nothing in custom locations, and a regular certbot certificate to force enable HTTPS.

My crowdsec.conf file is this:

ENABLED=true
API_URL=http://127.0.0.1:8080
API_KEY=my_API_key
CACHE_EXPIRATION=1
# bounce for all type of remediation that the bouncer can receive from the local API
BOUNCING_ON_TYPE=ban
FALLBACK_REMEDIATION=ban
REQUEST_TIMEOUT=3000
UPDATE_FREQUENCY=10
# live or stream
MODE=live
# exclude the bouncing on those location
EXCLUDE_LOCATION=
#those apply for "ban" action
# /!\ REDIRECT_LOCATION and RET_CODE can't be used together. REDIRECT_LOCATION take priority over RET_CODE
BAN_TEMPLATE_PATH=/data/etc/crowdsec/ban.html
REDIRECT_LOCATION=
RET_CODE=
#those apply for "captcha" action
#valid providers are recaptcha, hcaptcha, turnstile
CAPTCHA_PROVIDER=
# Captcha Secret Key
SECRET_KEY=
# Captcha Site key
SITE_KEY=
CAPTCHA_TEMPLATE_PATH=/data/etc/crowdsec/captcha.html
CAPTCHA_EXPIRATION=3600
#APPSEC_URL=http://127.0.0.1:7422
#APPSEC_FAILURE_ACTION=deny

The two bottom lines are currently commented out to make it work. If I uncomment them, it breaks with the broken pipe error message when uploading anything.

[Zoey2936]

I don't use Immich myself, but there seems to be an Issue with it, I already have 3 discussions about it with multiple people having the same issues:
ZoeyVid/NPMplus#1168
ZoeyVid/NPMplus#1123
ZoeyVid/NPMplus#1241

the issue is always related to appsec and immich and disabling appsec (or changing APPSEC_FAILURE_ACTION to passthrough) fixed it (modsec also needs to be disabled) — sometimes increasing the timeouts also fixed (at least for Nextcloud which had similar issues), so I increased the default timeouts, but the issue still seems to exist on new installations with the new timeouts.

[MaximumFish]

To hopefully contribute, I started discussion 1241 that @Zoey2936 linked above. It was suggested I post here answering the below questions so here we go:

Is the deployment local or remote (VPS)? Local

Is the domain being proxied by a CDN like cloudflare? For full transparency I do use a Cloudflare tunnel, but only for external connections. The tunnel isn't involved while on the LAN.

Does it happen on upload like the OP or when you said when immich makes a backup? I am the OP in that thread. I believe the single image upload and backup feature are basically the same thing using the same endpoint. The backup feature just does it automatically and in batches.

This is what happens when I attempt to upload (or backup) a single photo:

2024/11/19 17:20:06 [warn] 80110#80110: *51203 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000050 while reading request body, client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/19 17:20:06 [error] 80110#80110: *51203 send() failed (32: Broken pipe), client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/19 17:20:06 [error] 80110#80110: *51203 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: broken pipe, client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/19 17:20:06 [error] 80110#80110: *51203 [lua] crowdsec.lua:651: Allow(): AppSec check: broken pipe, client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/19 17:20:06 [alert] 80110#80110: *51203 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '10.42.0.11' with 'ban' (by appsec), client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"

(Domain name changed for privacy and yes, I use a non-typical subnet)

Disabling Appsec fixes the issue, albeit with the trade off of reduced security.

Thanks.

[blotus]

Hey,

Do you know how big is the body of the request when you get this error ?

Currently, the appsec will try to process any body it sees, regardless of the size, which will lead to issues (also tracked here: #71).

I haven't performed any real tests to see where the actual limit is currently, but I guess anything over a few hundred MBs will trigger this error.

Once #80 is merged (it includes a large refactoring of the code, so we have to wait for it), we plan to add additional configuration on how to handle large bodies (allow to set a maximum body size and whether to drop the request or just analyze the headers when it's over the limit).

[MFYDev]

I just met this issue today with my own Ghost blog post editing. appsec will block me from time to time as I was editing posts. No solution found yet but fortunately saw this issue, for post editing I think the body is not huge at all as I just started.

[MaximumFish]

Do you know how big is the body of the request when you get this error ?

The photos are a couple of MB each and it does one request per photo, so definitely not in the few hundred MB territory.

[yurividal]

Having the same issue.
I'm using nginxproxymanager with crowdsec_openresty_bouncer and appsec.

Issue happens even when inside the LAN, so no cloudflare on anything else involved.
Appsec is, in theory, setup properly and running fine:

curl -I -X POST localhost:7422/ -i -H 'x-crowdsec-appsec-api-key: {redacted}' -H 'x-crowdsec-appsec-ip: 42.42.42.42' -H 'x-crowdsec-appsec-uri: /test' -H 'x-crowdsec-appsec-host: test.com' -H 'x-crowdsec-appsec-verb: GET'   
HTTP/1.1 200 OK
Date: Tue, 03 Dec 2024 23:04:30 GMT
Content-Length: 36
Content-Type: text/plain; charset=utf-8

what could be causing the broken pipe between the appsec and the proxy?

edit:

Initially, the containers were on different docker networks, and communicating via ports exposed to the host.
I then moved the 2 containers to the same docker network. Now the error is a little different. connection reset by peer, instead of broken pipe.

error 10.10.10.20 -> photos.****.com : Allow(): AppSec check: connection reset by peer - POST /api/assets HTTP/1.1 - 
error 10.10.10.20 -> photos.****.com : AppSecCheck(): Fallback because of err: connection reset by peer - POST /api/assets HTTP/1.1 - 

[LaurenceJJones]

Just so it doesnt seem like we are being a PITA about replication steps, I have setup NPMPlus using the steps outlined in the repo.

I created a base immich deployment using immich setup, I then setup immich to be exposed locally via:

ports:
      - '127.0.0.1:2283:2283'

I then setup a local alias called app.debian.local for the immich app

I then proceed to upload an image of my cat which is 3.1mb and these are the logs for NPM and CrowdSec:

npmplus   | 2025/01/17 20:32:48 [warn] 369#369: *1 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000001 while reading request body, client: 192.168.121.1, server: app.debian.local, request: "POST /api/assets HTTP/1.1", host: "app.debian.local", referrer: "http://app.debian.local/photos"
crowdsec  | time="2025-01-17T20:33:04Z" level=info msg="172.17.0.1 - [Fri, 17 Jan 2025 20:33:04 GMT] \"GET /v1/decisions?ip=192.168.121.1 HTTP/1.1 200 7.569899ms \"crowdsec-npmplus-bouncer/v1.0.8\" \""

Note this is all done locally not over https as I dont have a spare domain and dont want to spend time spinning up a VPS as the TLS layer shouldnt impact this.

also appsec is live cause a request to .env causes a block

crowdsec  | time="2025-01-17T20:38:34Z" level=info msg="AppSec block: crowdsecurity/vpatch-env-access from 192.168.121.1 (172.17.0.1)"
npmplus   | 2025/01/17 20:38:34 [alert] 369#369: *240 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '192.168.121.1' with 'ban' (by appsec), client: 192.168.121.1, server: app.debian.local, request: "GET /.env HTTP/1.1", host: "app.debian.local"

Edit: just to ensure it nothing with TLS, I generated a self signed certificate so http2 is at play cause I think npm doesnt enable it unless their is a certificate and still struggling to replicate.

[yurividal]

@LaurenceJJones
Thanks for that. Can you try with these nginx configs? this is what is recommended by the immich team (just add it to your custom nginx config in npm plus:

client_max_body_size 50000M;

# Set headers
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

# enable websockets: http://nginx.org/en/docs/http/websocket.html
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_redirect off;

# set timeout
proxy_read_timeout 600s;
proxy_send_timeout 600s;
send_timeout 600s;

I also tried adding this, but no luck:

modsecurity_rules 'SecRequestBodyLimit 1048576';
modsecurity_rules 'SecRequestBodyNoFilesLimit 1048576';

[yurividal]

One more thing, @LaurenceJJones
Where exactly is this error message being generated?

[lua] crowdsec.lua:651: Allow(): AppSec check: broken pipe, client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"

Does this mean that nginx lua component is having issues when talking to crowdsec's appsec agent?
What would be the best way to debug this? can the logs be made more verbose?

[Zoey2936]

client_max_body_size 50000M;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

These are all set by npmplus, setting them twice causes more issues than anything else, also you should not use $http_host, $host is correct and the default used by npmplus

modsecurity_rules 'SecRequestBodyLimit 1048576';
modsecurity_rules 'SecRequestBodyNoFilesLimit 1048576';

you need to set them in the modsec config file, not as nginx directives