Issues Uploading Container Images to Sonatype Nexus3

[rfpludwick]

I run Nexus3 for an artifactory, and at some point (unsure of the trigger) I have started to see Concourse unable to properly push container images to my registry. I do not observe this behavior when using the docker CLI tool to push from my laptop to the same registry. It can pull container images from my Nexus registry just fine.

My redactions are shown below as <redacted> just so you know those did not come from Concourse.

Here's my main NGINX reverse proxy config for Nexus (it's running in k8s, as is Concourse, but this reverse proxy is a separate device not on k8s):

upstream k8s_nx {
    zone                 upstream_k8s 128k;

    server               <redacted>:31013;
}

server {
    listen               443 ssl;
    server_name          <redacted>;

    error_log            /var/log/nginx/nx/error.log warn;
    access_log           /var/log/nginx/nx/access.log;

    include              /etc/nginx/proxy_params;
    include              /etc/nginx/ssl_params;
    include              /etc/nginx/external_error_handling;

    client_max_body_size 0;

    location ~ ^/(v[12])/[^/]+/?[^/]+/blobs/ {
        include          /etc/nginx/internal_only;

        if ($request_method ~* (POST|PUT|DELETE|PATCH|HEAD)) {
            rewrite      ^/(.*)$ /repository/docker-hosted/$1 last;
        }

        rewrite          ^/(.*)$ /repository/docker-group/$1 last;
    }

    location ~ ^/(v[12])/ {
        include          /etc/nginx/internal_only;

        if ($request_method ~* (POST|PUT|DELETE|PATCH)) {
            rewrite      ^/(.*)$ /repository/docker-hosted/$1 last;
        }

        rewrite          ^/(.*)$ /repository/docker-group/$1 last;
    }

    location / {
        include          /etc/nginx/internal_only;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_ssl_verify off;
        proxy_buffering  off;
        proxy_pass       http://k8s_nx;
    }
}

Here's an output from the put job:

elected worker: concourse-worker-0
INFO[0000] pushing tag(s) latest, ((redacted))                
2026/02/23 18:25:31 existing blob: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
2026/02/23 18:25:31 retrying PATCH https://((redacted))/v2/docker-hosted/home-lab/<redacted>/blobs/uploads/0b88aa2c-3ca4-4dc9-b81d-30de26b17049: UNKNOWN: unknown
2026/02/23 18:25:31 retrying PATCH https://((redacted))/v2/docker-hosted/home-lab/<redacted>/blobs/uploads/4e238eab-8eb2-45ca-a897-590b3432e9fa: UNKNOWN: unknown
2026/02/23 18:25:31 retrying PATCH https://((redacted))/v2/docker-hosted/home-lab/<redacted>/blobs/uploads/96463ff8-cc7d-46c8-8b85-ede5a6598112: UNKNOWN: unknown
2026/02/23 18:25:32 retrying PATCH https://((redacted))/v2/docker-hosted/home-lab/<redacted>/blobs/uploads/a49bdc63-0f50-4034-9e33-3d615c72e72e: UNKNOWN: unknown
2026/02/23 18:25:32 retrying PATCH https://((redacted))/v2/docker-hosted/home-lab/<redacted>/blobs/uploads/77e45e00-c2be-4f19-806a-b2bc3ad5c0f2: UNKNOWN: unknown
2026/02/23 18:25:32 retrying PATCH https://((redacted))/v2/docker-hosted/home-lab/<redacted>/blobs/uploads/47d4291b-fb12-4f95-b746-dbeaf9a4fe67: UNKNOWN: unknown
2026/02/23 18:25:35 retrying PATCH https://((redacted))/v2/docker-hosted/home-lab/<redacted>/blobs/uploads/164116e4-34bb-4ad2-a0e8-32a1a5dd389e: UNKNOWN: unknown
2026/02/23 18:25:35 retrying Head "https://((redacted))/v2/home-lab/<redacted>/blobs/sha256:c7289bb90496598f7662a2729909228dc1de895d3358115e43c97a5502e868f7": PATCH https://((redacted))/v2/docker-hosted/home-lab/<redacted>/blobs/uploads/164116e4-34bb-4ad2-a0e8-32a1a5dd389e: UNKNOWN: unknown
2026/02/23 18:25:35 retrying Head "https://((redacted))/v2/home-lab/<redacted>/blobs/sha256:d8ad8cd72600f46cc068e16c39046ebc76526e41051f43a8c249884b200936c0": PATCH https://((redacted))/v2/docker-hosted/home-lab/<redacted>/blobs/uploads/164116e4-34bb-4ad2-a0e8-32a1a5dd389e: UNKNOWN: unknown
ERRO[0004] pushing image failed: pushing tag(s): PATCH https://((redacted))/v2/docker-hosted/home-lab/<redacted>/blobs/uploads/164116e4-34bb-4ad2-a0e8-32a1a5dd389e: UNKNOWN: unknown 

I realize this issue may be on the Nexus side and not Concourse. I suspect a recent Nexus upgrade I ran is what caused this, but I cannot be sure as I cannot nail down the timeline.

[taylorsilva]

In the registry-image, the uploading of images is done by https://github.com/google/go-containerregistry
and searching in there for UNKNOWN, leads me here: https://github.com/google/go-containerregistry/blob/85f2bf5f7303fc95161f334e4fc2e66c36e12b1e/pkg/v1/remote/transport/error.go#L132
Which suggests to me that we're getting back an unknown error code from the server.

Your hunch that it's an issue on the Nexus end seems likely. You could try checking their repo for similar issues as well.

[rfpludwick]

@taylorsilva I finally found the line in the Nexus log which points me to my reverse-proxy configuration. For some reason when uploading via the docker CLI I do not encounter this issue; I do in Concourse. I'll continue my investigation. Unless you have some insight, I consider this discussion closed.

2026-03-09 08:50:16,165-0700 WARN  [qtp283669489-83] concourse_ci_worker org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: class org.sonatype.nexus.repository.view.Request /v2/docker-hosted/home-lab/<redacted>/blobs/uploads/7866c769-88f7-4848-b7b3-c143bb0b4d6e
java.lang.IllegalStateException: Attempting to add to upload with uuid: 7866c769-88f7-4848-b7b3-c143bb0b4d6e and name: home-lab/<redacted> with different name: docker-hosted/home-lab/<redacted>

[rfpludwick]

Further closing the loop here for @taylorsilva - Unsure exactly how this happened, but my Docker settings in Nexus had path-based routing enabled for the docker-hosted repository. This seemed to be breaking the uploads from Concourse. I'm not entirely sure how it was enabled; it was from one of two possibilities:

1. I had to rebuild my Docker repos when I upgraded Nexus from an age-old version and maybe I missed this setting (unlikely as I believe I tested uploading from Concourse afterwards)
2. An interim Nexus upgrade flipped that on (seems more likely than 1, but it shouldn't be likely at all)

Flipping it to blank solved my problem.

To everybody who reads this discussion, let this be a lesson - check all of your possible logs before raising a GitHub discussion. I should have checked my Nexus logs earlier.

[taylorsilva]

Hopefully this proves helpful to someone else with a similar setup in the future! I find submitting issues like is a nice way to share knowledge and help yourself "rubber duck" through the problem :)