'podman push' fails if GET/HEAD succeeds without authentication, but POST returns 401

[ffplexim]

Issue Description

I have a local container registry behind an nginx proxy that is configured as follows:

• GET, HEAD, and OPTIONS requests do not require authentication;
• every other HTTP method requires HTTP basic authentication.

The idea is that everyone can pull, but only authenticated users can push.

Now I've logged into the registry with podman login (that is always successful, also for wrong credentials, since the requests it does do not require authentication), but when I try to push an image with podman push it fails:

% podman --log-level=debug push myregistry.local/tests/hello-world:latest
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called push.PersistentPreRunE(podman --log-level=debug push myregistry.local/tests/hello-world:latest) 
DEBU[0000] SSH Ident Key "/Users/me/.local/share/containers/podman/machine/machine" SHA256:21Nl633r+TBVZjIpNYTam9FgZHdNsuOJRNTCDIejQn4 ssh-ed25519 
DEBU[0000] DoRequest Method: GET URI: http://d/v6.0.1/libpod/_ping 
DEBU[0000] Found credentials for myregistry.local in credential helper containers-auth.json in file /Users/me/.config/containers/auth.json 
DEBU[0000] DoRequest Method: POST URI: http://d/v6.0.1/libpod/images/myregistry.local/tests/hello-world:latest/push 
Getting image source signatures
Copying blob sha256:55f6d36790cbd258a2173e1599268519414f9973696bcfc0f5193c1accee0951
Error: writing blob: initiating layer upload to /v2/tests/hello-world/blobs/uploads/ in myregistry.local: authentication required
DEBU[0000] Shutting down engines                        

The nginx log:

192.168.1.10 - - [22/Jul/2026:11:51:18 +0000] "GET /v2/ HTTP/1.1" 200 2 "-" "containers/5.34.1 (github.com/containers/image)" "-"
192.168.1.10 - - [22/Jul/2026:11:51:18 +0000] "HEAD /v2/tests/hello-world/blobs/sha256:55f6d36790cbd258a2173e1599268519414f9973696bcfc0f5193c1accee0951 HTTP/1.1" 404 0 "-" "containers/5.34.1 (github.com/containers/image)" "-"
192.168.1.10 - - [22/Jul/2026:11:51:18 +0000] "HEAD /v2/tests/hello-world/blobs/sha256:1ff9adeff4443b503b304e7aa4c37bb90762947125f4a522b370162a7492ff47 HTTP/1.1" 404 0 "-" "containers/5.34.1 (github.com/containers/image)" "-"
192.168.1.10 - - [22/Jul/2026:11:51:18 +0000] "POST /v2/tests/hello-world/blobs/uploads/ HTTP/1.1" 401 179 "-" "containers/5.34.1 (github.com/containers/image)" "-"

As you can see, the GET/HEAD requests just work, and from that podman apparently infers that the POST request should also just work. When the POST request returns 401, instead of re-trying with the stored credentials, it simply fails.

(I also temporarily changed the nginx log output to debug format, and it explicitly confirmed no user/password was provided for basic authentication for the POST - so it explicitly wasn't a typo in the credentials, but simply no Authorization header got sent.)

Steps to reproduce the issue

Steps to reproduce the issue

1. Set up a registry where GET/HEAD/OPTIONS can be done by any user, but other HTTP methods require authorization
2. Log into the registry with podman login
3. Try to push an image with podman push

Describe the results you received

podman push fails, and the HTTP server log shows that podman simply gave up after retrieving a 401, despite it having credentials it can use.

Describe the results you expected

I can push the image. (Or at least I would get a step further, namely that after the 401 podman push tries again with an Authorization header.)

podman info output

(I also tried this with a different podman version on a Linux box, with the same result.)

Client:
  APIVersion: 6.0.2
  BuildOrigin: brew
  Built: 1784664683
  BuiltTime: Tue Jul 21 22:11:23 2026
  GitCommit: ""
  GoVersion: go1.26.5
  Os: darwin
  OsArch: darwin/arm64
  Version: 6.0.2
host:
  arch: arm64
  buildahVersion: 1.39.2
  cdiSpecDirs: null
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-3.fc41.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 96.9
    systemPercent: 1.84
    userPercent: 1.26
  cpus: 6
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "41"
  eventLogger: journald
  freeLocks: 2047
  hostname: localhost.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.12.13-200.fc41.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 3015950336
  memTotal: 3791691776
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.14.0-1.fc41.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: netavark-1.14.0-1.fc41.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.14.0
  ociRuntime:
    name: crun
    package: crun-1.20-2.fc41.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.20
      commit: 9c9a76ac11994701dd666c4f0b869ceffb599a66
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250121.g4f2c8e7-2.fc41.aarch64
    version: |
      pasta 0^20250121.g4f2c8e7-2.fc41.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: unix:///run/podman/podman.sock
  rootlessNetworkCmd: pasta
  rootlessPortForwarder: ""
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  swapFree: 0
  swapTotal: 0
  uptime: 0h 0m 28.00s
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.additionalImageStores:
    - /usr/lib/containers/storage
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 98899800064
  graphRootUsed: 8574558208
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 16
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.4.1
  BuildOrigin: Fedora Project
  Built: 1741651200
  BuiltTime: Tue Mar 11 01:00:00 2025
  GitCommit: b79bc8afe796cba51dd906270a7e1056ccdfcf9e
  GoVersion: go1.23.7
  Os: linux
  OsArch: linux/arm64
  Version: 5.4.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional information

[ffplexim]

The same happens when using skopeo copy. On the other hand, docker push works.

[Luap99]

I defer to @mtrmac on the protocol details but always making /v2/ return 200 OK even with an invalid auth header is at least making podman login always succeed and the spec seems to hint it should return 401
https://github.com/opencontainers/distribution-spec/blob/main/spec.md#determining-support

As per the code it expects WWW-Authenticate to be set on the GET /v2/ requests.

container-libs/image/docker/docker_client.go

Lines 730 to 737 in 4281c42

	// we're using the challenges from the /v2/ ping response and not the one from the destination
	// URL in this request because:
	//
	// 1) docker does that as well
	// 2) gcr.io is sending 401 without a WWW-Authenticate header in the real request
	//
	// debugging: https://github.com/containers/image/pull/211#issuecomment-273426236 and follows up
	func (c *dockerClient) setupRequestAuth(req *http.Request, extraScope *authScope) error {

Anyhow not a problem with the podman code base so I move it into container-libs.

[ffplexim]

I defer to @mtrmac on the protocol details but always making /v2/ return 200 OK even with an invalid auth header is at least making podman login always succeed

That's not the case - no auth header is sent at all if /v2/ does not return a 401, but 200. So there is no invalid auth header, it's just that podman / skopeo / containerlib thinks that everything is fine. Also docker login shows the same behavior, and from my POV this isn't a problem (I also don't see what any of them could do differently for login).

In any case, thanks for moving this to the right place!

[mtrmac]

Thanks for reaching out.

Yes, you have found the relevant code/comment, and that also explains why we can’t change the behavior (or at least why we couldn’t historically have changed it).

That said, there already is some code there to retry if a later HTTP operation requires more / different permissions than the original /v2/; that could possibly be extended (needsRetryWithUpdatedScope could also be updating c.challenges).

For a start, can you collect the relevant debug logs, please, to confirm whether we are even hitting that code path now? The quoted log is from the remote client; a log from the server should be rather more detailed about the HTTP operations.

[felixfontein]

What kind of server log are you asking for? I've included the log from the HTTP proxy (nginx), which shows the requests podman send to the registry.

[mtrmac]

The Podman process that is a server to podman remote (and a client to the nginx).

[ffplexim]

I used podman machine ssh to get into the VM, I guess that's the right log:

% podman machine ssh podman-machine-default
Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
Fedora CoreOS 41.20250215.3.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

Last login: Wed Jul 22 12:04:37 2026
root@localhost:~# podman --log-level=debug push myregistry.local/tests/hello-world:latest
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called push.PersistentPreRunE(podman --log-level=debug push myregistry.local/tests/hello-world:latest) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using sqlite as database backend             
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /run/containers/storage       
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/libpod                    
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: imagestore=/usr/lib/containers/storage 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is being used 
DEBU[0000] NewControl(/var/lib/containers/storage/overlay): nextProjectID = 2758333996 
DEBU[0000] Cached value indicated that native-diff is not being used 
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled 
DEBU[0000] backingFs=xfs, projectQuotaSupported=true, useNativeDiff=false, usingMetacopy=true 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 19             
DEBU[0000] Looking up image "myregistry.local/tests/hello-world:latest" in local containers storage 
DEBU[0000] Normalized platform linux/arm64 to {arm64 linux  [] } 
DEBU[0000] Trying "myregistry.local/tests/hello-world:latest" ... 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/usr/lib/containers/storage,overlay.mountopt=nodev,metacopy=on]@83fc7ce1224f5ed3885f6aaec0bb001c0bbb2a308e3250d7408804a720c72a32" 
DEBU[0000] Found image "myregistry.local/tests/hello-world:latest" as "myregistry.local/tests/hello-world:latest" in local containers storage 
DEBU[0000] Found image "myregistry.local/tests/hello-world:latest" as "myregistry.local/tests/hello-world:latest" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/usr/lib/containers/storage,overlay.mountopt=nodev,metacopy=on]@83fc7ce1224f5ed3885f6aaec0bb001c0bbb2a308e3250d7408804a720c72a32) 
DEBU[0000] exporting opaque data as blob "sha256:83fc7ce1224f5ed3885f6aaec0bb001c0bbb2a308e3250d7408804a720c72a32" 
DEBU[0000] Pushing image myregistry.local/tests/hello-world:latest to myregistry.local/tests/hello-world:latest 
DEBU[0000] Normalized platform linux/arm64 to {arm64 linux  [] } 
DEBU[0000] Copying source image [overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/usr/lib/containers/storage,overlay.mountopt=nodev,metacopy=on]@83fc7ce1224f5ed3885f6aaec0bb001c0bbb2a308e3250d7408804a720c72a32 to destination image //myregistry.local/tests/hello-world:latest 
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/999-podman-desktop-registries-from-host.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/999-podman-machine.conf" 
DEBU[0000] No credentials matching myregistry.local/tests/hello-world found in /run/user/0/containers/auth.json 
DEBU[0000] No credentials matching myregistry.local/tests/hello-world found in /root/.config/containers/auth.json 
DEBU[0000] No credentials matching myregistry.local/tests/hello-world found in /root/.docker/config.json 
DEBU[0000] No credentials matching myregistry.local/tests/hello-world found in /root/.dockercfg 
DEBU[0000] No credentials for myregistry.local/tests/hello-world found 
DEBU[0000]  No signature storage configuration found for myregistry.local/tests/hello-world:latest, using built-in default file:///var/lib/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/myregistry.local 
DEBU[0000] Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite 
DEBU[0000] IsRunningImageAllowed for image containers-storage:[overlay@/var/lib/containers/storage]@83fc7ce1224f5ed3885f6aaec0bb001c0bbb2a308e3250d7408804a720c72a32 
DEBU[0000]  Using transport "containers-storage" policy section "" 
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
DEBU[0000] exporting opaque data as blob "sha256:83fc7ce1224f5ed3885f6aaec0bb001c0bbb2a308e3250d7408804a720c72a32" 
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0000] ... will first try using the original manifest unmodified 
DEBU[0000] Checking if we can reuse blob sha256:55f6d36790cbd258a2173e1599268519414f9973696bcfc0f5193c1accee0951: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar" = true 
DEBU[0000] Checking /v2/tests/hello-world/blobs/sha256:55f6d36790cbd258a2173e1599268519414f9973696bcfc0f5193c1accee0951 
DEBU[0000] GET https://myregistry.local/v2/ 
DEBU[0000] Ping https://myregistry.local/v2/ status 200 
DEBU[0000] HEAD https://myregistry.local/v2/tests/hello-world/blobs/sha256:55f6d36790cbd258a2173e1599268519414f9973696bcfc0f5193c1accee0951 
DEBU[0000] ... not present                              
DEBU[0000] Trying to reuse blob with cached digest sha256:55f6d36790cbd258a2173e1599268519414f9973696bcfc0f5193c1accee0951 in destination repo with no location match, checking current repo 
DEBU[0000] ... Already tried the primary destination    
DEBU[0000] Trying to reuse blob with cached digest sha256:1ff9adeff4443b503b304e7aa4c37bb90762947125f4a522b370162a7492ff47 compressed with gzip with no location match, checking current repo 
DEBU[0000] Checking /v2/tests/hello-world/blobs/sha256:1ff9adeff4443b503b304e7aa4c37bb90762947125f4a522b370162a7492ff47 
DEBU[0000] HEAD https://myregistry.local/v2/tests/hello-world/blobs/sha256:1ff9adeff4443b503b304e7aa4c37bb90762947125f4a522b370162a7492ff47 
DEBU[0000] ... not present                              
DEBU[0000] exporting filesystem layer "55f6d36790cbd258a2173e1599268519414f9973696bcfc0f5193c1accee0951" without compression for blob "sha256:55f6d36790cbd258a2173e1599268519414f9973696bcfc0f5193c1accee0951" 
DEBU[0000] No compression detected                      
DEBU[0000] Compressing blob on the fly                  
DEBU[0000] Uploading /v2/tests/hello-world/blobs/uploads/ 
DEBU[0000] POST https://myregistry.local/v2/tests/hello-world/blobs/uploads/ 
DEBU[0000] Error initiating layer upload, response http.Response{Status:"401 Unauthorized", StatusCode:401, Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Connection":[]string{"keep-alive"}, "Content-Length":[]string{"179"}, "Content-Type":[]string{"text/html"}, "Date":[]string{"Thu, 23 Jul 2026 06:49:03 GMT"}, "Server":[]string{"nginx/1.30.4"}, "Www-Authenticate":[]string{"Basic realm=\"Password expected here ...\""}}, Body:(*http.bodyEOFSignal)(0x40001600c0), ContentLength:179, TransferEncoding:[]string(nil), Close:false, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0x40004d8000), TLS:(*tls.ConnectionState)(0x4000636000)} 
Copying blob 55f6d36790cb [--------------------------------------] 8.0b / 561.0KiB | 3.0 MiB/s
DEBU[0000] Looking up image "myregistry.local/tests/hello-world:latest" in local containers storage 
DEBU[0000] Normalized platform linux/arm64 to {arm64 linux  [] } 
DEBU[0000] Trying "myregistry.local/tests/hello-world:latest" ... 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/usr/lib/containers/storage,overlay.mountopt=nodev,metacopy=on]@83fc7ce1224f5ed3885f6aaec0bb001c0bbb2a308e3250d7408804a720c72a32" 
DEBU[0000] Found image "myregistry.local/tests/hello-world:latest" as "myregistry.local/tests/hello-world:latest" in local containers storage 
Error: writing blob: initiating layer upload to /v2/tests/hello-world/blobs/uploads/ in myregistry.local: authentication required
DEBU[0000] Shutting down engines                        
INFO[0000] Received shutdown.Stop(), terminating!        PID=2445

[mtrmac]

Thanks, yes, that’s the right log.

DEBU[0000] Error initiating layer upload, response http.Response{Status:"401 Unauthorized", StatusCode:401, … "Www-Authenticate":[]string{"Basic realm=\"Password expected here ...\""}}

OK, that runs into one more reason why the needsRetryWithUpdatedScope code does not engage:

• [As mentioned previously, it does not update c.challenges, so it doesn’t currently help]
• The returned error message does not report an insufficient_scope error. Reasonably solvable if we did not have any challenge previously.