Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to get the image from the http repository (http: server gave HTTP response to HTTPS client) #5667

Closed
lomtom opened this issue Jan 16, 2025 · 9 comments

Comments

@lomtom
Copy link

lomtom commented Jan 16, 2025

I built the image with k8s + job, but I couldn't get the mirror image after FROM in Dockerfile.

apiVersion: batch/v1
kind: Job
metadata:
  name: buildkit
spec:
  template:
    metadata:
      annotations:
        container.apparmor.security.beta.kubernetes.io/buildkit: unconfined
    # see buildkit/docs/rootless.md for caveats of rootless mode
    spec:
      restartPolicy: Never
      initContainers:
        - name: prepare-dockerfile
          image: bash:5
          command:
            - sh
            - -c
            - |
              cat > /workspace/Dockerfile <<EOF
              FROM 192.168.80.235:32255/library/bash:5
              EOF
          securityContext:
            runAsUser: 1000
            runAsGroup: 1000
          volumeMounts:
            - name: workspace
              mountPath: /workspace
        - name: prepare-docker-config
          image: bash:5
          command:
            - /bin/sh
            - -c
            - "echo '{\"auths\":{\"192.168.80.235:32255\":{\"auth\":\"*********\"}}}' > /workspace/config.json && cat /workspace/config.json"
          securityContext:
            runAsUser: 1000
            runAsGroup: 1000
          volumeMounts:
            - mountPath: /workspace
              name:  workspace
      containers:
        - name: buildkit
          image: moby/buildkit:master-rootless
          env:
            - name: BUILDKITD_FLAGS
              value: --oci-worker-no-process-sandbox
            - name: DOCKER_CONFIG
              value: /workspace
          command:
            - buildctl-daemonless.sh
          args:
            - build
            - --frontend
            - dockerfile.v0
            - --local
            - context=/workspace
            - --local
            - dockerfile=/workspace
            - --opt
            - platform=linux/amd64,linux/arm64
            - --output
            - type=image,name=192.168.80.235:32255/library/bash:5-1,push=true,registry.insecure=true
          securityContext:
            # Needs Kubernetes >= 1.19
            seccompProfile:
              type: Unconfined
            # To change UID/GID, you need to rebuild the image
            runAsUser: 1000
            runAsGroup: 1000
          volumeMounts:
            - name: workspace
              readOnly: true
              mountPath: /workspace
      volumes:
        - name: workspace
          emptyDir: {}

it will show some error

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile:
#1 transferring dockerfile: 158B done
#1 DONE 0.9s

#2 [linux/amd64 internal] load metadata for 192.168.80.235:32255/library/bash:5
#2 CANCELED

#3 [linux/arm64 internal] load metadata for 192.168.80.235:32255/library/bash:5
#3 ERROR: failed to do request: Head "https://192.168.80.235:32255/v2/library/bash/manifests/5": http: server gave HTTP response to HTTPS client
------
 > [linux/arm64 internal] load metadata for 192.168.80.235:32255/library/bash:5:
------
Dockerfile:1
--------------------
   1 | >>> FROM 192.168.80.235:32255/library/bash:5
   2 |     
--------------------
error: failed to solve: 192.168.80.235:32255/library/bash:5: failed to resolve source metadata for 192.168.80.235:32255/library/bash:5: failed to do request: Head "https://192.168.80.235:32255/v2/library/bash/manifests/5": http: server gave HTTP response to HTTPS client

I may not be able to access it because the harbor is http.

But I looked through all the issues and documents, but I couldn't find a solution. What should I do to make it executed correctly?

Thank you

@hf43hf
Copy link

hf43hf commented Jan 17, 2025

I had the same problem

@AkihiroSuda AkihiroSuda changed the title Unable to get the image from the http repository Unable to get the image from the http repository (http: server gave HTTP response to HTTPS client) Jan 17, 2025
@tonistiigi
Copy link
Member

Pulling images from insecure HTTP repositories requires these registries to be defined as insecure in buildkitd TOML config.

@lomtom
Copy link
Author

lomtom commented Jan 20, 2025

I get a solution through daemonless: cannot add private repository certificates

Pulling images from insecure HTTP repositories requires these registries to be defined as insecure in buildkitd TOML config.

we Need to create a buildkitd configuration file in the directory /home/user/.config/buildkit for daemonless

cat > /home/user/.config/buildkit/buildkitd.toml <<EOF
[registry."192.168.80.235:32255"]
http = true
insecure = true
EOF

Finally

apiVersion: batch/v1
kind: Job
metadata:
  name: buildkit
spec:
  template:
    metadata:
      annotations:
        container.apparmor.security.beta.kubernetes.io/buildkit: unconfined
    spec:
      restartPolicy: Never
      initContainers:
        - name: prepare-dockerfile
          image: library/bash:5
          command:
            - sh
            - -c
            - |
              cat > /workspace/Dockerfile <<EOF
              FROM 192.168.80.235:32255/library/bash:5
              EOF
          securityContext:
            runAsUser: 1000
            runAsGroup: 1000
          volumeMounts:
            - name: workspace
              mountPath: /workspace
        - name: prepare-docker-config
          image: library/bash:5
          command:
            - /bin/sh
            - -c
            - "echo '{\"auths\":{\"192.168.80.235:32255\":{\"auth\":\"******\"}}}' > /workspace/config.json && cat /workspace/config.json"
          securityContext:
            runAsUser: 1000
            runAsGroup: 1000
          volumeMounts:
            - mountPath: /workspace
              name:  workspace
        - name: prepare-toml
          image: library/bash:5
          command:
            - /bin/sh
            - -c
            - |
              cat > /home/user/.config/buildkit/buildkitd.toml <<EOF
              [registry."192.168.80.235:32255"]
              http = true
              insecure = true
              EOF
          securityContext:
            runAsUser: 1000
            runAsGroup: 1000
          volumeMounts:
            - mountPath: /home/user/.config/buildkit
              name:  buildkit
      containers:
        - name: buildkit
          image: moby/buildkit:master-rootless
          env:
            - name: BUILDKITD_FLAGS
              value: --oci-worker-no-process-sandbox
            - name: DOCKER_CONFIG
              value: /workspace
          command:
            - buildctl-daemonless.sh
          args:
            - build
            - --frontend
            - dockerfile.v0
            - --local
            - context=/workspace
            - --local
            - dockerfile=/workspace
            - --opt
            - platform=linux/amd64,linux/arm64
            - --output
            - type=image,name=192.168.80.235:32255/library/bash:5-1,push=true
          securityContext:
            # Needs Kubernetes >= 1.19
            seccompProfile:
              type: Unconfined
            # To change UID/GID, you need to rebuild the image
            runAsUser: 1000
            runAsGroup: 1000
          volumeMounts:
            - name: workspace
              mountPath: /workspace
            - name:  buildkit
              mountPath: /home/user/.config/buildkit
      volumes:
        - name: workspace
          emptyDir: {}
        - name: buildkit
          emptyDir: {}

the log:

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 134B done
#1 DONE 0.0s

#2 [auth] ouyanglongtong/bash:pull token for 192.168.80.235:32255
#2 DONE 0.0s

#3 [linux/amd64 internal] load metadata for 192.168.80.235:32255/library/bash:5
#3 ...

#4 [linux/arm64 internal] load metadata for 192.168.80.235:32255/library/bash:5
#4 DONE 0.8s

#3 [linux/amd64 internal] load metadata for 192.168.80.235:32255/library/bash:5
#3 DONE 0.8s

#5 [internal] load .dockerignore
#5 transferring context: 2B done
#5 DONE 0.0s

#6 [linux/arm64 1/1] FROM 192.168.80.235:32255/library/bash:5@sha256:1ed86e5fdfd1868db6926222367e8bf95951d77e2369d2248ccaef0935c3d368
#6 resolve 192.168.80.235:32255/library/bash:5@sha256:1ed86e5fdfd1868db6926222367e8bf95951d77e2369d2248ccaef0935c3d368 0.0s done
#6 DONE 0.1s

#7 [linux/amd64 1/1] FROM 192.168.80.235:32255/library/bash:5@sha256:1ed86e5fdfd1868db6926222367e8bf95951d77e2369d2248ccaef0935c3d368
#7 resolve 192.168.80.235:32255/library/bash:5@sha256:1ed86e5fdfd1868db6926222367e8bf95951d77e2369d2248ccaef0935c3d368 0.0s done
#7 DONE 0.1s

#8 [auth] library/bash:pull,push token for 192.168.80.235:32255
#8 DONE 0.0s

#9 exporting to image
#9 exporting layers done
#9 exporting manifest sha256:97e41d808c474ec951f03d2b42043d02e9b90e1113c00385d496cc93e3b404fb done
#9 exporting config sha256:bd4206c5bc03671d4628a19a518f7e4c7ddf5ff8026fc035188b9615386f6df1 done
#9 exporting manifest sha256:db3f28b9b1bdf1c6ba413b6f94e97339a1fe84efe5272382a1750f1fc2565c68 done
#9 exporting config sha256:5c664b1446dec19998870821e26efedfce64b394040431c38a08531b78f73942 done
#9 exporting manifest list sha256:1ed86e5fdfd1868db6926222367e8bf95951d77e2369d2248ccaef0935c3d368 done
#9 pushing layers
#9 pushing layers 0.3s done
#9 pushing manifest for 192.168.80.235:32255/library/bash:5-1@sha256:1ed86e5fdfd1868db6926222367e8bf95951d77e2369d2248ccaef0935c3d368

@lomtom lomtom closed this as completed Jan 20, 2025
@hf43hf
Copy link

hf43hf commented Jan 20, 2025

I get a solution through daemonless: cannot add private repository certificates

Pulling images from insecure HTTP repositories requires these registries to be defined as insecure in buildkitd TOML config.

we Need to create a buildkitd configuration file in the directory /home/user/.config/buildkit for daemonless

cat > /home/user/.config/buildkit/buildkitd.toml <<EOF
[registry."192.168.80.235:32255"]
http = true
insecure = true
EOF
Finally

apiVersion: batch/v1
kind: Job
metadata:
name: buildkit
spec:
template:
metadata:
annotations:
container.apparmor.security.beta.kubernetes.io/buildkit: unconfined
spec:
restartPolicy: Never
initContainers:
- name: prepare-dockerfile
image: library/bash:5
command:
- sh
- -c
- |
cat > /workspace/Dockerfile <<EOF
FROM 192.168.80.235:32255/library/bash:5
EOF
securityContext:
runAsUser: 1000
runAsGroup: 1000
volumeMounts:
- name: workspace
mountPath: /workspace
- name: prepare-docker-config
image: library/bash:5
command:
- /bin/sh
- -c
- "echo '{"auths":{"192.168.80.235:32255":{"auth":"******"}}}' > /workspace/config.json && cat /workspace/config.json"
securityContext:
runAsUser: 1000
runAsGroup: 1000
volumeMounts:
- mountPath: /workspace
name: workspace
- name: prepare-toml
image: library/bash:5
command:
- /bin/sh
- -c
- |
cat > /home/user/.config/buildkit/buildkitd.toml <<EOF
[registry."192.168.80.235:32255"]
http = true
insecure = true
EOF
securityContext:
runAsUser: 1000
runAsGroup: 1000
volumeMounts:
- mountPath: /home/user/.config/buildkit
name: buildkit
containers:
- name: buildkit
image: moby/buildkit:master-rootless
env:
- name: BUILDKITD_FLAGS
value: --oci-worker-no-process-sandbox
- name: DOCKER_CONFIG
value: /workspace
command:
- buildctl-daemonless.sh
args:
- build
- --frontend
- dockerfile.v0
- --local
- context=/workspace
- --local
- dockerfile=/workspace
- --opt
- platform=linux/amd64,linux/arm64
- --output
- type=image,name=192.168.80.235:32255/library/bash:5-1,push=true
securityContext:
# Needs Kubernetes >= 1.19
seccompProfile:
type: Unconfined
# To change UID/GID, you need to rebuild the image
runAsUser: 1000
runAsGroup: 1000
volumeMounts:
- name: workspace
mountPath: /workspace
- name: buildkit
mountPath: /home/user/.config/buildkit
volumes:
- name: workspace
emptyDir: {}
- name: buildkit
emptyDir: {}
the log:

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 134B done
#1 DONE 0.0s

#2 [auth] ouyanglongtong/bash:pull token for 192.168.80.235:32255
#2 DONE 0.0s

#3 [linux/amd64 internal] load metadata for 192.168.80.235:32255/library/bash:5
#3 ...

#4 [linux/arm64 internal] load metadata for 192.168.80.235:32255/library/bash:5
#4 DONE 0.8s

#3 [linux/amd64 internal] load metadata for 192.168.80.235:32255/library/bash:5
#3 DONE 0.8s

#5 [internal] load .dockerignore
#5 transferring context: 2B done
#5 DONE 0.0s

#6 [linux/arm64 1/1] FROM 192.168.80.235:32255/library/bash:5@sha256:1ed86e5fdfd1868db6926222367e8bf95951d77e2369d2248ccaef0935c3d368
#6 resolve 192.168.80.235:32255/library/bash:5@sha256:1ed86e5fdfd1868db6926222367e8bf95951d77e2369d2248ccaef0935c3d368 0.0s done
#6 DONE 0.1s

#7 [linux/amd64 1/1] FROM 192.168.80.235:32255/library/bash:5@sha256:1ed86e5fdfd1868db6926222367e8bf95951d77e2369d2248ccaef0935c3d368
#7 resolve 192.168.80.235:32255/library/bash:5@sha256:1ed86e5fdfd1868db6926222367e8bf95951d77e2369d2248ccaef0935c3d368 0.0s done
#7 DONE 0.1s

#8 [auth] library/bash:pull,push token for 192.168.80.235:32255
#8 DONE 0.0s

#9 exporting to image
#9 exporting layers done
#9 exporting manifest sha256:97e41d808c474ec951f03d2b42043d02e9b90e1113c00385d496cc93e3b404fb done
#9 exporting config sha256:bd4206c5bc03671d4628a19a518f7e4c7ddf5ff8026fc035188b9615386f6df1 done
#9 exporting manifest sha256:db3f28b9b1bdf1c6ba413b6f94e97339a1fe84efe5272382a1750f1fc2565c68 done
#9 exporting config sha256:5c664b1446dec19998870821e26efedfce64b394040431c38a08531b78f73942 done
#9 exporting manifest list sha256:1ed86e5fdfd1868db6926222367e8bf95951d77e2369d2248ccaef0935c3d368 done
#9 pushing layers
#9 pushing layers 0.3s done
#9 pushing manifest for 192.168.80.235:32255/library/bash:5-1@sha256:1ed86e5fdfd1868db6926222367e8bf95951d77e2369d2248ccaef0935c3d368

My environment consists of using nerdctl+buildkitd to build images within Jenkins running on Kubernetes. Currently, I'm mounting /usr/bin/buildctl, /var/run/buildkit/buildkitd.sock, /usr/local/bin/nerdctl, /usr/local/bin/containerd, and /run/containerd/containerd.sock into Jenkins for use. I've tried configuring /etc/buildkit/buildkitd.toml on the Kubernetes node to allow HTTP image registries, but the configuration isn't taking effect. How can I make the /etc/buildkit/buildkitd.toml configuration work?

@lomtom
Copy link
Author

lomtom commented Jan 20, 2025

铁汁,按理说你应该在jenkins用的那个镜像中将buildkitd.toml配置好,或者你直接将主机上的buildkitd.toml配置文件挂载进去试试呢

@hf43hf
Copy link

hf43hf commented Jan 20, 2025

铁汁,按理说你应该在jenkins用的那个镜像中将buildkitd.toml配置好,或者你直接将主机上的buildkitd.toml配置文件挂载进去试试呢

刚才试了下,把/etc/buildkit/buildkitd.toml也一起挂载到jenkins中了,还是没有生效,仍然提示"failed to do request: Head "https://192.168.51.249:8083/v2/library/eclipse-temurin/manifests/17.0.9_9-jdk-jammy": http: server gave HTTP response to HTTPS client",我的buildkitd.toml配置如下:
debug = true
[registry."192.168.51.249:8083"]
http = true
insecure = true

[registry."192.168.51.249:8083".credentials]
username = "admin"
password = "xxxxxxxx"

@lomtom
Copy link
Author

lomtom commented Jan 20, 2025

铁汁,按理说你应该在jenkins用的那个镜像中将buildkitd.toml配置好,或者你直接将主机上的buildkitd.toml配置文件挂载进去试试呢

刚才试了下,把/etc/buildkit/buildkitd.toml也一起挂载到jenkins中了,还是没有生效,仍然提示"failed to do request: Head "https://192.168.51.249:8083/v2/library/eclipse-temurin/manifests/17.0.9_9-jdk-jammy": http: server gave HTTP response to HTTPS client",我的buildkitd.toml配置如下: debug = true [registry."192.168.51.249:8083"] http = true insecure = true

[registry."192.168.51.249:8083".credentials] username = "admin" password = "xxxxxxxx"

你得确认你是rootless还是rootful模式,rootless的配置文件应该在/home/user/.config/buildkit/buildkitd.toml,rootful的才是在/etc/buildkit/buildkitd.toml

@hf43hf
Copy link

hf43hf commented Jan 20, 2025

铁汁,按理说你应该在jenkins用的那个镜像中将buildkitd.toml配置好,或者你直接将主机上的buildkitd.toml配置文件挂载进去试试呢

刚才试了下,把/etc/buildkit/buildkitd.toml也一起挂载到jenkins中了,还是没有生效,仍然提示"failed to do request: Head "https://192.168.51.249:8083/v2/library/eclipse-temurin/manifests/17.0.9_9-jdk-jammy": http: server gave HTTP response to HTTPS client",我的buildkitd.toml配置如下: debug = true [registry."192.168.51.249:8083"] http = true insecure = true
[registry."192.168.51.249:8083".credentials] username = "admin" password = "xxxxxxxx"

你得确认你是rootless还是rootful模式,rootless的配置文件应该在/home/user/.config/buildkit/buildkitd.toml,rootful的才是在/etc/buildkit/buildkitd.toml

应该是rootful模式,因为我从未配置过rootless相关的东西,我直接将buildkitd注册为系统Service服务了:
[Unit]
Description=BuildKit
Documentation=https://github.com/moby/buildkit
[Service]
ExecStart=/usr/local/buildkit/bin/buildkitd --config=/etc/buildkit/buildkitd.toml --oci-worker=false --containerd-worker=true
[Install]
WantedBy=multi-user.target

@lomtom
Copy link
Author

lomtom commented Jan 20, 2025

铁汁,按理说你应该在jenkins用的那个镜像中将buildkitd.toml配置好,或者你直接将主机上的buildkitd.toml配置文件挂载进去试试呢

刚才试了下,把/etc/buildkit/buildkitd.toml也一起挂载到jenkins中了,还是没有生效,仍然提示"failed to do request: Head "https://192.168.51.249:8083/v2/library/eclipse-temurin/manifests/17.0.9_9-jdk-jammy": http: server gave HTTP response to HTTPS client",我的buildkitd.toml配置如下: debug = true [registry."192.168.51.249:8083"] http = true insecure = true
[registry."192.168.51.249:8083".credentials] username = "admin" password = "xxxxxxxx"

你得确认你是rootless还是rootful模式,rootless的配置文件应该在/home/user/.config/buildkit/buildkitd.toml,rootful的才是在/etc/buildkit/buildkitd.toml

应该是rootful模式,因为我从未配置过rootless相关的东西,我直接将buildkitd注册为系统Service服务了: [Unit] Description=BuildKit Documentation=https://github.com/moby/buildkit [Service] ExecStart=/usr/local/buildkit/bin/buildkitd --config=/etc/buildkit/buildkitd.toml --oci-worker=false --containerd-worker=true [Install] WantedBy=multi-user.target

我刚刚拿了一台虚拟机测试了一下,应该是生效的,你自己看看你的配置是不是哪里写错了

# 安装
wget https://ghproxy.cn/github.com/moby/buildkit/releases/download/v0.18.2/buildkit-v0.18.2.linux-amd64.tar.gz
mkdir /usr/local/buildkit
tar -xf buildkit-v0.18.2.linux-amd64.tar.gz -C /usr/local/buildkit
echo 'export PATH=/usr/local/buildkit/bin:$PATH' >> /etc/profile
source /etc/profile

# 配置
mkdir /etc/buildkit
cat > /etc/buildkit/buildkitd.toml <<EOF
[registry."10.1.192.92:30990"]
http = true
insecure = true
EOF


cat <<EOF > /usr/lib/systemd/system/buildkitd.service
[Unit]
Description=buildkitd
After=network.target

[Service]
ExecStart=/usr/local/buildkit/bin/buildkitd  --config=/etc/buildkit/buildkitd.toml

[Install]
WantedBy=multi-user.target
EOF

# 重新加载Unit file
systemctl daemon-reload
# 启动服务
systemctl start buildkitd
# 开机自启动
systemctl enable buildkitd

# 测试
cat > ./Dockerfile <<EOF
FROM 10.1.192.92:30990/library/alpine:3.16
EOF

buildctl build \
--frontend=dockerfile.v0 \
--local context=. \
--local dockerfile=. \
--output type=image,name=test:latest

日志:

[root@node buildkit]# buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=. --output type=image,name=test:latest
[+] Building 0.5s (5/5) FINISHED                                                                                                                                                                        
 => [internal] load build definition from Dockerfile                                                                                                                                               0.0s
 => => transferring dockerfile: 80B                                                                                                                                                                0.0s
 => [internal] load metadata for 10.1.192.92:30990/library/alpine:3.16                                                                                                                             0.4s
 => [internal] load .dockerignore                                                                                                                                                                  0.0s
 => => transferring context: 2B                                                                                                                                                                    0.0s
 => [1/1] FROM 10.1.192.92:30990/library/alpine:3.16@sha256:d4817104439dd042cc33e813eb059dadd791676c040a03c3db21dd79e9af22b2                                                                       0.0s
 => => resolve 10.1.192.92:30990/library/alpine:3.16@sha256:d4817104439dd042cc33e813eb059dadd791676c040a03c3db21dd79e9af22b2                                                                       0.0s
 => exporting to image                                                                                                                                                                             0.0s
 => => exporting layers                                                                                                                                                                            0.0s
 => => exporting manifest sha256:38122242c0798c65e079ae3b7fbac4663ee5b6c17a152d6f6b105b995f9036dd                                                                                                  0.0s
 => => exporting config sha256:eed2584f8d0145e7ac6ea52de9de7d3001a16c89b1aba7575d6a55b30f8075ba                                                                                                    0.0s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants