When I supply --set mongodb.version="7.0" the mongo pods achieve a Ready state but the underlying application is clearly not healthy and Graylog and Data Node pods timeout when connecting to them.
However, specifying version 7.0.32 (7.0.latest) works fine.
Here are the mongo pod logs when using 7.0:
│ mongodb-agent [2026-05-07T20:02:34.308+0000] [.debug] [util/distros/distros.go:LinuxFlavorAndVersionUncached:140] Detected linux flavor rhel version 9.6 │
│ mongod 2026-05-07T20:02:34.057Z INFO versionhook/main.go:33 Running version change post-start hook │
│ mongod 2026-05-07T20:02:34.058Z INFO versionhook/main.go:40 Waiting for agent health status... │
│ mongod 2026-05-07T20:02:35.058Z INFO versionhook/main.go:46 Agent health status file not found, mongod will start │
│ mongod Waiting for config and keyfile files to be created by the agent... │
│ mongod Waiting... │
│ mongod Waiting... │
│ mongod Waiting... │
│ mongod Waiting... │
│ mongod Waiting...
Here's the describe:
Name: graylog-mongo-rs-0
Namespace: graylog
Priority: 0
Service Account: graylog-mongo-sa
Node: ip-172-31-21-46.ec2.internal/172.31.21.46
Start Time: Thu, 07 May 2026 16:02:18 -0400
Labels: app=graylog-mongo-rs-svc
apps.kubernetes.io/pod-index=0
controller-revision-hash=graylog-mongo-rs-64f869fb7f
statefulset.kubernetes.io/pod-name=graylog-mongo-rs-0
topology.kubernetes.io/region=us-east-1
topology.kubernetes.io/zone=us-east-1a
Annotations: agent.mongodb.com/version: 1
Status: Running
IP: 172.31.22.214
IPs:
IP: 172.31.22.214
Controlled By: StatefulSet/graylog-mongo-rs
Init Containers:
mongod-posthook:
Container ID: containerd://f4ce55dece34e5e60027c63cac5bc6aef5dfbe98f2f127183b4b3f2538d155a3
Image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.10
Image ID: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:f321ec1d25d6e98805b8be9321f2a926d702835136dde88d5fffe917c2df1d0a
Port: <none>
Host Port: <none>
Command:
cp
version-upgrade-hook
/hooks/version-upgrade
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 07 May 2026 16:02:32 -0400
Finished: Thu, 07 May 2026 16:02:32 -0400
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 500M
Requests:
cpu: 500m
memory: 400M
Environment: <none>
Mounts:
/hooks from hooks (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rspwn (ro)
mongodb-agent-readinessprobe:
Container ID: containerd://fbda4639ae2b64313313cbec4020ff6eac6e1c5b6cf8a697199b42543cfec18b
Image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.23
Image ID: quay.io/mongodb/mongodb-kubernetes-readinessprobe@sha256:436fc328f3887f022a4760afd03da1a7091d285baf3d627a17d80bbdaab0ee47
Port: <none>
Host Port: <none>
Command:
cp
/probes/readinessprobe
/opt/scripts/readinessprobe
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 07 May 2026 16:02:33 -0400
Finished: Thu, 07 May 2026 16:02:33 -0400
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 500M
Requests:
cpu: 500m
memory: 400M
Environment: <none>
Mounts:
/opt/scripts from agent-scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rspwn (ro)
Containers:
mongod:
Container ID: containerd://bb7efd65c7d83d73de93f101e48a726e0f80873f770d48d1db2483a818572bc2
Image: quay.io/mongodb/mongodb-community-server:7.0-ubi8
Image ID: quay.io/mongodb/mongodb-community-server@sha256:48c57f0883d4b8f358aabe5f02a03dd1205d178c37c861e39267eda422d100e8
Port: <none>
Host Port: <none>
Command:
/bin/sh
-c
if [ -e "/hooks/version-upgrade" ]; then
#run post-start hook to handle version changes (if exists)
/hooks/version-upgrade
fi
# wait for config and keyfile to be created by the agent
echo "Waiting for config and keyfile files to be created by the agent..."
while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do
sleep 3;
echo "Waiting..."
done
# sleep is important after agent issues shutdown command
# k8s restarts the mongod container too quickly for the agent to realize mongod is down
echo "Sleeping for 15s..."
sleep 15
# start mongod with this configuration
echo "Starting mongod..."
exec mongod -f /data/automation-mongod.conf
Args:
State: Running
Started: Thu, 07 May 2026 16:02:34 -0400
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 500M
Requests:
cpu: 500m
memory: 400M
Environment:
AGENT_STATUS_FILEPATH: /healthstatus/agent-health-status.json
Mounts:
/data from data-volume (rw)
/healthstatus from healthstatus (rw)
/hooks from hooks (rw)
/tmp from tmp (rw)
/var/lib/mongodb-mms-automation/authentication from graylog-mongo-rs-keyfile (rw)
/var/log/mongodb-mms-automation from logs-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rspwn (ro)
mongodb-agent:
Container ID: containerd://cdce1bbdc9c9cb8279f451c2eacf800b89f2afd6e1f6960eda831fb7ddac0bba
Image: quay.io/mongodb/mongodb-agent:108.0.2.8729-1
Image ID: quay.io/mongodb/mongodb-agent@sha256:b422829aac63f068aed4a6948fd14cbd3c3bfad6d47ab76523f6839bf20b0203
Port: <none>
Host Port: <none>
Command:
/bin/bash
-c
current_uid=$(id -u)
declare -r current_uid
if ! grep -q "${current_uid}" /etc/passwd ; then
sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
export NSS_WRAPPER_PASSWD=/tmp/passwd
export LD_PRELOAD=libnss_wrapper.so
export NSS_WRAPPER_GROUP=/etc/group
fi
agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile /var/log/mongodb-mms-automation/automation-agent.log -logLevel INFO -maxLogFileDurationHrs 24
State: Running
Started: Thu, 07 May 2026 16:02:34 -0400
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 500M
Requests:
cpu: 500m
memory: 400M
Readiness: exec [/opt/scripts/readinessprobe] delay=5s timeout=1s period=10s #success=1 #failure=40
Environment:
AGENT_STATUS_FILEPATH: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
AUTOMATION_CONFIG_MAP: graylog-mongo-rs-config
HEADLESS_AGENT: true
POD_NAMESPACE: graylog (v1:metadata.namespace)
Mounts:
/data from data-volume (rw)
/opt/scripts from agent-scripts (rw)
/tmp from tmp (rw)
/var/lib/automation/config from automation-config (ro)
/var/lib/mongodb-mms-automation/authentication from graylog-mongo-rs-keyfile (rw)
/var/log/mongodb-mms-automation from logs-volume (rw)
/var/log/mongodb-mms-automation/healthstatus from healthstatus (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rspwn (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
data-volume:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-volume-graylog-mongo-rs-0
ReadOnly: false
logs-volume:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: logs-volume-graylog-mongo-rs-0
ReadOnly: false
agent-scripts:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
automation-config:
Type: Secret (a volume populated by a Secret)
SecretName: graylog-mongo-rs-config
Optional: false
graylog-mongo-rs-keyfile:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
healthstatus:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
hooks:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-rspwn:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
Optional: false
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m12s default-scheduler Successfully assigned graylog/graylog-mongo-rs-0 to ip-172-31-21-46.ec2.internal
Normal SuccessfulAttachVolume 3m1s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-3841c815-8c23-461f-9932-b805a66dcdfc"
Normal SuccessfulAttachVolume 2m59s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-13c2f262-6a93-4116-b7f5-dd8395cc7c1d"
Normal Pulling 2m58s kubelet Pulling image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.10"
Normal Pulled 2m58s kubelet Successfully pulled image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.10" in 78ms (78ms including waiting). Image size: 59082744 bytes.
Normal Created 2m58s kubelet Container created
Normal Started 2m58s kubelet Container started
Normal Pulling 2m58s kubelet Pulling image "quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.23"
Normal Pulled 2m58s kubelet Successfully pulled image "quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.23" in 70ms (70ms including waiting). Image size: 60690112 bytes.
Normal Created 2m58s kubelet Container created
Normal Started 2m57s kubelet Container started
Normal Pulled 2m57s kubelet Container image "quay.io/mongodb/mongodb-community-server:7.0-ubi8" already present on machine and can be accessed by the pod
Normal Created 2m57s kubelet Container created
Normal Started 2m56s kubelet Container started
Normal Pulling 2m56s kubelet Pulling image "quay.io/mongodb/mongodb-agent:108.0.2.8729-1"
Normal Pulled 2m56s kubelet Successfully pulled image "quay.io/mongodb/mongodb-agent:108.0.2.8729-1" in 81ms (81ms including waiting). Image size: 182416151 bytes.
Normal Created 2m56s kubelet Container created
Normal Started 2m56s kubelet Container started
Looks like 7.0 is a valid version value, and I can confirm 7.0.32 is running in the container:
bash-4.4$ mongod --version
db version v7.0.32
Build Info: {
"version": "7.0.32",
"gitVersion": "40a2e0f542b0a68f0ed51fa216e9d38b9917fe14",
"openSSLVersion": "OpenSSL 1.1.1k FIPS 25 Mar 2021",
"modules": [],
"allocator": "tcmalloc",
"environment": {
"distmod": "rhel80",
"distarch": "x86_64",
"target_arch": "x86_64"
}
}
But I can't connect to the mongo shell like I can when installing with 7.0.32:
bash-4.4$ mongosh
Warning: Could not access file: EACCES: permission denied, mkdir '/data/db/.mongodb'
Current Mongosh Log ID: 69fcf29d8d38bb04b98ce5af
Connecting to: mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+2.6.0
MongoNetworkError: connect ECONNREFUSED 127.0.0.1:27017
When I supply
--set mongodb.version="7.0"the mongo pods achieve a Ready state but the underlying application is clearly not healthy and Graylog and Data Node pods timeout when connecting to them.However, specifying version
7.0.32(7.0.latest) works fine.Here are the mongo pod logs when using
7.0:Here's the
describe:Looks like
7.0is a valid version value, and I can confirm7.0.32is running in the container:But I can't connect to the mongo shell like I can when installing with
7.0.32: