[CI] - bug, web server not binding to address

[JoshuaSBrown]

Description

Observing odd behavior, where the web server in the CI is not binding to the port.

Context

I'm thinking this could be a docker layer issue in the images, but am not sure. I have seen that error before. In an unrelated PR here https://code.ornl.gov/dlsw/datafed/datafed/-/pipelines/669166, where the repo service seemingly just dies.

Found the problem. The repo service is not able to download the certs from the web server. All requests are being denied

datafed-core-key.pub not found, downloading from the core server
https://ci-datafed.ornl.gov/datafed-core-key.pub
Resolving ci-datafed.ornl.gov (ci-datafed.ornl.gov)... 10.64.194.16
Connecting to ci-datafed.ornl.gov (ci-datafed.ornl.gov)|10.64.194.16|:443... failed: Connection refused. 

docker shows the web server is running but it is unreachable from the browser.

Output from the web server doesn't give us anything useful.

2025-02-11T04:53:25.938Z datafed-ws INFO datafed-ws.js:loadSettings:2315 { "thread_id": 0, "message": "Reading configuration from file: /datafed/source/config/datafed-ws.cfg"  }
2025-02-11T04:53:25.950Z datafed-ws INFO datafed-ws.js:protobuf.load:2199 { "thread_id": 0, "message": "Running Version: 2024.6.17.10.40"  }
2025-02-11T04:53:25.975Z datafed-ws INFO datafed-ws.js:startServer:156 { "thread_id": 0, "message": "Host: ci-datafed.ornl.gov"  }
2025-02-11T04:53:25.975Z datafed-ws INFO datafed-ws.js:startServer:157 { "thread_id": 0, "message": "Port: 443"  }
2025-02-11T04:53:25.975Z datafed-ws INFO datafed-ws.js:startServer:159 { "thread_id": 0, "message": "Yes"  }
2025-02-11T04:53:25.976Z datafed-ws INFO datafed-ws.js:startServer:177 { "thread_id": 0, "message": "External URL: https://ci-datafed.ornl.gov"  }
-
2025-02-11T04:53:25.976Z datafed-ws INFO datafed-ws.js:startServer:179 { "thread_id": 0, "message": "Test mode: false"  } 

Getting a connection refused error when trying to grab the website from localhost:

wget https://localhost/
--2025-02-11 05:12:38-- https://localhost/
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:443... failed: Connection refused.
Connecting to localhost (localhost)|127.0.0.1|:443... failed: Connection refused.

web cert keys don't expire until much later verified by running.

/shared/keys$ sudo openssl x509 -enddate -noout -in cert.crt

At least on the host vm.
 
Running netstat on the container shows that, node is not listening on port 443. 

/opt/datafed/keys# netstat -tlupn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

Weird because it is running.

1 0.0 0.0 8472 7436 pts/0 Ss+ 04:53 0:00 /bin/bash /datafed/source/web/docker/entrypoint.sh /datafed/source/web/datafed-ws.js /datafed/source/config/datafed-ws.cfg
768 0.0 0.0 4504 2824 pts/0 S+ 04:53 0:00 su datafed -c "$@" -- argv0 /datafed/source/web/datafed-ws.js /datafed/source/config/datafed-ws.cfg
769 0.0 0.0 4504 2624 pts/0 S+ 04:53 0:00 su datafed -c tee /datafed/logs/datafed-ws.log
770 0.2 0.6 11207064 53720 ? Ssl 04:53 0:05 node /datafed/source/web/datafed-ws.js /datafed/source/config/datafed-ws.cfg
771 0.0 0.0 2524 508 ? Ss 04:53 0:00 tee /datafed/logs/datafed-ws.log
801 0.0 0.0 4248 3572 pts/1 Ss 05:27 0:00 /bin/bash
1077 0.0 0.0 5900 2924 pts/1 R+ 05:32 0:00 ps aux

Running a dummy service with the same certs in the container works.

42910fad628d camden.ornl.gov/datafed/ws-1277-bug-proxy-basic:baf797ebcfac0316053680a8f8c9d4f0999f368b "/datafed/source/web…" 52 minutes ago Up 52 minutes 0.0.0.0:443->443/tcp, :::443->443/tcp ws-1277-bug-proxy-basic-baf797eb-3HTkFADI6v

Running a simple web server on the same container with the same certs, we are able to bind to the address correctly. 

Dummy application.

const https = require("https");
const fs = require("fs");

// Load SSL certificate and key
const options = {
key: fs.readFileSync("cert.key"), // Path to your private key
cert: fs.readFileSync("cert.crt") // Path to your certificate
};

// Create an HTTPS server
https.createServer(options, (req, res) => {
res.writeHead(200, { "Content-Type": "text/plain" });
res.end("Hello, this is a secure Node.js server on port 443!\n");
}).listen(443, "0.0.0.0", () => {
console.log("Server is running on https://localhost:443");
});

[JoshuaSBrown]

Looks like web server is waiting for core service. To respond.

TODO: Need to create a log message to indicate that the web server is hung waiting on the core service to start.