Non-deterministic database initialization caused by overlapping retry logic and async lifecycle handling

[johnny603]

Description

During local Docker startup, the application produces repeated PromiseRejectionHandledWarning messages and exhibits non-deterministic database initialization behavior. The issue appears to stem from overlapping retry logic between the bash installation script and Node.js database initialization code, combined with improper async lifecycle termination.

This results in noisy logs, repeated DB creation attempts, and unclear startup state during docker compose up.

Reproduction Steps

Clone the repository
Ensure no existing containers or volumes:

docker compose down -v
docker system prune -af --volumes

Rebuild and start the stack:

docker compose build --no-cache
docker compose up

Observe startup logs during web service initialization

Observed Behavior

During startup, the following occurs:

1. Node warning spam
   (node:23) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: XXXXX)
   (node:23) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: XXXXX)
   (node:23) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: XXXXX)
2. Schema initialization retry behavior
   npm run createdb is invoked repeatedly from installOED.sh
   Node-side DB initialization (createDB.js) also performs async schema setup
   These two mechanisms overlap and are not coordinated

Expected Behavior

Database initialization should occur exactly once per startup
No repeated retries when DB init is already in progress
No PromiseRejectionHandledWarning logs
Deterministic startup lifecycle:
DB ready → schema created → web server starts

Root Cause Analysis (based on investigation)

1. Overlapping retry systems
   Bash script retries:

installOED.sh → npm run createdb (loop with retries)
Node script performs async DB initialization internally

Result: duplicate execution attempts and race conditions

2. Improper async lifecycle termination in Node

Current pattern in createDB.js:

(async function createSchemaWrapper() {
try {
await createSchema(conn);
await insertStandardUnits(conn);
await insertStandardConversions(conn);
await redoCik(conn);
process.exitCode = 0;
} catch (err) {
process.exitCode = 1;
}
}());

Issues:

Uses IIFE without explicit process termination
Relies on process.exitCode instead of deterministic process.exit()
Allows event loop to continue during Docker retry cycles
3. Delayed Promise rejection handling
Errors are not always handled immediately

Leads to Node emitting:

PromiseRejectionHandledWarning
Likely caused by asynchronous retry overlap between bash + Node

Impact

Noisy and confusing startup logs
Unclear DB initialization state during development
Potential race conditions in schema creation
Reduced confidence in Docker startup reliability
Harder debugging due to repeated retries masking root errors

Suggested Fixes

1. Make DB initialization deterministic (Node side)
   Replace IIFE with explicit main function:
   async function main() {
   try {
   const conn = getConnection();
   await createSchema(conn);
   await insertStandardUnits(conn);
   await insertStandardConversions(conn);
   await redoCik(conn);

   process.exit(0);
   } catch (err) {
   console.error(err);
   process.exit(1);
   }
   }

main().catch(err => {
console.error("Fatal error:", err);
process.exit(1);
});
2. Remove overlapping retry logic (choose one system)
Either:
Keep bash retry loop
OR move retry logic into Node
Avoid dual-layer retries
3. Fix bash piping (error propagation)

Replace:

npm run createdb |& tee /tmp/oed.error > /dev/null

With:

set -o pipefail
npm run createdb 2>&1 | tee /tmp/oed.error
createdb_code=${PIPESTATUS[0]}

4. Add global async safety net (optional but recommended)
   process.on("unhandledRejection", (err) => {
   console.error("UNHANDLED REJECTION:", err);
   process.exit(1);
   });

Notes

Issue is reproducible in clean Docker environment with no existing volumes
Database itself is not corrupted; issue is strictly initialization orchestration + async handling
Removing volumes does not resolve behavior