Dynamic Load Balancer — Lab 6

This project extends a secure persistent group chat across three backend systems behind one performance-based dynamic load balancer. It provides the required HTTP API, shared encrypted PostgreSQL persistence, duplicate protection, health-aware routing, automatic failover, and cluster-wide live WebSocket broadcasting.

The implementation history is intentionally preserved in two supported branches:

Branch	Purpose
`main`	Complete final application: dynamic HTTP load balancing plus distributed secure WebSocket broadcast
`leaderboard-session`	Exact pre-distributed-WSS HTTP implementation at commit `87dca66`; useful for reproducing the scored leaderboard architecture

The scored leaderboard evaluated only POST /message and GET /feed. The distributed WSS event bus was added afterward and tested separately. This is documented explicitly rather than presenting the later WSS work as part of the official score.

Assignment status

The implementation, deployment path, custom generator, figures and final report are complete.

Lab 6 requirement	Status
Preserve secure WebSocket chat	Implemented on `main`; cross-backend delivery tested independently
`POST /message` and `GET /feed`	Implemented
Unique message IDs and duplicate protection	Implemented
Shared persistent database design	Implemented and hosted PostgreSQL path tested
Three backend processes behind one load balancer	Hosted path tested
Health-aware dynamic load balancing	Implemented
Threshold configuration	Implemented
Experimentally optimize the threshold	Current mixed-load candidate uses threshold 60; earlier write-only tests selected 75
Variable-user/message/interval load generator	Implemented
Response-time and Sys1–Sys4 utilization plots	Complete
Report	Complete

Early write-only comparisons selected threshold 75. The later evaluator-like mixed message/feed profile selected threshold 60, so those results are not directly interchangeable. The best balanced complete official result was Test 18 (1.07% static error, 758.07 ms static mean, and 38,445/40,000 breakpoint successes). The final recorded assessment was 12 marks at mean rank 13. See the report for the complete progression and the limitations of each comparison.

Project components

backend/server.py — HTTP API, encrypted persistence, idempotency, statistics, and secure WebSocket chat.
backend/message_bus.py — opt-in PostgreSQL LISTEN/NOTIFY transport for cluster-wide live WebSocket messages.
load-balancer/main.go — dynamic reverse proxy, health checking, performance scoring, failover, and aggregated metrics.
load-balancer/wss_proxy.go — optional TLS-pass-through WSS entry point with health-, memory-, and connection-aware backend selection.
load-generator/main.go — staged concurrent message workloads, variable message sizes/intervals, latency and routing measurements, feed verification, and CSV/JSON metric export.
scripts/integration-test.sh — starts three temporary backends and one load balancer and tests sharing, concurrent duplicates, WSS, and failure handling.
previous-codes-used/ — unchanged earlier-lab reference implementations.

Choose and run a branch

Clone once, then explicitly select the architecture to run:

git clone https://github.com/SandstromPL/CSD-Dynamic-Load-Balancing-and-Persistent-Chat.git DLB
cd DLB

# Complete final chat, including distributed WSS:
git switch main

# OR: exact HTTP leaderboard-session boundary:
# git switch leaderboard-session

Run the same branch on Sys1, Sys2, Sys3 and Sys4. Never combine a main load balancer with leaderboard-session backends or the reverse.

Shared backend environment

Install the Python dependencies and copy the same AES key to all three backend systems. PostgreSQL runs on Sys3 at local port 4000; Sys2 and Sys4 reach its mapped port 4279.

cd ~/DLB
python3 -m pip install -r backend/requirements.txt
mkdir -p ~/DLB/data
chmod 700 ~/DLB/data

Set these values in each backend's shell/tmux session:

cd ~/DLB/backend

export HTTP_PORT=6000
export WS_PORT=5000
export MESSAGE_ENCRYPTION_KEY_FILE=/home/student/DLB/data/encryption.key

export DB_POOL_MIN=1
export DB_POOL_MAX=6
export DB_HEALTH_TIMEOUT=1
export HEALTH_CHECK_TIMEOUT=1.5
export DB_WRITE_BATCH_SIZE=64
export DB_WRITE_BATCH_DELAY_MS=2
export DB_WRITE_BATCH_WORKERS=1
export DB_WRITE_QUEUE_MAX=256
export DB_WRITE_QUEUE_TIMEOUT=5
export FEED_CACHE_CHUNK_BYTES=65536
export HTTP_SOCKET_READ_BUFFER_BYTES=32768
export HTTP_SOCKET_WRITE_BUFFER_BYTES=32768

Add the system-specific identity and database URL:

# Sys2
export BACKEND_ID=sys2
export DATABASE_URL='postgresql://chat_user:URL_ENCODED_PASSWORD@10.1.75.53:4279/chat?sslmode=require'

# Sys3 (use this pair instead in the Sys3 session)
export BACKEND_ID=sys3
export DATABASE_URL='postgresql://chat_user:URL_ENCODED_PASSWORD@127.0.0.1:4000/chat?sslmode=require'

# Sys4 (use this pair instead in the Sys4 session)
export BACKEND_ID=sys4
export DATABASE_URL='postgresql://chat_user:URL_ENCODED_PASSWORD@10.1.75.53:4279/chat?sslmode=require'

Do not create three encryption keys. Every backend must receive the identical key file or it will be unable to decrypt rows written by another backend.

Run `leaderboard-session`

This branch reproduces the HTTP architecture before distributed WSS was added. For an HTTP-only evaluator reproduction, start one backend on each of Sys2, Sys3 and Sys4 with:

cd ~/DLB
git switch leaderboard-session
cd backend

export ENABLE_WEBSOCKET=0
python3 server.py

Build and start the DLB on Sys1. The hosted Go installation is 1.22, so GOTOOLCHAIN=local prevents an unavailable toolchain download:

cd ~/DLB
git switch leaderboard-session
cd load-balancer

GOTOOLCHAIN=local go test ./...
GOTOOLCHAIN=local go build -buildvcs=false -o dlb .

GOMEMLIMIT=192MiB GOGC=100 ./dlb \
  -listen :5000 \
  -backends http://10.1.75.53:6278,http://10.1.75.53:6279,http://10.1.75.53:6280 \
  -threshold 60 \
  -max-memory 90 \
  -max-total-memory 98 \
  -lb-max-memory 82 \
  -lb-max-socket-bytes 167772160 \
  -max-inflight 100 \
  -admission-limit 0 \
  -backend-cap 200 \
  -backend-queue-timeout 10s \
  -latency-target 100ms \
  -stats-interval 500ms \
  -health-interval 1s \
  -monitor-timeout 2500ms \
  -backend-timeout 15s \
  -failure-limit 2 \
  -proxy-max-idle 256 \
  -proxy-max-idle-per-host 64 \
  -proxy-max-conns-per-host 200 \
  -proxy-max-feed-conns-per-host 4 \
  -proxy-idle-timeout 10s \
  -client-idle-timeout 5s \
  -client-max-connections 2600 \
  -client-connection-burst 0 \
  -client-new-timeout 2s \
  -socket-read-buffer-bytes 16384 \
  -socket-write-buffer-bytes 16384 \
  -proxy-feed-read-buffer-bytes 131072 \
  -client-feed-write-buffer-bytes 131072 \
  -proxy-buffer-bytes 32768 \
  -proxy-feed-buffer-bytes 32768

The public HTTP endpoint is http://10.1.75.53:5277. The flat 2,600 connection cap is the post-Test-23 repair validated locally before the deadline but not submitted for another official score; the report distinguishes it from the actual Test-23 configuration.

Run `main` with distributed WSS

On every backend, select main, retain the common/system-specific values above, and additionally enable the TLS listener and cluster event bus:

cd ~/DLB
git switch main
cd backend

export ENABLE_WEBSOCKET=1
export ENABLE_CLUSTER_WEBSOCKET=1
export WS_EVENT_CHANNEL=chat_message_events
export WS_EVENT_QUEUE_MAX=4096
export TLS_CERT=/home/student/chat-ssl/cert.pem
export TLS_KEY=/home/student/chat-ssl/key.pem

python3 server.py

Build the main DLB on Sys1 using the same HTTP command above, but add the three WSS flags before starting it:

cd ~/DLB
git switch main
cd load-balancer

GOTOOLCHAIN=local go test ./...
GOTOOLCHAIN=local go build -buildvcs=false -o dlb .

GOMEMLIMIT=192MiB GOGC=100 ./dlb \
  -listen :5000 \
  -backends http://10.1.75.53:6278,http://10.1.75.53:6279,http://10.1.75.53:6280 \
  -wss-listen :7000 \
  -wss-backends 10.1.75.53:5278,10.1.75.53:5279,10.1.75.53:5280 \
  -wss-max-connections 512 \
  -threshold 60 \
  -max-memory 90 \
  -max-total-memory 98 \
  -lb-max-memory 82 \
  -lb-max-socket-bytes 167772160 \
  -max-inflight 100 \
  -admission-limit 0 \
  -backend-cap 200 \
  -backend-queue-timeout 10s \
  -latency-target 100ms \
  -stats-interval 500ms \
  -health-interval 1s \
  -monitor-timeout 2500ms \
  -backend-timeout 15s \
  -failure-limit 2 \
  -proxy-max-idle 256 \
  -proxy-max-idle-per-host 64 \
  -proxy-max-conns-per-host 200 \
  -proxy-max-feed-conns-per-host 4 \
  -proxy-idle-timeout 10s \
  -client-idle-timeout 5s \
  -client-max-connections 2600 \
  -client-connection-burst 0 \
  -client-new-timeout 2s \
  -socket-read-buffer-bytes 16384 \
  -socket-write-buffer-bytes 16384 \
  -proxy-feed-read-buffer-bytes 131072 \
  -client-feed-write-buffer-bytes 131072 \
  -proxy-buffer-bytes 32768 \
  -proxy-feed-buffer-bytes 32768

The client entry points are:

HTTP: http://10.1.75.53:5277
WSS: wss://10.1.75.53:7277

The WSS connection remains sticky to one selected backend. After a successful HTTP or WSS commit, PostgreSQL LISTEN/NOTIFY distributes batched message IDs; each backend fetches those committed rows once per batch and broadcasts only to its local connected clients. New clients load old history explicitly rather than receiving the entire history automatically.

Verify either deployment

curl -sS http://10.1.75.53:5277/lb/health | python3 -m json.tool
curl -sS http://10.1.75.53:5277/lb/status | python3 -m json.tool

curl -sS -H 'Content-Type: application/json' \
  -d '{"client-name":"Alice","msg":"hello","message-id":"readme-check-1"}' \
  http://10.1.75.53:5277/message | python3 -m json.tool

curl -sS http://10.1.75.53:5277/feed | python3 -m json.tool

On main, also verify the single WSS entry point:

python3 scripts/wss-cluster-test.py \
  --url wss://10.1.75.53:7277 \
  --insecure

For a same-machine development check on whichever branch is currently selected:

bash scripts/integration-test.sh

Local four-terminal test

For this same-machine test, all three processes use data/chat.db and data/encryption.key. SQLite WAL supports these local processes, but it must not be placed on a network filesystem for deployment. The final three-system setup must use PostgreSQL.

Run every command from the project root unless the command says otherwise.

Terminal 1 — Sys2 simulation

BACKEND_ID=sys2 HTTP_PORT=6001 WS_PORT=5001 ENABLE_WEBSOCKET=0 \
  python3 backend/server.py

Terminal 2 — Sys3 simulation

BACKEND_ID=sys3 HTTP_PORT=6002 WS_PORT=5002 ENABLE_WEBSOCKET=0 \
  python3 backend/server.py

Terminal 3 — Sys4 simulation

BACKEND_ID=sys4 HTTP_PORT=6003 WS_PORT=5003 ENABLE_WEBSOCKET=0 \
  python3 backend/server.py

Terminal 4 — dynamic load balancer

cd load-balancer
go run -buildvcs=false . \
  -listen :8080 \
  -backends http://127.0.0.1:6001,http://127.0.0.1:6002,http://127.0.0.1:6003 \
  -threshold 75

Wait until the balancer prints UP for all three backends. Call the public API through port 8080, not the backend ports.

Public API reference

The professor's load generator should only require the load-balancer URL and the two mandatory routes.

`POST /message`

Stores one message. client-name and msg are required. message-id is optional but strongly recommended for retry safety.

JSON request:

curl -i \
  -H 'Content-Type: application/json' \
  -d '{"client-name":"Alice","msg":"Hello","message-id":"alice-0001"}' \
  http://127.0.0.1:8080/message

Successful first insertion returns HTTP 201:

{
  "status": "created",
  "duplicate": false,
  "message": {
    "message-id": "alice-0001",
    "client-name": "Alice",
    "msg": "Hello",
    "timestamp": "2026-09-10T14:00:00+00:00",
    "sequence": 1,
    "source": "http"
  }
}

The API also accepts URL-encoded form data:

curl -i \
  --data-urlencode 'client-name=Alice' \
  --data-urlencode 'msg=Message sent as form data' \
  --data-urlencode 'message-id=alice-0002' \
  http://127.0.0.1:8080/message

For compatibility with a simple evaluator, GET /message with query parameters is also supported:

curl -i -G http://127.0.0.1:8080/message \
  --data-urlencode 'client-name=Alice' \
  --data-urlencode 'msg=Message sent as query data' \
  --data-urlencode 'message-id=alice-0003'

An idempotency header can provide the ID instead of putting it in the body:

curl -i \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: alice-0004' \
  -d '{"client-name":"Alice","msg":"Header ID example"}' \
  http://127.0.0.1:8080/message

If no ID is supplied, the backend generates a UUID. A client cannot safely retry an uncertain request with the same identity unless it supplied and retained the original ID.

Duplicate behavior:

Same ID and same content: HTTP 200, status: duplicate; no new row.
Same ID but different client or content: HTTP 409; the original row remains.
The database unique constraint is authoritative, including concurrent requests routed to different backends.

Inspect load-balancing response headers:

curl -sS -D - -o /dev/null \
  -H 'Content-Type: application/json' \
  -d '{"client-name":"Alice","msg":"headers","message-id":"headers-1"}' \
  http://127.0.0.1:8080/message

Relevant headers are X-Load-Balancer-Backend, X-Load-Balancer-Score, X-Backend, and, when necessary, X-Load-Balancer-Fallback.

`GET /feed`

Returns every committed message in stable database sequence order:

curl -sS http://127.0.0.1:8080/feed | python3 -m json.tool

Response structure:

{
  "messages": [
    {
      "message-id": "alice-0001",
      "client-name": "Alice",
      "msg": "Hello",
      "timestamp": "2026-09-10T14:00:00+00:00",
      "sequence": 1,
      "source": "http"
    }
  ],
  "count": 1
}

The message is decrypted by the backend for the API response. In the database, msg is stored as AES-GCM ciphertext with a nonce and authenticated message ID.

Load-balancer monitoring API

These routes belong to the load balancer and are not forwarded.

`GET /lb/health`

Returns HTTP 200 when at least one backend is healthy, otherwise HTTP 503.

curl -i http://127.0.0.1:8080/lb/health

`GET /lb/status`

Shows the routing configuration and each backend's health, score components, in-flight requests, request counters, last error, and metric freshness.

curl -sS http://127.0.0.1:8080/lb/status | python3 -m json.tool

This is the best endpoint for watching a backend become overloaded, go down, or recover.

`GET /lb/metrics`

Returns total/success/failed requests, backend errors, fallback count, throughput, p50/p95/p99 latency, Sys1 CPU/memory/load, connection-cap reclaims and rejections, per-backend statistics, and /feed phase timing. Feed phase metrics separate backend selection, the wait for backend response headers, and body delivery to the client; header failures and interrupted bodies are counted independently.

curl -sS http://127.0.0.1:8080/lb/metrics | python3 -m json.tool

The percentile window is bounded to the latest 20,000 samples so monitoring does not grow memory without limit.

Custom load generator

Run the generator from a separate client machine when possible. Running it on Sys1 consumes CPU and changes the Sys1 utilization being measured.

Build it once:

cd load-generator
go test -buildvcs=false ./...
go build -buildvcs=false -o loadgen .

Start with a small hosted smoke test:

./loadgen \
  -url http://10.1.75.53:5277 \
  -experiment smoke-threshold-75 \
  -users 20 \
  -requests 200 \
  -message-min 16 \
  -message-max 512 \
  -interval-min 0ms \
  -interval-max 5ms \
  -metrics-interval 1s \
  -out-dir results

A capture-calibrated paired leaderboard run is:

ulimit -n 10000
./loadgen \
  -url http://10.1.75.53:5277 \
  -experiment leaderboard-paired-v2 \
  -profile leaderboard-paired-v2 \
  -message-min 16 \
  -message-max 512 \
  -interval-min 0ms \
  -interval-max 0ms \
  -seed 20260911 \
  -metrics-interval 1s \
  -out-dir results

leaderboard-paired-v2 runs the static and breakpoint boards sequentially against one clean database. This matches the lifecycle recovered from the test-19 feed: static rows remain stored when breakpoint begins. Reset the messages table once before the paired run, not between its boards. Static uses the recorded 250, 500, 750, and 1,000-user ladder. Breakpoint uses the recorded 200, 350, 500, 750, 1,000, 1,500, 2,000, and 2,500-user ladder and stops after a stage exceeds 20% errors. Every stage has a 5,000-operation ceiling.

The v2 payload model is calibrated to the captured evaluator feed. It uses a continuous deterministic operation stream within each board, UUID-shaped message IDs, evaluator-shaped client names, printable 16--512 byte payloads, and the observed message marker/timestamp envelope. At the default 5% feed mix, a complete clean run should store approximately 19,000 static messages and 38,000 breakpoint messages. The report records an after-static feed-size and integrity checkpoint as well as the final combined feed verification.

The older leaderboard-suite profile remains available for comparing historical local results, but it uses the legacy alphanumeric payload model. The profiles use a deterministic approximation of the evaluator's operation pool: approximately 5% GET /feed and 95% POST /message. Static gets a unique message pool per stage. Breakpoint repeats the same deterministic operation and message-content sequence while assigning a distinct message-ID namespace to each stage. A complete eight-stage run therefore inserts approximately 38,000 messages, matching the database growth observed in leaderboard test 16.

Use -profile duplicate-breakpoint when deliberately testing retries and idempotency with the old behavior: it reuses the same message IDs at every stage and grows the database by only approximately 4,750 messages. It is not the current official-score approximation.

The evaluator also stops slow stages at a safety deadline before their 5,000 request budget is exhausted. The local defaults are inferred as a 20-second static issue window and a 30-second breakpoint issue window. Already-issued requests are allowed to finish or reach their individual ten-second timeout. Override both with -stage-issue-timeout 25s when calibrating; the exact official deadline is not published.

For the closest current ranking comparison, use leaderboard-paired-v2 after a single clean reset. The separate leaderboard-static and leaderboard-breakpoint profiles remain useful for isolating one board, and the older stress workloads remain available as stress-static and stress-breakpoint. To exercise a deliberately harsher slow-reader case, add -response-read-delay 250ms.

For the assignment's variable-interval workload, use a nonzero range such as -interval-min 20ms -interval-max 500ms. A duration-based single stage is also supported, for example -users 100 -duration 2m.

Every run creates:

<experiment>-<timestamp>.json: configuration, per-stage and overall latency, throughput, errors, backend distribution, feed-integrity result, raw metrics, and separate message/feed completion diagnostics.
<experiment>-<timestamp>-metrics.csv: one row per sample for Sys1, Sys2, Sys3, and Sys4, including CPU, process RSS, total and working-set cgroup memory, reclaimable inactive-file memory, score, active requests, message/feed proxy-pool activity and peaks, DB queue/batch timing, encryption timing, feed refresh/serialization timing, and cache counters. Sys1 rows also include feed selection/header/body timing and completion counters from the load balancer.
<experiment>-<timestamp>-series.csv: official-style one-second successful RPS, error count, and mean successful latency for each stage.
summary.csv: appended stage/overall results for comparing thresholds.
operation-summary.csv: message and feed rows reported separately, including response headers received, complete bodies, transport errors, body-read errors, response bytes, throughput, and header/full-response latency.
header-acceptance.csv: a parallel phase view that counts successful 2xx response headers separately from complete response bodies. This helps compare evaluator timeout behavior without weakening the strict full-body result.

Leaderboard-profile JSON reports also include an automatic qualification verdict. A run is marked promotable only when every configured stage ran, no stage reached its issue deadline, all message requests succeeded, final feed verification passed completely, metrics collection remained available, and the LB reported no memory-guard activation, memory rejection, or connection rejection. Feed timeouts remain performance results rather than an automatic qualification failure. Always start qualification from fresh LB counters; a pre-existing guard or rejection counter deliberately rejects the run.

JSON and console summaries expose both definitions. successful, failed, error_percent, and successful_latency remain the strict complete-body metrics used by all historical local results. header_accepted, header_error_percent, and header_accepted_latency are diagnostic projections; they do not turn an interrupted feed into a completed local request.

The /feed checks verify every accepted message ID, client name, and body. They also validate the declared count, unique IDs, and increasing database sequence. Failed POSTs are reported separately as ambiguous and excluded from loss, matching the leaderboard. Custom and stress profiles retain the stricter check of every generated ID. Disable verification only for an intermediate stress test with -verify-feed=false.

Use the same workload and -seed for each threshold. Restart the load balancer at thresholds such as 55, 60, 65, 70, and 75, wait for all three backends to be healthy, then run the generator with a distinct experiment name. Do not choose an "optimal" threshold until summary.csv and the utilization CSVs have been compared for successful mean/p95/p99 latency, throughput, errors, fallback routes, and backend balance.

-connection-mode auto selects reuse for custom tests and fresh-user for leaderboard profiles. reuse uses one HTTP connection pool for the whole run. fresh-stage creates and retains one shared pool for every stage. fresh-user creates a persistent HTTP session for every concurrent virtual user and retains the sessions through stage transitions. That mode reproduces the connection-cap pressure visible in leaderboard test 14 much more closely. no-keepalive opens a new connection for every request and is useful as a worst-case churn test. Use -feed-percent to override the profile's 5% inferred feed mixture.

To test the separate HTTP StateNew case exposed by leaderboard test 15, open raw TCP sockets before a selected stage without sending request headers:

./load-generator \
  -url http://LOAD_BALANCER_IP:5000 \
  -profile leaderboard-breakpoint \
  -connection-mode fresh-user \
  -held-new-connections 500 \
  -held-new-stage 8 \
  -held-new-age 3s \
  -experiment stale-new-pressure

The sockets remain open for the selected stage and close before final feed verification. held-new-age should exceed the load balancer's client-new-timeout. Metric output records new/active/idle counts, oldest-new age, idle and stale-new reclamation, guarded-burst admissions, and hard listener rejections.

Backend-only API

These endpoints are intended for the load balancer, diagnostics, and the private network. Do not submit a backend URL to the leaderboard.

Route	Purpose
`GET /health`	Backend and database readiness; `503` if the database ping fails
`GET /internal/stats`	CPU, cgroup memory components, request load, DB-queue, encryption, and feed-cache timings
`GET /info`	Backend ID, host, ports, and storage mode
`GET /`	Basic service information
`OPTIONS /*`	CORS preflight support

Examples:

curl -sS http://127.0.0.1:6001/health | python3 -m json.tool
curl -sS http://127.0.0.1:6001/internal/stats | python3 -m json.tool
curl -sS http://127.0.0.1:6001/info | python3 -m json.tool

The public /message and /feed routes exist on every backend because the load balancer is a reverse proxy. External clients should still call them only through the load balancer.

How shared messages work

There is no delayed file copy or three-way synchronization between servers. PostgreSQL is the single source of truth:

Client
  │
  ▼
Load Balancer ──selects──► Sys2, Sys3, or Sys4
                              │
                              ▼
                    One shared PostgreSQL DB
                              ▲
                              │
                 every backend reads/writes here

sequence has a unique constraint, whose PostgreSQL index already serves ORDER BY sequence. Backend initialization removes the older redundant messages_sequence_idx, avoiding a second index update and extra WAL for every message without changing feed ordering or uniqueness.

For a message write:

The client should create and retain a stable message ID.
The load balancer selects a healthy, suitable backend.
That backend encrypts the message.
PostgreSQL requests arriving within the small batching window are combined into one atomic transaction with the same unique constraint.
Only after the database operation completes does it return success.
/feed on any backend refreshes from the same committed PostgreSQL rows.

Batching reduces transaction and network overhead without acknowledging data before it is durable. The first occurrence of a message ID receives created; duplicates receive the already committed row.

The normal unique-message path returns only the database-generated sequence, ID, and committed timestamp from INSERT ... RETURNING; the backend already has the other values and does not download them again. Only IDs that encounter a uniqueness conflict require a complete follow-up lookup, avoiding a second PostgreSQL round trip and redundant payload transfer for every normal batch. When queued writes are already available, workers drain them without creating a separate timeout task for each item; the batching timer is used only while the queue is empty.

A recovered backend does not download missed messages. It simply reconnects to PostgreSQL and can immediately read all committed history.

Each backend keeps a non-authoritative, incrementally built serialized feed cache. The first request reads, decrypts, and serializes the committed history; later requests ask PostgreSQL only for rows above the cached sequence and append those rows to immutable 64 KiB response chunks. Concurrent feed requests share one refresh. Each refresh also compares its cached count with the row count from the same database snapshot. PostgreSQL can commit sequence numbers out of order; if a late commit appears below the cache watermark, that mismatch causes a one-time full rebuild instead of silently omitting the row. A sequence reset (for example TRUNCATE ... RESTART IDENTITY) similarly causes a full rebuild. Restarting a backend starts with an empty cache, so no cache state is required for recovery.

This removes repeated decryption and JSON serialization of the complete history without weakening write durability or returning uncommitted messages. Set FEED_CACHE_CHUNK_BYTES only when measuring a different streaming chunk size; 65536 is the deployment default.

Backend HTTP sockets also use bounded receive and send buffers. The defaults are 32 KiB (HTTP_SOCKET_READ_BUFFER_BYTES=32768 and HTTP_SOCKET_WRITE_BUFFER_BYTES=32768). This prevents concurrent full-feed streams from allowing kernel TCP autotuning to consume a backend's 512 MiB cgroup. Linux normally reports twice the requested value through SO_RCVBUF and SO_SNDBUF; the backend logs both requested and effective listener values.

For local testing, the processes similarly share one SQLite file. This works only because they are on the same computer.

WebSocket scope

Secure WSS registration, RSA-PSS verification, encrypted persistence, explicit history, and broadcasting are retained. With ENABLE_CLUSTER_WEBSOCKET=1 on every writer, each successful new HTTP or WSS commit publishes its message ID through PostgreSQL LISTEN/NOTIFY. Backends with connected WSS users load that committed row and broadcast it locally, so clients distributed across backend processes receive the same new messages. Publication is queued after commit and does not block the HTTP response path. With the option disabled, the evaluated HTTP /message and /feed paths retain their previous behavior.

Registration no longer pushes old messages automatically. The first response is:

{"type":"registered","username":"Alice"}

History is loaded only when that client requests it, and is never broadcast to other users:

{"type":"load_history","limit":25,"before_sequence":1000}

before_sequence is optional. The response includes has_more and next_before_sequence for backward pagination. HISTORY_LIMIT is the maximum page size.

The load balancer can expose a single secure WebSocket endpoint using TLS passthrough. It chooses only HTTP-health-checked, memory-safe backends and then prefers the one with the fewest active WSS connections. TLS and message signatures remain end-to-end between the client and selected backend.

Dynamic selection and health checking

The performance score is tuned for the observed I/O-bound PostgreSQL workload:

score = 25% CPU + 45% active-request saturation + 30% latency saturation

Memory uses separate routing and safety guards. Backend working-set memory is memory.current - inactive_file, clamped at zero. This avoids treating reclaimable PostgreSQL file cache as permanently occupied application memory; total cgroup usage remains authoritative for OOM protection. The default behavior is:

Poll /internal/stats every 500ms and cache the result.
Poll /health every 1s with a 2.5s monitoring timeout. Health and statistics probes are single-flight per backend: a new tick is skipped while its previous probe is still running.
Remove unhealthy backends from routing after two consecutive failures.
Among healthy backends, consider those with score below the configured threshold, working-set memory below 95%, and total cgroup memory below the hard 98% limit.
Choose the acceptable backend with the fewest outstanding requests and rotate exact ties. Locally reserved requests are included immediately, even between statistic polls.
As soon as a backend crosses the threshold, exclude it until its score recovers. Observed proxy latency decays with a five-second half-life so an old slow request cannot penalize a recovered backend forever.
If every healthy backend exceeds the score threshold, use the least-loaded backend that is still below both memory guards and the concurrency limit. Fallback never bypasses those safety constraints. Working-set pressure must recover five percentage points below its limit, and total pressure four points below its limit, before that backend becomes eligible again.
If no backend is healthy, return HTTP 503.
Optional admission control can bound concurrent POST /message requests and reject excess work immediately. Feed and health requests always bypass it.
With a per-backend cap enabled, requests wait at the load balancer for a safe slot instead of consuming memory in an already saturated Python process. Message reservations cover the complete response. A feed releases its dynamic-selection slot after headers. Message and feed proxy connections use separate bounded transports, so long feed bodies cannot consume the message connection pool. The feed transport remains occupied until the complete body is read or closed.
Sys1 memory is monitored independently from backend memory. At the total-memory or socket-memory threshold, the LB closes idle client and backend connections, forces an emergency heap scavenging pass, and temporarily rejects new /message and /feed operations with Connection: close. Health and LB metrics remain available, and hysteresis re-enables application traffic after pressure falls.
When the normal accepted-client cap is reached, the listener first closes tracked HTTP-idle connections and sufficiently old no-request connections left by previous stages. If no safe connection can be reclaimed, a bounded guarded reserve can absorb a stage transition while memory and socket usage remain below their safety margins. Active requests are never reclaimed.
Accepted client connections, outbound connections per backend, and TCP send/receive buffers are explicitly bounded so Linux socket autotuning cannot consume the entire 512 MiB cgroup.

If statistics become stale, the backend remains usable when its health check is good, but receives a score penalty and is less likely to be selected.

Configuration flags:

Flag	Default	Meaning
`-listen`	`:8080`	Public load-balancer address
`-backends`	required	Comma-separated backend base URLs
`-threshold`	`75`	Maximum normal eligibility score
`-max-memory`	`95`	Backend working-set memory exclusion percentage
`-max-total-memory`	`98`	Hard backend total-cgroup memory exclusion percentage
`-lb-max-memory`	`80`	Sys1 memory percentage that activates application shedding
`-lb-max-socket-bytes`	`167772160`	TCP socket bytes that activate the early guard; `0` disables it
`-max-inflight`	`100`	In-flight count treated as 100% request load
`-admission-limit`	`0`	Hard concurrent message limit; `0` disables rejection
`-backend-cap`	`0`	Hard concurrent request limit per backend; `0` disables it
`-backend-queue-timeout`	`10s`	Maximum wait for a safe backend slot
`-latency-target`	`100ms`	Latency treated as 100% latency load
`-stats-interval`	`500ms`	Statistics polling interval
`-health-interval`	`1s`	Health polling interval
`-monitor-timeout`	`2.5s`	Health/statistics request timeout
`-backend-timeout`	`15s`	Proxied backend response-header timeout
`-failure-limit`	`2`	Consecutive failures required to mark down
`-proxy-max-idle`	`256`	Total retained idle backend connections
`-proxy-max-idle-per-host`	`64`	Retained idle connections per backend
`-proxy-max-conns-per-host`	`200`	Total active and idle non-feed proxy connections per backend
`-proxy-max-feed-conns-per-host`	`4`	Total active and idle feed proxy connections per backend
`-proxy-idle-timeout`	`10s`	Backend connection idle timeout
`-client-idle-timeout`	`5s`	Incoming client connection idle timeout
`-client-max-connections`	`2200`	Simultaneously accepted client connections; `0` disables the cap
`-client-connection-burst`	`400`	Guarded slots above the normal client cap; disabled near memory/socket limits
`-client-new-timeout`	`2s`	Minimum age before a no-request `StateNew` socket can be reclaimed at the cap
`-socket-read-buffer-bytes`	`16384`	Requested TCP receive-buffer size on client/backend sockets
`-socket-write-buffer-bytes`	`16384`	Requested TCP send-buffer size on client/backend sockets
`-proxy-feed-read-buffer-bytes`	`0`	Feed backend receive-buffer size; `0` inherits the general read buffer
`-client-feed-write-buffer-bytes`	`0`	Client send-buffer size while serving `/feed`; `0` inherits the general write buffer
`-proxy-buffer-bytes`	`32768`	Shared reverse-proxy copy-buffer size
`-proxy-feed-buffer-bytes`	`0`	Feed-only copy-buffer size; `0` inherits `-proxy-buffer-bytes`
`-wss-listen`	disabled	Optional public TCP listener for WSS TLS passthrough
`-wss-backends`	disabled	Backend WSS `host:port` values in the same order as `-backends`
`-wss-max-connections`	`512`	Maximum simultaneous WSS connections through the LB

Failure and recovery behavior

When one backend process or machine fails

Existing requests on that machine can fail with HTTP 502.
Proxy connection errors count as passive health failures immediately.
Active /health polling also detects the failure.
After the configured failure limit, the backend is marked down and receives no new requests.
Remaining healthy backends continue serving from the same database.
Health polling continues for the failed backend.
After it restarts, its first successful /health check marks it up.
Its statistics are refreshed and it becomes dynamically eligible again.
It reads all previously committed messages from PostgreSQL; no backlog replay or local database merge is required.

With the defaults, active detection normally requires two failed checks roughly one second apart, plus connection timeouts. Passive proxy failures may detect a problem sooner.

If failure happens while storing a message

There are two important cases:

Failure before database commit: no message is acknowledged or stored. The client may retry the same message ID through the load balancer; another backend can insert it.
Database commit succeeds but the backend dies before its response reaches the client: the client may retry the same ID. The new backend finds the existing row and returns status: duplicate, preventing a second insertion.

The load balancer intentionally does not automatically replay POST bodies. An automatic retry would be unsafe when the caller did not provide a stable message ID. Retry responsibility currently belongs to the client or load generator.

If the caller disconnects while a request is waiting for PostgreSQL, aiohttp cancels that backend handler. This releases its pool wait and prevents abandoned requests from building a hidden backlog across load-test stages. A cancellation that arrives after the database commit remains safe: retrying the same message ID returns the existing row instead of inserting a duplicate.

If PostgreSQL itself fails

Every backend's /health check fails because the service cannot safely persist or read shared data. The load balancer eventually reports no healthy backend and returns HTTP 503. Messages are not queued in backend memory because acknowledging uncommitted data could lose it or create duplicates. When PostgreSQL recovers, successful health checks automatically return the backends to service.

One PostgreSQL instance is therefore a database single point of failure. Database replication/high availability can be added later, but is not the same feature as the assignment's three application backends.

Installing the PostgreSQL service on Sys3

These commands assume Ubuntu. The selected hosted layout reserves Sys1 for the load balancer and runs PostgreSQL beside the Sys3 backend. Sys2 and Sys4 reach it through the platform-mapped database port; Sys3 uses the local PostgreSQL port. PostgreSQL listens only on localhost by default, so remote backends require both listen_addresses and pg_hba.conf changes.

Official references: Ubuntu PostgreSQL setup, PostgreSQL connection settings, and PostgreSQL client authentication.

1. Install and start PostgreSQL on Sys3

sudo apt update
sudo apt install -y postgresql postgresql-contrib
sudo systemctl enable --now postgresql
sudo systemctl status postgresql --no-pager

2. Create the role and database

The first command securely prompts for the database password:

sudo -u postgres createuser --pwprompt chat_user
sudo -u postgres createdb --owner=chat_user chat
sudo -u postgres psql -c '\l'

3. Locate and edit the active configuration

sudo -u postgres psql -tAc 'SHOW config_file;'
sudo -u postgres psql -tAc 'SHOW hba_file;'

In the reported postgresql.conf, listen on Sys3's interfaces. This deployment uses local port 4000:

listen_addresses = '*'
port = 4000

In this classroom platform, mapped connections arrive through the container gateway. Restrict access to that observed source address; /32 means exactly one IPv4 address:

hostssl  chat  chat_user  172.17.0.1/32  scram-sha-256

For an isolated classroom network without database TLS, host can replace hostssl, but encrypted database connections are preferable.

Restart and verify:

sudo systemctl restart postgresql
sudo systemctl is-active postgresql
sudo ss -ltnp | grep 4000

When systemd is not PID 1, use:

sudo pg_ctlcluster --skip-systemctl-redirect 16 main restart
pg_isready -h 127.0.0.1 -p 4000

512 MiB per-system memory limits

Each allotted system has a 512 MiB cgroup. Sys1 now runs only the LB; Sys3 runs PostgreSQL and one backend. Use small connection pools on all backends:

export DB_POOL_MIN=1
export DB_POOL_MAX=6
export DB_HEALTH_TIMEOUT=1
export HEALTH_CHECK_TIMEOUT=1.5
export DB_WRITE_BATCH_SIZE=64
export DB_WRITE_BATCH_DELAY_MS=2
export DB_WRITE_BATCH_WORKERS=1
export DB_WRITE_QUEUE_MAX=256
export DB_WRITE_QUEUE_TIMEOUT=5
export FEED_CACHE_CHUNK_BYTES=65536
export HTTP_SOCKET_READ_BUFFER_BYTES=32768
export HTTP_SOCKET_WRITE_BUFFER_BYTES=32768

This permits at most 21 application database connections across the three backends, including dedicated health connections.

Size PostgreSQL for the same limit:

sudo -u postgres psql -p 4000 -c "ALTER SYSTEM SET shared_buffers = '64MB';"
sudo -u postgres psql -p 4000 -c "ALTER SYSTEM SET max_connections = 40;"
sudo -u postgres psql -p 4000 -c "ALTER SYSTEM SET work_mem = '1MB';"
sudo -u postgres psql -p 4000 -c "ALTER SYSTEM SET maintenance_work_mem = '32MB';"
sudo -u postgres psql -p 4000 -c "ALTER SYSTEM SET autovacuum_work_mem = '16MB';"
sudo pg_ctlcluster --skip-systemctl-redirect 16 main restart

Both the Python backend and Go load balancer prefer cgroup v2 CPU and memory counters over host-wide values, so /internal/stats and /lb/metrics report actual container pressure. CPU usage is normalized against the quota in cpu.max; this prevents workloads in neighboring containers from changing the backend routing score. The load_1m field remains a host-wide informational value because cgroup v2 has no equivalent load-average counter, and it is not used by the routing algorithm.

If UFW is enabled, allow only the platform gateway/source address. Do not enable or broadly reconfigure a firewall over SSH without confirming existing access rules.

sudo ufw allow from 172.17.0.1 to any port 4000 proto tcp

4. Test database access from Sys2, Sys3, and Sys4

Install the PostgreSQL client if necessary:

sudo apt install -y postgresql-client
# From Sys2 and Sys4, use Sys3's mapped external database port.
psql 'postgresql://chat_user:YOUR_PASSWORD@SYS3_PRIVATE_IP:EXTERNAL_DB_PORT/chat?sslmode=require' \
  -c 'SELECT current_database(), current_user, NOW();'

# From Sys3, use PostgreSQL directly.
psql 'postgresql://chat_user:YOUR_PASSWORD@127.0.0.1:4000/chat?sslmode=require' \
  -c 'SELECT current_database(), current_user, NOW();'

If host rather than hostssl was configured, remove ?sslmode=require.

5. Install backend dependencies on all three backend systems

The pinned dependency set targets Python 3.14. Install the same versions on Sys2, Sys3, and Sys4 so backend behavior does not vary between machines.

python3.14 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r backend/requirements.txt
python -m pip check

asyncpg is the PostgreSQL driver. It is intentionally required for the hosted configuration but was not installed in the original local environment. URL-encode special characters in database usernames or passwords before putting them in the URI.

6. Generate one shared AES key

Run this once, store the output securely, and configure the identical value on Sys2, Sys3, and Sys4:

python3 -c 'import base64,os; print(base64.urlsafe_b64encode(os.urandom(32)).decode())'

Do not generate a separate encryption key on each backend; otherwise one backend cannot decrypt messages written by another.

7. Detailed `main` backend reference

The branch-specific quick-start section above is authoritative. This expanded example shows the same main backend environment in one block.

Example environment on Sys2:

export BACKEND_ID=sys2
export HTTP_PORT=6000
export WS_PORT=5000
export DATABASE_URL='postgresql://chat_user:YOUR_PASSWORD@SYS3_PRIVATE_IP:EXTERNAL_DB_PORT/chat?sslmode=require'
export MESSAGE_ENCRYPTION_KEY_FILE=/home/student/DLB/data/encryption.key
export DB_POOL_MIN=1
export DB_POOL_MAX=6
export DB_HEALTH_TIMEOUT=1
export HEALTH_CHECK_TIMEOUT=1.5
export DB_WRITE_BATCH_SIZE=64
export DB_WRITE_BATCH_DELAY_MS=2
export DB_WRITE_BATCH_WORKERS=1
export DB_WRITE_QUEUE_MAX=256
export DB_WRITE_QUEUE_TIMEOUT=5
export FEED_CACHE_CHUNK_BYTES=65536
export HTTP_SOCKET_READ_BUFFER_BYTES=32768
export HTTP_SOCKET_WRITE_BUFFER_BYTES=32768
export ENABLE_WEBSOCKET=1
export ENABLE_CLUSTER_WEBSOCKET=1
export WS_EVENT_CHANNEL=chat_message_events
export WS_EVENT_QUEUE_MAX=4096
export TLS_CERT=/home/student/chat-ssl/cert.pem
export TLS_KEY=/home/student/chat-ssl/key.pem
python3 backend/server.py

On Sys4, use the same remote database URL and encryption-key file but change BACKEND_ID. On Sys3, set DATABASE_URL='postgresql://chat_user:YOUR_PASSWORD@127.0.0.1:4000/chat?sslmode=require'. The identical encryption key file must exist on all three backends. The backend creates its required tables automatically.

Test shared storage before starting the load balancer:

# Send to Sys2.
curl -H 'Content-Type: application/json' \
  -d '{"client-name":"DB-test","msg":"shared","message-id":"shared-test-1"}' \
  http://SYS2_IP:6000/message

# Read the same committed row through Sys3 and Sys4.
curl -sS http://SYS3_IP:6000/feed | python3 -m json.tool
curl -sS http://SYS4_IP:6000/feed | python3 -m json.tool

8. Detailed `main` load-balancer reference

This is the complete-application command. Omit the three -wss-* flags when running leaderboard-session, as shown in the branch guide above.

cd load-balancer
go build -buildvcs=false -o dlb .
GOMEMLIMIT=192MiB GOGC=100 ./dlb \
  -listen :5000 \
  -backends http://SYS2_IP:6000,http://SYS3_IP:6000,http://SYS4_IP:6000 \
  -wss-listen :7000 \
  -wss-backends SYS2_WSS_HOST:PORT,SYS3_WSS_HOST:PORT,SYS4_WSS_HOST:PORT \
  -wss-max-connections 512 \
  -threshold 60 \
  -max-memory 90 \
  -max-total-memory 98 \
  -lb-max-memory 82 \
  -lb-max-socket-bytes 167772160 \
  -max-inflight 100 \
  -admission-limit 0 \
  -backend-cap 200 \
  -backend-queue-timeout 10s \
  -proxy-max-idle 256 \
  -proxy-max-idle-per-host 64 \
  -proxy-max-conns-per-host 200 \
  -proxy-max-feed-conns-per-host 4 \
  -proxy-idle-timeout 10s \
  -client-idle-timeout 5s \
  -client-max-connections 2600 \
  -client-connection-burst 0 \
  -client-new-timeout 2s \
  -socket-read-buffer-bytes 16384 \
  -socket-write-buffer-bytes 16384 \
  -proxy-feed-read-buffer-bytes 131072 \
  -client-feed-write-buffer-bytes 131072 \
  -proxy-buffer-bytes 32768 \
  -proxy-feed-buffer-bytes 32768

The feed pool is deliberately limited to four connections per backend because each populated feed is about 16 MiB. In the fixed-seed 50,000-operation suite, cap 4 completed all 47,607 message requests and reduced total errors from 4.188% at cap 48 to 4.132%, while improving throughput from 234.55 to 259.11 requests/second. This prevents concurrent large feeds from starving the higher-volume message path.

Feed TCP windows are independently set to 128 KiB while general/message sockets remain at 16 KiB. In two clean static comparisons, the selected feed setting completed 376 and 383 of 953 feed operations with 2.885% and 2.85% total error, beating the 64 KiB repeat's 365 feeds and 2.94% error. Sys1 stayed below 60% memory in both static runs, with less than 7.1 MiB of socket memory. Application copy buffers remain 32 KiB because both the 16 and 64 KiB feed-only copy-buffer experiments increased errors.

The load balancer deliberately proxies /feed directly through the isolated, four-connection-per-backend feed transport. The rejected load-balancer snapshot cache and its pressure/delivery controls have been removed from the deployable code, leaving one explainable feed path and no dormant cache flags. A routed feed releases its general backend-cap reservation before it waits for that isolated transport; consequently slow queued feeds cannot consume the 200 slots needed by message requests, while MaxConnsPerHost=4 continues to bound actual feed connections to each backend.

The final code uses a flat 2,600 client-connection cap. Test 23 used a 2,200 base plus a guarded 400-connection reserve; the reserve became unavailable at the 2,500-user transition and rejected 5,573 connections even though the Sys1 memory guard never fired. Two later local qualification runs used 2,600+0, admitted peaks of 2,501 and 2,581 clients, produced zero listener rejection and completed the final stage normally. This repair was validated but was not submitted for a new official score before the deadline. At the cap, the listener may reclaim HTTP-idle sockets and no-request StateNew sockets older than two seconds; active requests are never reclaimed.

Keep admission control disabled on the hosted three-backend deployment. In the fixed-seed 20,000-message comparison, -admission-limit 800 rejected 2,125 requests, reduced throughput from 486.92 to 411.53 requests/second, and increased mean latency from 1,128 to 1,257 ms. The option remains available for deployments that explicitly prefer fast load shedding over message acceptance. /lb/metrics reports rejected_requests, and each backend snapshot reports skipped overlapping probes.

The hosted systems are limited to 512 MiB. A leaderboard run killed Sys2 at 94.57% reported memory with 247 active requests, so the deployment uses a 90% hard memory guard and a 200-request per-backend cap. Repeated proxy errors are rate-limited in the terminal; proxy_errors_suppressed retains their count.

After PostgreSQL was moved off Sys1, leaderboard test 9 still OOM-killed the LB with GOMEMLIMIT=320MiB. The selected 192 MiB target leaves room for goroutine stacks, kernel TCP buffers, page tables, and memory outside the Go runtime. /lb/metrics exposes Go heap/runtime/stack values, cgroup anon/file/kernel/socket bytes, incoming connection states, memory-guard events, memory-triggered rejections, and idle connections closed.

Leaderboard test 10 confirmed that Go memory was no longer the immediate bottleneck: the runtime was about 70 MiB while cgroup socket memory reached 442 MiB and triggered one new OOM kill. The current socket limits, connection caps, 100 ms guard sampling, and mixed-feed generator profile directly target that measured failure instead of lowering GOMEMLIMIT again.

Submit the reachable Sys1 load-balancer URL, never a backend URL.

For long-running deployment, systemd service units should be added after the IPs, project paths, environment file locations, and Linux usernames are finalized.

Automated checks

Static and race checks:

python3 -m py_compile backend/server.py
cd load-balancer
go test -race -buildvcs=false ./...

Full local integration check:

bash scripts/integration-test.sh

After enabling the PostgreSQL event bus on all three hosted backends, verify cluster-wide live delivery through the single Sys1 WSS endpoint:

python3 scripts/wss-cluster-test.py \
  --url wss://10.1.75.53:7277 \
  --insecure

The script connects three users through the DLB, checks that all three receive one newly committed signed message, then connects a late user and verifies that the message appears only after an explicit history request.

It verifies:

Three initially healthy backends.
HTTP creation through the load balancer.
Shared feed visibility through a different backend.
Fifty concurrent retries producing exactly one row.
TLS WebSocket registration through the load balancer, explicit history, and RSA-PSS-signed message persistence.
One backend being removed after shutdown.
Continued writes through the two remaining backends.

Name		Name	Last commit message	Last commit date
backend		backend
load-balancer		load-balancer
load-generator		load-generator
previous-codes-used		previous-codes-used
report		report
scripts		scripts
.gitignore		.gitignore
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation

Dynamic Load Balancer — Lab 6

Assignment status

Project components

Choose and run a branch

Shared backend environment

Run leaderboard-session

Run main with distributed WSS

Verify either deployment

Local four-terminal test

Terminal 1 — Sys2 simulation

Terminal 2 — Sys3 simulation

Terminal 3 — Sys4 simulation

Terminal 4 — dynamic load balancer

Public API reference

POST /message

GET /feed

Load-balancer monitoring API

GET /lb/health

GET /lb/status

GET /lb/metrics