Refactor Balance Update

[Ygohr]

Context

The transaction handler currently performs balance persistence synchronously within the same flow as transaction and operation writes. This coupling causes blocking database writes during transaction processing, where hot accounts experience disproportionate latency due to frequent balance updates.

The architecture follows Hexagonal Architecture (Ports & Adapters) with CQRS patterns. The proposed change introduces Event-Driven Decoupling with Eventual Consistency for balance persistence.

Key Components:

• Transaction Handler (Command): Processes transaction requests
• Cache Layer (Hot Balance): Stores real-time balance state with atomic operations
• Balance Sync Worker (Async): Persists hot balances to cold storage
• Relational DB (Cold Balance): Durable storage for balance snapshots

Current Behavior

The transaction handler executes all operations in a single synchronous flow:

1. Update Cache (atomic cache script)
2. Update DB (BLOCKING) - Balance persistence blocks the handler
3. Persist Transaction
4. Persist Operations

This causes:

• Lock contention on append-only operations
• High latency for hot accounts with frequent updates
• 1:1 ratio of balance DB writes per transaction

Goal

Decouple balance persistence from the transaction processing flow to enable append-only operations without blocking on balance synchronization.

Proposed Data Flow:

1. Transaction Handler (sync):

   • Update Cache (atomic)
   • Schedule Sync (non-blocking sorted set)
   • Persist Transaction (append-only)
   • Persist Operations (append-only)

2. Balance Sync Worker (async):

   • Poll cache for near-expiry balances
   • Aggregate by composite key (OrganizationID, LedgerID, AccountID, AssetCode, Key)
   • Persist only the latest version per balance
   • Remove synced balances from schedule

Architecture Changes:

• Remove UpdateBalances call from transaction handler
• Extend existing worker with in-memory aggregation capability
• Add observability for aggregation (metrics, logs, traces)

Success Metrics

• Balance DB writes reduced from 1:1 to N:1 ratio (aggregated)
• Transaction throughput increased by >2X ops/s
• Transaction P99 latency reduced (no balance write wait)
• Transaction handler lock time excludes balance write
• Cold balance synced before cache TTL expires
• Aggregation ratio and queue depth metrics available
• Sync lag alerts configured