Memory Leak in Sidekiq Worker Processes

[ahnv]

Sidekiq worker processes exhibit unbounded memory growth in production. RSS increases monotonically across sync cycles and does not recover between jobs, eventually triggering OOM kills.

Instrumentation Added

Two pieces of instrumentation were added to aid reproduction and diagnosis. These changes should be kept for ongoing monitoring but the flamegraph feature should remain opt-in in production.

1. lib/sidekiq/memory_profiling_middleware.rb (new file)
A Sidekiq server middleware that logs GC heap stats and object type deltas after every job:

[MemoryProfiling] job=Sidekiq::ActiveJob::Wrapper queue=high_priority
heap_live_delta=488847 objects_allocated=774155 gc_runs=1
T_OBJECT=+111069 T_STRING=+124374 T_ARRAY=+52819 T_HASH=+37179 T_DATA=+95859

Registered in config/initializers/sidekiq.rb.

2. Process-wide StackProf flamegraph export (in config/initializers/sidekiq.rb)
When SIDEKIQ_FLAMEGRAPH=1 is set, starts a stackprof object-allocation profile on worker boot and writes a .dump file to SIDEKIQ_FLAMEGRAPH_DIR (default /tmp) on graceful shutdown:

[MemoryProfiling] Process profile written: /tmp/stackprof_process_88444_1234567890.dump

Convert with:

bundle exec stackprof --json /tmp/stackprof_process_*.dump > flame.json
# upload flame.json to speedscope.app → Sandwich view, sort by Self

stackprof gem moved out of the development group in Gemfile so it is available in production.

---

Observed Behavior

Per-job middleware shows each SyncJob retaining ~488k objects after GC:

heap_live_delta=488847 objects_allocated=774155 gc_runs=1
T_OBJECT=+111069 T_STRING=+124374 T_ARRAY=+52819 T_HASH=+37179 T_DATA=+95859

With Sidekiq concurrency of 3, a single sync cycle retains ~1.4M objects. Workers are killed by OOM before completing their queue.

Investigation

Object allocation flamegraph (Sandwich view, sorted by Self) identified:

Method	Self allocations	%
Class#new	63,163	12%
Kernel#dup	47,289	8.7%
Kernel#BigDecimal	42,569	7.8%
BigDecimal#round	31,273	5.7%
PG::Result#each	21,592	4.0%
ActiveModel::AttributeAssignment#_assign_attribute	21,420	3.9%
ActiveRecord::AttributeAssignment#_assign_attributes	21,405	3.9%
Balance::Materializer#persist_balances	21,202	3.9%
Balance::SyncCache#get_entries	3,199	0.6%

Date#upto accounts for 45% of total allocations as the call tree root.

Suspected Root Cause

Balance::Materializer#persist_balances and Holding::Materializer#persist_holdings instantiate full AR models (Balance.new(...), Holding.new(...)) for every date in the calculation range, then immediately call .attributes.slice(...) to serialize them for upsert_all — incurring Class#new, Kernel#dup, and the full AR attribute machinery per row with no benefit.

Balance::SyncCache#converted_entries calls e.dup on every Entry AR model for FX conversion (secondary issue).

Files to Investigate

• app/models/balance/materializer.rb — persist_balances
• app/models/balance/base_calculator.rb — build_balance
• app/models/holding/materializer.rb — persist_holdings
• app/models/holding/forward_calculator.rb / reverse_calculator.rb — build_holdings
• app/models/balance/sync_cache.rb — converted_entries

Steps to Reproduce

SIDEKIQ_FLAMEGRAPH=1 SIDEKIQ_FLAMEGRAPH_INTERVAL=50 bin/dev
# trigger a sync, then Ctrl+C for graceful shutdown
bundle exec stackprof --json /tmp/stackprof_process_*.dump > flame.json
# upload flame.json to speedscope.app → Sandwich view, sort by Self

[dosubot]

Hi @ahnv! I'm Dosu and I’m helping the sure team.

Great investigation — the suspected root cause checks out. I dug into the relevant code and can confirm all three allocation hotspots.

1. persist_balances / build_balance — build_balance creates a full Balance.new(...) with 14 attributes [1], then persist_balances immediately discards the AR machinery via .attributes.slice(...) [2]. For a 2-year account that's ~730 AR objects instantiated and thrown away per sync.

2. persist_holdings / build_holdings — Same pattern: Holding.new(...) is built [3] then sliced to a hash for upsert_all [4]. A 2-year investment account with 10 securities creates ~7,300 throwaway Holding objects.

3. converted_entries — e.dup on every Entry AR model for FX conversion [5]. Notably, the analogous pattern for holdings was already fixed in PR #1581, which replaced converted_holdings duplication with numeric aggregation and eliminated 100,000+ allocations per sync.

Why .attributes.slice() exists: The Balance model has database-generated virtual columns (start_balance, end_cash_balance, etc.) that can't be passed to upsert_all. So the code instantiates full objects and then strips those fields. The fix is to never build the AR object in the first place — just construct the hash directly in build_balance/build_holdings.

Additional context: PR #1910 (open) proposes jemalloc, showing 57% less memory growth vs. glibc malloc. This would complement the allocation reduction fixes since jemalloc is better at returning freed pages to the OS between GC cycles. Community reports confirm persistent memory creep in self-hosted deployments even with MALLOC_ARENA_MAX=2.

Both the hash-based builder refactor (eliminating the allocations) and jemalloc (improving allocator behavior for the remaining allocations) are likely needed for a full resolution.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[ahnv]

Confirmed Leak Site: Per-history accumulation in Balance::Materializer (and Holding::Materializer)

What the StackProf chain says

SyncJob → Sync#perform
  → Account::Syncer#perform_sync (92.3%)
      → Account::Syncer#materialize_balances → Balance::Materializer#materialize_balances (89.9%)
          ├─ calculate_balances      218,035 (43.5%)   ← Balance::ReverseCalculator builds full array
          ├─ persist_balances        180,682 (36.1%)   ← AR attribute hashes for every row
          └─ materialize_holdings    100,868 (20.1%)
                ├─ calculate_holdings 48,589 (48.2%)   ← Holding::ReverseCalculator builds full array
                └─ persist_holdings   35,577 (35.3%)

These three accumulators (@balances, @holdings, and the hash arrays they're persisted from) are exactly what shows up retained in your live_after_gc snapshots: Balance, Holding, ActiveModel::Attribute::FromDatabase/FromUser, BigDecimal, BelongsToAssociation.

The flat top confirms the cost driver is BigDecimal-heavy AR object construction over a daily loop

Class#new                                   180,475 (32.4%)   ← AR objects (Balance/Holding) built one-per-row
Kernel#dup                                   76,621 (13.8%)
Kernel#BigDecimal                            43,010  (7.7%)
BigDecimal#round                             31,231  (5.6%)
BigDecimal#<=>                               23,424  (4.2%)
ActiveRecord::AttributeAssignment#_assign_attributes  64,347 (11.6%)
ActiveModel::AttributeAssignment#_assign_attribute    43,053  (7.7%)
Date#upto                                   246,164 (44.2%)   ← daily loop is the umbrella
Enumerable#index_with                        43,809  (7.9%)
Hash#each                                    94,019 (16.9%)

Date#upto covering 44.2% of all allocations means every allocation downstream is multiplied by the number of days in the account history. For an account with several years of data this is tens of thousands of iterations, each producing N Balance and Holding AR objects with their BigDecimal attribute pairs.

Specific smoking guns

1. Holding::ReverseCalculator#calculate_holdings — app/models/holding/reverse_calculator.rb:26

    def calculate_holdings
      # Start with the portfolio snapshot passed in from the materializer
      current_portfolio = portfolio_snapshot.to_h
      previous_portfolio = {}

      holdings = []

      Date.current.downto(account.start_date).each do |date|
        today_trades = portfolio_cache.get_trades(date: date)
        previous_portfolio = transform_portfolio(current_portfolio, today_trades, direction: :reverse)

        # If current day, always use holding prices ...
        holdings.concat(build_holdings(current_portfolio, date, price_source: date == Date.current ? "holding" : nil))
        current_portfolio = previous_portfolio

holdings = [] accumulates every Holding for every day from Date.current back to account.start_date, for every security in the portfolio. At peak you saw Holding=9,552 retained post-GC.

2. Balance::ReverseCalculator#calculate — app/models/balance/reverse_calculator.rb:13

      # Calculates in reverse-chronological order (End of day -> Start of day)
      account.current_anchor_date.downto(account.opening_anchor_date).map do |date|

Same pattern: .map { … build_balance(...) } materializes one Balance per day from anchor to opening — typically years — and returns the whole array. Balance::Materializer#calculate_balances then keeps the result in @balances until persist_balances runs.

3. Balance::Materializer#materialize_balances — app/models/balance/materializer.rb:11

  def materialize_balances
    Balance.transaction do
      materialize_holdings
      calculate_balances

      Rails.logger.info("Persisting #{@balances.size} balances")
      persist_balances

Both @balances (all dates × cash/non-cash) and @holdings (all dates × securities) live simultaneously inside one transaction. persist_balances then walks @balances to build attribute hashes for insert_all/upsert_all — that's the additional 180,682 allocations and the 11.6% _assign_attributes cost.

Why this matches the runtime symptoms exactly

Log observation	Explained by
heap_live_delta spikes of 600k–1.1M slots per SyncJob	@balances + @holdings arrays of full AR objects across the entire account window
Post-GC ActiveModel::Attribute::FromDatabase=174,384, FromUser=196,764	Attribute objects on the AR Holding/Balance instances kept reachable while the materializer/transaction is on the stack
Post-GC BigDecimal=94,485	Two BigDecimals per Balance (cash, non-cash) plus per-Holding amount/qty/price across all dates
Post-GC BelongsToAssociation=14,549, Holding=9,552, Entry=5,530, Balance=5,790	The accumulated arrays themselves
Memory drops sharply on the next job	Materializer goes out of scope after sync finishes; nothing is permanently rooted — this is per-job peak, not a true leak

Diagnosis

This is a peak-retention / unbatched-accumulation problem, not a classic "leak that grows forever." Memory is reclaimed after each SyncJob finishes (you can see it in the logs: baselines fall back to ~280k Strings, ~7.7k FromDatabase). But within a single SyncJob, the materializers hold the entire account history in memory at once, producing peaks large enough to OOM the worker and to inflate the post-GC snapshot mid-run.

The slow long-term creep (Symbol +24%, Class +136, Proc +210) is a separate, smaller issue unrelated to this StackProf — likely route-cache / dynamic to_sym / anonymous-class generation elsewhere. The dominant memory pressure is the per-job peak above.

Where to fix

The minimal-impact directions, in priority order:

1. Batch the daily loop in Balance::ReverseCalculator and Holding::ReverseCalculator — yield/persist per N-day chunks (e.g. monthly or yearly windows) instead of returning one giant array. The transaction in Balance::Materializer#materialize_balances can stay; the in-memory window doesn't need to.
2. Skip AR object construction in the persist path. persist_balances / persist_holdings should build plain attribute hashes directly for insert_all / upsert_all rather than instantiating Balance.new / Holding.new and then converting. That eliminates the _assign_attributes (11.6%) and a chunk of the Class#new (32.4%) costs.
3. Reduce BigDecimal churn — 43,010 BigDecimal() calls + 31,231 #round + 23,424 #<=> + 76,621 #dup is a lot for a daily loop. Memoize the per-day cash ratio and round once at boundaries instead of per arithmetic step.
4. Drop account.holdings.reload at holding/materializer.rb:33 if downstream code can be made to accept the freshly-built collection — the reload is 15.8% of materialize_holdings

Summary

• The "leak" is Balance::Materializer holding @balances (from Balance::ReverseCalculator#calculate's full-history .map) and @holdings (from Holding::ReverseCalculator#calculate_holdings's full-history accumulator) simultaneously, per-account, per-sync.
• This explains every retained class in the post-GC snapshots and the 600k–1.1M heap_live_delta peaks on SyncJob.
• It is not a permanent leak — memory is reclaimed between jobs — but the per-job peak is large enough to look like one and to risk OOM.
• The fix is to batch the daily loop and build attribute hashes (not AR objects) in the persist path. The slow Symbol/Class/Proc baseline creep is a separate, smaller issue worth investigating after this is addressed.