Skip to content

feat(POP-4266): introduce anon stats sampling_rate for di 2d stats - #143

Merged
leonanos8 merged 2 commits into
mainfrom
leonidasnanos/pop-4266-on-call-followup-handle-growing-disk-in-iris-mongo-due-to-di
Aug 28, 2026
Merged

feat(POP-4266): introduce anon stats sampling_rate for di 2d stats#143
leonanos8 merged 2 commits into
mainfrom
leonidasnanos/pop-4266-on-call-followup-handle-growing-disk-in-iris-mongo-due-to-di

Conversation

@leonanos8

@leonanos8 leonanos8 commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Description

Adds sampling support to the shared DI 2D anon-stats pipeline, allowing randomly sampled pairs and fully retained exception pairs to be stored, processed, and published separately.

Details:

  • Adds sampling_rate metadata to PostgreSQL rows and published 2D histograms.
  • Defines sampling_rate=20 for the unbiased 20% random sample.
  • Defines sampling_rate=100 for unsampled pairs retained by the score or opposite-mirror exception rules.
  • Separates PostgreSQL counting, fetching, processing, and cleanup by sampling rate and match flags.
  • Adds configuration for the sampling rates processed by the anon-stats server.
  • Adds a separate minimum job size for sparse fully retained rows.
  • Preserves existing behavior by defaulting existing rows and configurations to sampling_rate=100.
  • Adds validation and tests for sampling-rate configuration, serialization, and job-size limits.

Notes

@dkales @marsenis @evagelia I would like your perspective here on the isolated separate minimum job size for sparse fully retained rows. This controls on how often we publish histograms for cases that had big scores (>500). Default value is set to 20 but we maybe want to increase this to avoid a narrow time window privacy attack.

@leonanos8
leonanos8 force-pushed the leonidasnanos/pop-4266-on-call-followup-handle-growing-disk-in-iris-mongo-due-to-di branch from 7c6a6e5 to 5e6b522 Compare August 26, 2026 14:25
@leonanos8
leonanos8 requested a review from a team August 27, 2026 16:59
@leonanos8
leonanos8 requested a review from evagelia August 27, 2026 16:59
@leonanos8
leonanos8 requested a review from dkales August 27, 2026 16:59
@leonanos8
leonanos8 merged commit b15125b into main Aug 28, 2026
13 of 14 checks passed
@leonanos8
leonanos8 deleted the leonidasnanos/pop-4266-on-call-followup-handle-growing-disk-in-iris-mongo-due-to-di branch August 28, 2026 11:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants