diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/README.md b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/README.md new file mode 100644 index 00000000..ba555771 --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/README.md @@ -0,0 +1,58 @@ +# Differential Privacy Budget Allocation for Business Analytics + +Allocate a fixed differential-privacy budget across a batch of analytics queries. Each query has a sensitivity, business value, population coverage, minimum and maximum allowable epsilon, an error limit, and a population group. A solution must return one epsilon allocation per query. + +The verifier checks hard constraints first: exact query coverage, finite numeric allocations, per-query bounds, total budget, maximum estimation error, and bounded group-level average error disparity. Feasible solutions are scored by a positive raw utility metric owned by the verifier. + +## Candidate Interface + +Implement `solve(instance)` in `scripts/init.py`. + +Input fields: + +- `queries`: list of query objects. +- `epsilon_total`: total privacy budget. +- `fairness.max_group_error_ratio`: maximum allowed ratio between the largest and smallest group average error. + +Return: + +```python +{"allocations": {query_id: epsilon, ...}} +``` + +The output must include exactly the query identifiers in the instance. + +## Scoring + +For feasible solutions, the raw metric is strictly positive and larger is better. It combines weighted analytics value with deterministic estimation-error penalties. Invalid or infeasible outputs receive the framework invalid score. + +Framework scoring uses `log2_baseline_ratio` normalization outside the verifier. + + +## Evaluation Contract + +The verifier recomputes `Verifier-owned positive raw utility: 1.0 plus the sum over queries of business_value * population_coverage * log1p(epsilon) minus deterministic error penalties, evaluated only after all feasibility checks pass.` and candidates must maximize it. +Each valid case is scored by `log2` improvement over the baseline and the final score is the +mean across cases. Invalid solutions receive `-1e18`. + +## Evaluation Design + +Setting: `offline_batch`. Arrival model: All analytics-query portfolios are generated deterministically from fixed seeds before solving. A candidate receives a complete static instance containing query sensitivities, business values, population coverage, group memberships, bounds, fairness thresholds, accuracy requirements, and total budget, then returns one structured allocation for that instance. + +Objective rationale: The primary objective is appropriate because privacy budget is a scarce resource and the economically relevant decision is the feasible allocation that preserves the most weighted analytics utility while controlling estimation error. Accuracy and fairness are hard feasibility requirements so the objective cannot trade them away beyond accepted policy limits. + +Literature alignment: The supplied brief aligns with differential-privacy budget-allocation work at the level of allocating limited privacy loss across multiple analytics queries and recomputing privacy loss and error from first principles. This benchmark differs by making the task an offline batch portfolio optimization problem with explicit business values, population coverage, fairness constraints, and a structured candidate output rather than an interactive privacy accountant or a single-query mechanism design task. + +Local execution is only for reviewed code: + +```bash +python verification/evaluator.py scripts/init.py --local +``` + +Publish evaluation requires Docker and the pinned runtime image: + +```bash +docker pull python:3.12.11-slim-bookworm@sha256:519591d6871b7bc437060736b9f7456b8731f1499a57e22e6c285135ae657bf7 +python verification/evaluator.py scripts/init.py +``` + diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/README_zh-CN.md b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/README_zh-CN.md new file mode 100644 index 00000000..6555f45c --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/README_zh-CN.md @@ -0,0 +1,58 @@ +# 面向业务分析的差分隐私预算分配 + +本任务要求在一批分析查询之间分配固定的差分隐私预算。每个查询都有敏感度、业务价值、覆盖人群比例、epsilon 上下界、最大误差限制以及所属人群组。解需要为每个查询返回一个 epsilon 分配值。 + +验证器会先检查硬约束:查询 ID 是否完整且精确匹配、分配值是否为有限数字、是否满足每个查询的上下界、总预算、最大估计误差,以及组级平均误差差异限制。只有可行解才会计算由验证器定义的正向原始效用指标。 + +## 参赛接口 + +在 `scripts/init.py` 中实现 `solve(instance)`。 + +输入字段: + +- `queries`:查询对象列表。 +- `epsilon_total`:总隐私预算。 +- `fairness.max_group_error_ratio`:最大允许的组间平均误差比值。 + +返回: + +```python +{"allocations": {query_id: epsilon, ...}} +``` + +输出必须精确包含实例中的所有查询 ID。 + +## 评分 + +对可行解,原始指标严格为正,且越大越好。该指标结合了加权业务价值和确定性的估计误差惩罚。无效或不可行输出由框架赋予无效分数。 + +框架会在验证器外部使用 `log2_baseline_ratio` 进行归一化。 + + +## 评测契约 + +验证器会独立重算 `Verifier-owned positive raw utility: 1.0 plus the sum over queries of business_value * population_coverage * log1p(epsilon) minus deterministic error penalties, evaluated only after all feasibility checks pass.`,候选方案需要将其最大化。 +每个有效实例按照相对基线的 `log2` 改进计分,最终取所有实例分数的平均; +无效方案得分为 `-1e18`。 + +## 评测设计 + +问题设定:`离线批量`。到达模型:All analytics-query portfolios are generated deterministically from fixed seeds before solving. A candidate receives a complete static instance containing query sensitivities, business values, population coverage, group memberships, bounds, fairness thresholds, accuracy requirements, and total budget, then returns one structured allocation for that instance. + +目标理由:The primary objective is appropriate because privacy budget is a scarce resource and the economically relevant decision is the feasible allocation that preserves the most weighted analytics utility while controlling estimation error. Accuracy and fairness are hard feasibility requirements so the objective cannot trade them away beyond accepted policy limits. + +文献对齐:The supplied brief aligns with differential-privacy budget-allocation work at the level of allocating limited privacy loss across multiple analytics queries and recomputing privacy loss and error from first principles. This benchmark differs by making the task an offline batch portfolio optimization problem with explicit business values, population coverage, fairness constraints, and a structured candidate output rather than an interactive privacy accountant or a single-query mechanism design task. + +本地执行只适用于已经审核的代码: + +```bash +python verification/evaluator.py scripts/init.py --local +``` + +正式发布评测需要 Docker 和固定摘要的运行镜像: + +```bash +docker pull python:3.12.11-slim-bookworm@sha256:519591d6871b7bc437060736b9f7456b8731f1499a57e22e6c285135ae657bf7 +python verification/evaluator.py scripts/init.py +``` + diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/Task.md b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/Task.md new file mode 100644 index 00000000..ff55c386 --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/Task.md @@ -0,0 +1,146 @@ +# Task + +You are given a batch of analytics queries and a fixed total differential-privacy budget. Allocate a nonnegative epsilon value to every query. + +Each query contains: + +- `id`: query identifier. +- `sensitivity`: sensitivity used in the deterministic error model. +- `business_value`: value weight for the query. +- `population_coverage`: covered population fraction or weight. +- `epsilon_min`: minimum allowed privacy budget. +- `epsilon_max`: maximum allowed privacy budget. +- `max_error`: maximum allowed estimation error. +- `group`: population group label used for fairness checks. + +The estimation error for a query is `sensitivity / epsilon`. Allocations must satisfy every per-query bound, every per-query maximum error, the total budget limit, and the group fairness ratio over average group errors. + +## Required Output + +Return a dictionary with one key: + +```python +{ + "allocations": { + "query_id": epsilon + } +} +``` + +The allocation map must contain exactly the required query IDs and finite numeric epsilon values. + +## Objective + +After feasibility checks pass, the verifier computes a positive raw utility. Larger values are better. The metric rewards useful analytics budget and penalizes estimation error. Normalization is handled by the benchmark framework. + + +## Input Schema + +```json +{ + "type": "object", + "additionalProperties": false, + "required": [ + "queries", + "epsilon_total", + "fairness" + ], + "properties": { + "queries": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "sensitivity", + "business_value", + "population_coverage", + "epsilon_min", + "epsilon_max", + "max_error", + "group" + ], + "properties": { + "id": { + "type": "string" + }, + "sensitivity": { + "type": "number", + "minimum": 0 + }, + "business_value": { + "type": "number", + "minimum": 0 + }, + "population_coverage": { + "type": "number", + "minimum": 0 + }, + "epsilon_min": { + "type": "number", + "minimum": 0 + }, + "epsilon_max": { + "type": "number", + "minimum": 0 + }, + "max_error": { + "type": "number", + "minimum": 0 + }, + "group": { + "type": "string" + } + } + } + }, + "epsilon_total": { + "type": "number", + "minimum": 0 + }, + "fairness": { + "type": "object", + "additionalProperties": false, + "required": [ + "max_group_error_ratio" + ], + "properties": { + "max_group_error_ratio": { + "type": "number", + "minimum": 1 + } + } + } + } +} +``` + +## Output Schema + +```json +{ + "type": "object", + "additionalProperties": false, + "required": [ + "allocations" + ], + "properties": { + "allocations": { + "type": "object", + "additionalProperties": true + } + } +} +``` + +## Constraints and Objective + +The output must satisfy every hard constraint described above. The frozen verifier independently +checks feasibility and recomputes `Verifier-owned positive raw utility: 1.0 plus the sum over queries of business_value * population_coverage * log1p(epsilon) minus deterministic error penalties, evaluated only after all feasibility checks pass.`. The objective is to maximize +that strictly positive raw metric. Valid cases use `log2` baseline improvement and are aggregated +with the mean; invalid solutions receive `-1e18`. + +The problem setting is `offline_batch`. Objective rationale: The primary objective is appropriate because privacy budget is a scarce resource and the economically relevant decision is the feasible allocation that preserves the most weighted analytics utility while controlling estimation error. Accuracy and fairness are hard feasibility requirements so the objective cannot trade them away beyond accepted policy limits. +Literature alignment: The supplied brief aligns with differential-privacy budget-allocation work at the level of allocating limited privacy loss across multiple analytics queries and recomputing privacy loss and error from first principles. This benchmark differs by making the task an offline batch portfolio optimization problem with explicit business values, population coverage, fairness constraints, and a structured candidate output rather than an interactive privacy accountant or a single-query mechanism design task. + diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/Task_zh-CN.md b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/Task_zh-CN.md new file mode 100644 index 00000000..74473afc --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/Task_zh-CN.md @@ -0,0 +1,144 @@ +# 任务 + +给定一批分析查询和固定的差分隐私总预算,请为每个查询分配一个非负 epsilon 值。 + +每个查询包含: + +- `id`:查询标识符。 +- `sensitivity`:用于确定性误差模型的敏感度。 +- `business_value`:查询价值权重。 +- `population_coverage`:覆盖人群比例或权重。 +- `epsilon_min`:允许的最小隐私预算。 +- `epsilon_max`:允许的最大隐私预算。 +- `max_error`:允许的最大估计误差。 +- `group`:用于公平性检查的人群组标签。 + +查询的估计误差为 `sensitivity / epsilon`。分配方案必须满足每个查询的上下界、每个查询的最大误差、总预算限制,以及基于组平均误差的公平性比例限制。 + +## 输出要求 + +返回一个只包含以下键的字典: + +```python +{ + "allocations": { + "query_id": epsilon + } +} +``` + +分配映射必须精确包含所有要求的查询 ID,并且 epsilon 必须是有限数值。 + +## 目标 + +通过可行性检查后,验证器会计算一个正向原始效用指标。数值越大越好。该指标奖励有用的分析预算,并惩罚估计误差。归一化由 benchmark 框架处理。 + + +## 输入 Schema + +```json +{ + "type": "object", + "additionalProperties": false, + "required": [ + "queries", + "epsilon_total", + "fairness" + ], + "properties": { + "queries": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "sensitivity", + "business_value", + "population_coverage", + "epsilon_min", + "epsilon_max", + "max_error", + "group" + ], + "properties": { + "id": { + "type": "string" + }, + "sensitivity": { + "type": "number", + "minimum": 0 + }, + "business_value": { + "type": "number", + "minimum": 0 + }, + "population_coverage": { + "type": "number", + "minimum": 0 + }, + "epsilon_min": { + "type": "number", + "minimum": 0 + }, + "epsilon_max": { + "type": "number", + "minimum": 0 + }, + "max_error": { + "type": "number", + "minimum": 0 + }, + "group": { + "type": "string" + } + } + } + }, + "epsilon_total": { + "type": "number", + "minimum": 0 + }, + "fairness": { + "type": "object", + "additionalProperties": false, + "required": [ + "max_group_error_ratio" + ], + "properties": { + "max_group_error_ratio": { + "type": "number", + "minimum": 1 + } + } + } + } +} +``` + +## 输出 Schema + +```json +{ + "type": "object", + "additionalProperties": false, + "required": [ + "allocations" + ], + "properties": { + "allocations": { + "type": "object", + "additionalProperties": true + } + } +} +``` + +## 约束与目标 + +输出必须满足上文全部硬约束。冻结验证器会独立检查可行性并重算 +`Verifier-owned positive raw utility: 1.0 plus the sum over queries of business_value * population_coverage * log1p(epsilon) minus deterministic error penalties, evaluated only after all feasibility checks pass.`;优化目标是将这个严格为正的原始指标最大化。 +有效实例使用相对基线的 `log2` 改进值,并取平均;无效方案得分为 `-1e18`。 +问题设定为 `离线批量`。目标设计理由:The primary objective is appropriate because privacy budget is a scarce resource and the economically relevant decision is the feasible allocation that preserves the most weighted analytics utility while controlling estimation error. Accuracy and fairness are hard feasibility requirements so the objective cannot trade them away beyond accepted policy limits. +文献对齐:The supplied brief aligns with differential-privacy budget-allocation work at the level of allocating limited privacy loss across multiple analytics queries and recomputing privacy loss and error from first principles. This benchmark differs by making the task an offline batch portfolio optimization problem with explicit business values, population coverage, fairness constraints, and a structured candidate output rather than an interactive privacy accountant or a single-query mechanism design task. + diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/baseline/heuristic.py b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/baseline/heuristic.py new file mode 100644 index 00000000..d6641f55 --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/baseline/heuristic.py @@ -0,0 +1,20 @@ +from __future__ import annotations + +import json +import sys +from pathlib import Path +from typing import Any + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from verification.problem import solve_baseline # noqa: E402 + + +def solve(instance: dict[str, Any]) -> dict[str, Any]: + return solve_baseline(instance) + + +if __name__ == "__main__": + json.dump(solve(json.load(sys.stdin)), sys.stdout, allow_nan=False) + sys.stdout.write("\n") diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/baseline/weak.py b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/baseline/weak.py new file mode 100644 index 00000000..87d24d4d --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/baseline/weak.py @@ -0,0 +1,20 @@ +from __future__ import annotations + +import json +import sys +from pathlib import Path +from typing import Any + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from verification.problem import solve_random # noqa: E402 + + +def solve(instance: dict[str, Any]) -> dict[str, Any]: + return solve_random(instance) + + +if __name__ == "__main__": + json.dump(solve(json.load(sys.stdin)), sys.stdout, allow_nan=False) + sys.stdout.write("\n") diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/benchmark.yaml b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/benchmark.yaml new file mode 100644 index 00000000..3f506d3f --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/benchmark.yaml @@ -0,0 +1,175 @@ +api_version: benchgen/v1 +benchmark_id: PrivacyEngineering/DifferentialPrivacyBudgetAllocation +title: Differential Privacy Budget Allocation for Business Analytics +summary: Allocate a fixed differential-privacy budget across analytics queries with + heterogeneous sensitivity, business value, population coverage, fairness requirements, + and estimation-accuracy needs. Candidate solutions return a structured per-query + budget allocation that is checked against deterministic benchmark instances generated + from fixed seeds. +archetype: structured-solution +candidate: + path: scripts/init.py + function: solve + input_schema: + type: object + additionalProperties: false + required: + - queries + - epsilon_total + - fairness + properties: + queries: + type: array + items: + type: object + additionalProperties: false + required: + - id + - sensitivity + - business_value + - population_coverage + - epsilon_min + - epsilon_max + - max_error + - group + properties: + id: + type: string + sensitivity: + type: number + minimum: 0 + business_value: + type: number + minimum: 0 + population_coverage: + type: number + minimum: 0 + epsilon_min: + type: number + minimum: 0 + epsilon_max: + type: number + minimum: 0 + max_error: + type: number + minimum: 0 + group: + type: string + epsilon_total: + type: number + minimum: 0 + fairness: + type: object + additionalProperties: false + required: + - max_group_error_ratio + properties: + max_group_error_ratio: + type: number + minimum: 1 + output_schema: + type: object + additionalProperties: false + required: + - allocations + properties: + allocations: + type: object + additionalProperties: true + timeout_s: 10.0 + max_output_bytes: 1048576 +instances: + generator_module: verification.problem + public_seeds: + - 104729 + - 130363 + - 155921 + - 181081 + evaluation_seeds: + - 210001 + - 210013 + - 210031 + - 210049 + - 210067 + - 210089 + - 210103 + - 210113 + - 210131 + - 210151 + instances_per_seed: 1 +design: + problem_setting: offline_batch + arrival_model: All analytics-query portfolios are generated deterministically from + fixed seeds before solving. A candidate receives a complete static instance containing + query sensitivities, business values, population coverage, group memberships, + bounds, fairness thresholds, accuracy requirements, and total budget, then returns + one structured allocation for that instance. + objective_rationale: The primary objective is appropriate because privacy budget + is a scarce resource and the economically relevant decision is the feasible allocation + that preserves the most weighted analytics utility while controlling estimation + error. Accuracy and fairness are hard feasibility requirements so the objective + cannot trade them away beyond accepted policy limits. + literature_alignment: The supplied brief aligns with differential-privacy budget-allocation + work at the level of allocating limited privacy loss across multiple analytics + queries and recomputing privacy loss and error from first principles. This benchmark + differs by making the task an offline batch portfolio optimization problem with + explicit business values, population coverage, fairness constraints, and a structured + candidate output rather than an interactive privacy accountant or a single-query + mechanism design task. +objective: + raw_metric: 'Verifier-owned positive raw utility: 1.0 plus the sum over queries + of business_value * population_coverage * log1p(epsilon) minus deterministic error + penalties, evaluated only after all feasibility checks pass.' + direction: maximize + aggregation: mean + normalization: log2_baseline_ratio + invalid_score: -1.0e+18 +baseline: + module: baseline.heuristic + callable: solve +reference: + module: reference.exact + callable: solve +runtime: + isolation: docker + docker_image: python:3.12.11-slim-bookworm@sha256:519591d6871b7bc437060736b9f7456b8731f1499a57e22e6c285135ae657bf7 + timeout_s: 130.0 + cpus: 1.0 + memory_mb: 512 + pids_limit: 128 + network_disabled: true +provenance: + sources: + - id: src-ae0a3959eced952d + title: 'NIST SP 800-226: Guidelines for Evaluating Differential Privacy Guarantees' + location: references/citations/src-ae0a3959eced952d.json + sha256: ae0a3959eced952d197762f882be99e4736b00517dd138e501d5280d721cfacb + license: citation-only + retrieved_at: '2026-07-30T15:35:02.425841+00:00' + origin: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-226.pdf + final_url: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-226.pdf + citation_only: true + publishable: true + - id: src-a2b0f1538ad8494f + title: The Algorithmic Foundations of Differential Privacy + location: references/citations/src-a2b0f1538ad8494f.json + sha256: a2b0f1538ad8494fab07a93d328c33c2da7575ed57ecdc7a895e1b35d8379923 + license: citation-only + retrieved_at: '2026-07-30T15:35:02.704493+00:00' + origin: https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf + final_url: https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf + citation_only: true + publishable: true + data_license: CC0-1.0 + generated_from_seeds: true +frontier: + enabled: true + domain: PrivacyEngineering + task: DifferentialPrivacyBudgetAllocation + languages: + - en + - zh-CN +calibration: + minimum_reference_score: 0.05 + deterministic_tolerance: 1.0e-12 + unified_score_tolerance: 1.0e-09 diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/data/seeds.json b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/data/seeds.json new file mode 100644 index 00000000..6458e243 --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/data/seeds.json @@ -0,0 +1,13 @@ +{ + "evaluation": { + "count": 10, + "visibility": "frozen-verifier-only" + }, + "instances_per_seed": 1, + "public": [ + 104729, + 130363, + 155921, + 181081 + ] +} diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/agent_files.txt b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/agent_files.txt new file mode 100644 index 00000000..5288d154 --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/agent_files.txt @@ -0,0 +1,5 @@ +README.md +README_zh-CN.md +Task.md +Task_zh-CN.md +scripts/init.py diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/artifact_files.txt b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/artifact_files.txt new file mode 100644 index 00000000..fb5eabb7 --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/artifact_files.txt @@ -0,0 +1,3 @@ +artifacts.json +metrics.json +outputs/*.json diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/candidate_destination.txt b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/candidate_destination.txt new file mode 100644 index 00000000..b9411b3d --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/candidate_destination.txt @@ -0,0 +1 @@ +scripts/init.py diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/constraints.txt b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/constraints.txt new file mode 100644 index 00000000..d6261120 --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/constraints.txt @@ -0,0 +1,102 @@ +Differential Privacy Budget Allocation for Business Analytics constraints: +1) Candidate file is `scripts/init.py` and must expose `solve(instance)`. +2) Candidate input must match this JSON Schema: +{ + "additionalProperties": false, + "properties": { + "epsilon_total": { + "minimum": 0, + "type": "number" + }, + "fairness": { + "additionalProperties": false, + "properties": { + "max_group_error_ratio": { + "minimum": 1, + "type": "number" + } + }, + "required": [ + "max_group_error_ratio" + ], + "type": "object" + }, + "queries": { + "items": { + "additionalProperties": false, + "properties": { + "business_value": { + "minimum": 0, + "type": "number" + }, + "epsilon_max": { + "minimum": 0, + "type": "number" + }, + "epsilon_min": { + "minimum": 0, + "type": "number" + }, + "group": { + "type": "string" + }, + "id": { + "type": "string" + }, + "max_error": { + "minimum": 0, + "type": "number" + }, + "population_coverage": { + "minimum": 0, + "type": "number" + }, + "sensitivity": { + "minimum": 0, + "type": "number" + } + }, + "required": [ + "id", + "sensitivity", + "business_value", + "population_coverage", + "epsilon_min", + "epsilon_max", + "max_error", + "group" + ], + "type": "object" + }, + "type": "array" + } + }, + "required": [ + "queries", + "epsilon_total", + "fairness" + ], + "type": "object" +} +3) Candidate output must match this JSON Schema: +{ + "additionalProperties": false, + "properties": { + "allocations": { + "additionalProperties": true, + "type": "object" + } + }, + "required": [ + "allocations" + ], + "type": "object" +} +4) Every hard constraint is recomputed by the frozen verifier. +5) Candidate timeout is 10 seconds per instance. +6) Candidate output is limited to 1048576 bytes. +7) Read-only benchmark assets include verification/, baseline/, reference/, data/, and references/. +8) Publish evaluation requires Docker image `python:3.12.11-slim-bookworm@sha256:519591d6871b7bc437060736b9f7456b8731f1499a57e22e6c285135ae657bf7`; candidate and verifier + run in separate network-disabled, non-root containers. +9) Frontier unified must use its process isolation mode because the benchmark evaluator owns the + inner candidate/verifier containers; do not wrap this evaluator in another Docker runtime. diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/copy_files.txt b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/copy_files.txt new file mode 100644 index 00000000..9c558e35 --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/copy_files.txt @@ -0,0 +1 @@ +. diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/eval_command.txt b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/eval_command.txt new file mode 100644 index 00000000..613443e7 --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/eval_command.txt @@ -0,0 +1 @@ +{python} {benchmark}/verification/evaluator.py {candidate} diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/eval_cwd.txt b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/eval_cwd.txt new file mode 100644 index 00000000..9c558e35 --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/eval_cwd.txt @@ -0,0 +1 @@ +. diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/initial_program.txt b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/initial_program.txt new file mode 100644 index 00000000..b9411b3d --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/initial_program.txt @@ -0,0 +1 @@ +scripts/init.py diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/readonly_files.txt b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/readonly_files.txt new file mode 100644 index 00000000..ab2c6d75 --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/readonly_files.txt @@ -0,0 +1,8 @@ +baseline/ +data/ +reference/ +references/ +verification/docker/ +verification/evaluator.py +verification/problem.py +verification/process_runner.py diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/reference/exact.py b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/reference/exact.py new file mode 100644 index 00000000..191b5705 --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/reference/exact.py @@ -0,0 +1,20 @@ +from __future__ import annotations + +import json +import sys +from pathlib import Path +from typing import Any + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from verification.problem import solve_reference # noqa: E402 + + +def solve(instance: dict[str, Any]) -> dict[str, Any]: + return solve_reference(instance) + + +if __name__ == "__main__": + json.dump(solve(json.load(sys.stdin)), sys.stdout, allow_nan=False) + sys.stdout.write("\n") diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/references/citations/src-a2b0f1538ad8494f.json b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/references/citations/src-a2b0f1538ad8494f.json new file mode 100644 index 00000000..32b1eb38 --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/references/citations/src-a2b0f1538ad8494f.json @@ -0,0 +1,12 @@ +{ + "citation_only": true, + "final_url": "https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf", + "id": "src-a2b0f1538ad8494f", + "license": "citation-only", + "location": "sources/src-a2b0f1538ad8494f.raw", + "origin": "https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf", + "publishable": true, + "retrieved_at": "2026-07-30T15:35:02.704493+00:00", + "sha256": "a2b0f1538ad8494fab07a93d328c33c2da7575ed57ecdc7a895e1b35d8379923", + "title": "The Algorithmic Foundations of Differential Privacy" +} diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/references/citations/src-ae0a3959eced952d.json b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/references/citations/src-ae0a3959eced952d.json new file mode 100644 index 00000000..8d7e25c1 --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/references/citations/src-ae0a3959eced952d.json @@ -0,0 +1,12 @@ +{ + "citation_only": true, + "final_url": "https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-226.pdf", + "id": "src-ae0a3959eced952d", + "license": "citation-only", + "location": "sources/src-ae0a3959eced952d.raw", + "origin": "https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-226.pdf", + "publishable": true, + "retrieved_at": "2026-07-30T15:35:02.425841+00:00", + "sha256": "ae0a3959eced952d197762f882be99e4736b00517dd138e501d5280d721cfacb", + "title": "NIST SP 800-226: Guidelines for Evaluating Differential Privacy Guarantees" +} diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/references/provenance.json b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/references/provenance.json new file mode 100644 index 00000000..519991d9 --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/references/provenance.json @@ -0,0 +1,31 @@ +{ + "benchmark_id": "PrivacyEngineering/DifferentialPrivacyBudgetAllocation", + "data_license": "CC0-1.0", + "generated_from_seeds": true, + "sources": [ + { + "citation_only": true, + "final_url": "https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-226.pdf", + "id": "src-ae0a3959eced952d", + "license": "citation-only", + "location": "references/citations/src-ae0a3959eced952d.json", + "origin": "https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-226.pdf", + "publishable": true, + "retrieved_at": "2026-07-30T15:35:02.425841+00:00", + "sha256": "ae0a3959eced952d197762f882be99e4736b00517dd138e501d5280d721cfacb", + "title": "NIST SP 800-226: Guidelines for Evaluating Differential Privacy Guarantees" + }, + { + "citation_only": true, + "final_url": "https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf", + "id": "src-a2b0f1538ad8494f", + "license": "citation-only", + "location": "references/citations/src-a2b0f1538ad8494f.json", + "origin": "https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf", + "publishable": true, + "retrieved_at": "2026-07-30T15:35:02.704493+00:00", + "sha256": "a2b0f1538ad8494fab07a93d328c33c2da7575ed57ecdc7a895e1b35d8379923", + "title": "The Algorithmic Foundations of Differential Privacy" + } + ] +} diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/scripts/init.py b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/scripts/init.py new file mode 100644 index 00000000..27967d4f --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/scripts/init.py @@ -0,0 +1,96 @@ +from __future__ import annotations + +import json +import sys +from typing import Any + + +# EVOLVE-BLOCK-START +def solve(instance): + queries = list(instance['queries']) + allocations = {q['id']: float(q['epsilon_min']) for q in queries} + remaining = float(instance['epsilon_total']) - sum(allocations.values()) + if remaining <= 0: + return {'allocations': allocations} + + def error(q, eps): + return float(q['sensitivity']) / float(eps) + + def group_averages(): + totals = {} + counts = {} + for q in queries: + group = q['group'] + totals[group] = totals.get(group, 0.0) + error(q, allocations[q['id']]) + counts[group] = counts.get(group, 0) + 1 + return {group: totals[group] / counts[group] for group in totals} + + def fairness_ok(): + averages = list(group_averages().values()) + if not averages: + return True + smallest = min(averages) + if smallest <= 0.0: + return False + return max(averages) / smallest <= float(instance['fairness']['max_group_error_ratio']) + 1e-09 + while remaining > 1e-12 and (not fairness_ok()): + averages = group_averages() + worst_group = max(averages, key=averages.get) + candidates = [] + for q in queries: + qid = q['id'] + if q['group'] != worst_group: + continue + cap = float(q['epsilon_max']) - allocations[qid] + if cap > 1e-12: + reduction = float(q['sensitivity']) / (allocations[qid] * allocations[qid]) + candidates.append((reduction, qid, cap)) + if not candidates: + break + _reduction, qid, cap = max(candidates) + add = min(0.01, remaining, cap) + allocations[qid] += add + remaining -= add + remaining = min(remaining, 0.25 * float(instance['epsilon_total'])) + rounds = 0 + while remaining > 1e-12 and rounds < 10000: + used = 0.0 + for q in sorted(queries, key=lambda x: x['id']): + qid = q['id'] + cap = float(q['epsilon_max']) - allocations[qid] + if cap <= 1e-12: + continue + add = min(0.02, cap, remaining) + allocations[qid] += add + if fairness_ok(): + remaining -= add + used += add + else: + allocations[qid] -= add + if remaining <= 1e-12: + break + if used <= 1e-12: + break + rounds += 1 + return {'allocations': allocations} +# EVOLVE-BLOCK-END + + +def main() -> int: + try: + instance = json.load(sys.stdin) + if not isinstance(instance, dict): + raise TypeError("input must be a JSON object") + solution = solve(instance) + if not isinstance(solution, dict): + raise TypeError("solve() must return a JSON object") + json.dump(solution, sys.stdout, allow_nan=False, separators=(",", ":")) + sys.stdout.write("\n") + return 0 + except Exception as exc: + print(f"candidate error: {type(exc).__name__}: {exc}", file=sys.stderr) + return 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/verification/docker/Dockerfile b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/verification/docker/Dockerfile new file mode 100644 index 00000000..9512d042 --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/verification/docker/Dockerfile @@ -0,0 +1,13 @@ +FROM python:3.12.11-slim-bookworm@sha256:519591d6871b7bc437060736b9f7456b8731f1499a57e22e6c285135ae657bf7 + +ENV PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 \ + PYTHONHASHSEED=0 + +RUN groupadd --gid 65532 benchgen \ + && useradd --uid 65532 --gid 65532 --no-create-home --shell /usr/sbin/nologin benchgen + +WORKDIR /workspace +USER 65532:65532 + +CMD ["python", "--version"] diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/verification/evaluator.py b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/verification/evaluator.py new file mode 100644 index 00000000..958712ab --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/verification/evaluator.py @@ -0,0 +1,681 @@ +from __future__ import annotations + +import argparse +import copy +import importlib.util +import json +import math +import os +import statistics +import sys +import time +import uuid +from pathlib import Path +from types import ModuleType +from typing import Any + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from verification.process_runner import BoundedResult, run_bounded # noqa: E402 + +SEEDS = [210001, 210013, 210031, 210049, 210067, 210089, 210103, 210113, 210131, 210151] +SMOKE_SEED = 4161752734 +INSTANCES_PER_SEED = 1 +INPUT_SCHEMA = json.loads( + "{\"additionalProperties\": false, \"propert" + "ies\": {\"epsilon_total\": {\"minimum\": 0, \"" + "type\": \"number\"}, \"fairness\": {\"addition" + "alProperties\": false, \"properties\": {\"ma" + "x_group_error_ratio\": {\"minimum\": 1, \"ty" + "pe\": \"number\"}}, \"required\": [\"max_group" + "_error_ratio\"], \"type\": \"object\"}, \"quer" + "ies\": {\"items\": {\"additionalProperties\":" + " false, \"properties\": {\"business_value\":" + " {\"minimum\": 0, \"type\": \"number\"}, \"epsi" + "lon_max\": {\"minimum\": 0, \"type\": \"number" + "\"}, \"epsilon_min\": {\"minimum\": 0, \"type\"" + ": \"number\"}, \"group\": {\"type\": \"string\"}" + ", \"id\": {\"type\": \"string\"}, \"max_error\":" + " {\"minimum\": 0, \"type\": \"number\"}, \"popu" + "lation_coverage\": {\"minimum\": 0, \"type\":" + " \"number\"}, \"sensitivity\": {\"minimum\": 0" + ", \"type\": \"number\"}}, \"required\": [\"id\"," + " \"sensitivity\", \"business_value\", \"popul" + "ation_coverage\", \"epsilon_min\", \"epsilon" + "_max\", \"max_error\", \"group\"], \"type\": \"o" + "bject\"}, \"type\": \"array\"}}, \"required\": " + "[\"queries\", \"epsilon_total\", \"fairness\"]" + ", \"type\": \"object\"}" +) +OUTPUT_SCHEMA = json.loads( + "{\"additionalProperties\": false, \"propert" + "ies\": {\"allocations\": {\"additionalProper" + "ties\": true, \"type\": \"object\"}}, \"requir" + "ed\": [\"allocations\"], \"type\": \"object\"}" +) +DIRECTION = "maximize" +AGGREGATION = "mean" +INVALID_SCORE = -1e+18 +TIMEOUT_S = 10.0 +MAX_OUTPUT_BYTES = 1048576 +RUNTIME_IMAGE = ( + "python:3.12.11-slim-bookworm@sha256:519591d6871b7bc437060736b9f7456b87" + "31f1499a57e22e6c285135ae657bf7" +) +MEMORY_MB = 512 +CPUS = 1.0 +PIDS_LIMIT = 128 + + +def _load_module(relative_module: str) -> ModuleType: + path = ROOT / (relative_module.replace(".", "/") + ".py") + module_spec = importlib.util.spec_from_file_location("benchgen_task_module", path) + if module_spec is None or module_spec.loader is None: + raise RuntimeError(f"cannot load task module: {path}") + module = importlib.util.module_from_spec(module_spec) + module_spec.loader.exec_module(module) + return module + + +def _valid( + problem: ModuleType, instance: dict[str, Any], solution: dict[str, Any] +) -> tuple[bool, str]: + result = problem.validate_solution(instance, solution) + if result is None: + return True, "" + if isinstance(result, bool): + return result, "" if result else "solution rejected by verifier" + if isinstance(result, tuple) and len(result) == 2: + return bool(result[0]), str(result[1]) + if hasattr(result, "valid"): + return bool(result.valid), str(getattr(result, "reason", "")) + raise TypeError("invalid validate_solution return value") + + +def _finite(value: Any) -> bool: + if isinstance(value, float): + return math.isfinite(value) + if isinstance(value, dict): + return all(_finite(key) and _finite(item) for key, item in value.items()) + if isinstance(value, list): + return all(_finite(item) for item in value) + return True + + +def _validate_json_schema(value: Any, schema: dict[str, Any], path: str = "$") -> list[str]: + """Validate the dependency-free JSON Schema subset supported by benchgen v1.""" + + errors: list[str] = [] + expected_type = schema.get("type") + if expected_type and not _matches_type(value, expected_type): + return [f"{path}: expected {expected_type}, got {type(value).__name__}"] + if "enum" in schema and not any(_json_equal(value, item) for item in schema["enum"]): + errors.append(f"{path}: value is not in enum") + + if isinstance(value, dict): + properties = schema.get("properties", {}) + required = schema.get("required", []) + for key in required: + if key not in value: + errors.append(f"{path}: missing required property {key!r}") + for key, item in value.items(): + if key in properties: + errors.extend(_validate_json_schema(item, properties[key], f"{path}.{key}")) + elif schema.get("additionalProperties") is False: + errors.append(f"{path}: unexpected property {key!r}") + elif isinstance(value, list): + if "minItems" in schema and len(value) < schema["minItems"]: + errors.append(f"{path}: fewer than minItems") + if "maxItems" in schema and len(value) > schema["maxItems"]: + errors.append(f"{path}: more than maxItems") + if schema.get("uniqueItems"): + canonical = [_json_key(item) for item in value] + if len(canonical) != len(set(canonical)): + errors.append(f"{path}: items are not unique") + if isinstance(schema.get("items"), dict): + for index, item in enumerate(value): + errors.extend(_validate_json_schema(item, schema["items"], f"{path}[{index}]")) + elif isinstance(value, str): + if "minLength" in schema and len(value) < schema["minLength"]: + errors.append(f"{path}: shorter than minLength") + if "maxLength" in schema and len(value) > schema["maxLength"]: + errors.append(f"{path}: longer than maxLength") + elif isinstance(value, (int, float)) and not isinstance(value, bool): + if "minimum" in schema and value < schema["minimum"]: + errors.append(f"{path}: below minimum") + if "maximum" in schema and value > schema["maximum"]: + errors.append(f"{path}: above maximum") + return errors + + +def _json_equal(left: Any, right: Any) -> bool: + return _json_key(left) == _json_key(right) + + +def _json_key(value: Any) -> Any: + if value is None: + return ("null",) + if isinstance(value, bool): + return ("boolean", value) + if isinstance(value, (int, float)): + return ("number", value) + if isinstance(value, str): + return ("string", value) + if isinstance(value, list): + return ("array", tuple(_json_key(item) for item in value)) + if isinstance(value, dict): + return ( + "object", + tuple(sorted((str(key), _json_key(item)) for key, item in value.items())), + ) + return (type(value).__name__, repr(value)) + + +def _matches_type(value: Any, expected: str | list[str]) -> bool: + if isinstance(expected, list): + return any(_matches_type(value, item) for item in expected) + checks = { + "null": lambda item: item is None, + "boolean": lambda item: isinstance(item, bool), + "object": lambda item: isinstance(item, dict), + "array": lambda item: isinstance(item, list), + "string": lambda item: isinstance(item, str), + "integer": lambda item: isinstance(item, int) and not isinstance(item, bool), + "number": lambda item: isinstance(item, (int, float)) and not isinstance(item, bool), + } + return expected in checks and checks[expected](value) + + +def _require_schema(value: Any, schema: dict[str, Any], label: str) -> None: + errors = _validate_json_schema(value, schema) + if errors: + raise ValueError(f"{label} violates JSON Schema: {'; '.join(errors[:5])}") + + +def _metric_value(value: Any, label: str) -> float: + if isinstance(value, bool) or not isinstance(value, (int, float)): + raise TypeError(f"{label} must be an int or float, got {type(value).__name__}") + metric = float(value) + if not math.isfinite(metric): + raise ValueError(f"{label} must be finite") + if metric <= 0: + raise ValueError(f"{label} must be strictly positive") + return metric + + +def _raw_metric( + problem: ModuleType, + instance: dict[str, Any], + solution: dict[str, Any], + label: str, +) -> float: + value = problem.evaluate_solution(instance, solution) + return _metric_value(value, f"{label} evaluate_solution result") + + +def _canonical_json(value: Any, label: str) -> str: + try: + return json.dumps(value, sort_keys=True, separators=(",", ":"), allow_nan=False) + except (TypeError, ValueError) as exc: + raise TypeError(f"{label} must be JSON serializable: {exc}") from exc + + +def _smoke_instances(problem: ModuleType) -> list[dict[str, Any]]: + generated = problem.generate_instances(SMOKE_SEED) + instances = [generated] if isinstance(generated, dict) else list(generated) + if len(instances) != INSTANCES_PER_SEED: + raise RuntimeError( + f"smoke generator returned {len(instances)} instances; " + f"expected exactly {INSTANCES_PER_SEED}" + ) + checked: list[dict[str, Any]] = [] + for instance in instances: + if not isinstance(instance, dict) or not _finite(instance): + raise TypeError("smoke instance must be a finite JSON object") + _require_schema(instance, INPUT_SCHEMA, "smoke instance") + _canonical_json(instance, "smoke instance") + checked.append(instance) + return checked + + +def _smoke_solver( + problem: ModuleType, + solver: Any, + instance: dict[str, Any], + label: str, +) -> tuple[dict[str, Any], float]: + solution = solver(copy.deepcopy(instance)) + if not isinstance(solution, dict) or not _finite(solution): + raise TypeError(f"{label} output must be a finite JSON object") + _require_schema(solution, OUTPUT_SCHEMA, f"{label} output") + _canonical_json(solution, f"{label} output") + valid, reason = _valid(problem, copy.deepcopy(instance), copy.deepcopy(solution)) + if not valid: + raise ValueError(f"{label} output is invalid: {reason or 'solution rejected'}") + metric = _raw_metric( + problem, + copy.deepcopy(instance), + copy.deepcopy(solution), + label, + ) + return solution, metric + + +def _smoke_payload() -> dict[str, Any]: + problem = _load_module("verification.problem") + first_instances = _smoke_instances(problem) + second_instances = _smoke_instances(problem) + if _canonical_json(first_instances, "generated instances") != _canonical_json( + second_instances, "repeated generated instances" + ): + raise ValueError("generate_instances is not deterministic") + + solvers = ( + ("random solver", problem.solve_random), + ("baseline solver", problem.solve_baseline), + ("reference solver", problem.solve_reference), + ) + solver_runs = 0 + for label, solver in solvers: + for instance in first_instances: + first_solution, first_metric = _smoke_solver(problem, solver, instance, label) + second_solution, second_metric = _smoke_solver(problem, solver, instance, label) + solver_runs += 2 + if _canonical_json(first_solution, f"{label} output") != _canonical_json( + second_solution, f"repeated {label} output" + ): + raise ValueError(f"{label} is not deterministic") + if first_metric != second_metric: + raise ValueError(f"{label} metric is not deterministic") + return { + "ok": True, + "instances_checked": len(first_instances), + "solver_runs": solver_runs, + "solvers_checked": [label for label, _solver in solvers], + } + + +def _score(baseline: float, candidate: float) -> float: + if not math.isfinite(baseline) or not math.isfinite(candidate): + raise ValueError("objective metrics must be finite") + if baseline <= 0 or candidate <= 0: + raise ValueError("log2 normalization requires positive metrics") + ratio = baseline / candidate if DIRECTION == "minimize" else candidate / baseline + return math.log2(ratio) + + +def _prepare_payload() -> dict[str, Any]: + problem = _load_module("verification.problem") + baseline_module = _load_module("baseline.heuristic") + baseline_solver = getattr(baseline_module, "solve") # noqa: B009 + cases: list[dict[str, Any]] = [] + for seed in SEEDS: + generated = problem.generate_instances(seed) + instances = [generated] if isinstance(generated, dict) else list(generated) + if len(instances) != INSTANCES_PER_SEED: + raise RuntimeError( + f"seed {seed} generated {len(instances)} instances; " + f"expected exactly {INSTANCES_PER_SEED}" + ) + for index, instance in enumerate(instances): + if not isinstance(instance, dict) or not _finite(instance): + raise TypeError("generated instance must be a finite JSON object") + _require_schema(instance, INPUT_SCHEMA, "generated instance") + baseline_solution = baseline_solver(instance) + if not isinstance(baseline_solution, dict) or not _finite(baseline_solution): + raise TypeError("baseline output must be a finite JSON object") + _require_schema(baseline_solution, OUTPUT_SCHEMA, "baseline output") + valid, reason = _valid(problem, instance, baseline_solution) + if not valid: + raise RuntimeError(f"invalid benchmark baseline: {reason}") + baseline_metric = _raw_metric( + problem, instance, baseline_solution, "baseline" + ) + cases.append( + { + "case_id": f"{seed}:{index}", + "seed": seed, + "index": index, + "instance": instance, + "baseline_metric": baseline_metric, + } + ) + return {"cases": cases} + + +def _score_payload(payload: dict[str, Any]) -> dict[str, Any]: + problem = _load_module("verification.problem") + prepared = payload.get("prepared", {}) + cases = prepared.get("cases", []) if isinstance(prepared, dict) else [] + submissions = payload.get("submissions", []) + if not isinstance(cases, list) or not isinstance(submissions, list): + raise TypeError("score payload must contain cases and submissions lists") + if len(cases) != len(submissions): + raise ValueError("submission count does not match prepared case count") + records: list[dict[str, Any]] = [] + for case, submission in zip(cases, submissions, strict=True): + started = time.monotonic() + record = {"case_index": len(records)} + try: + if submission.get("case_id") != case.get("case_id"): + raise ValueError("submission case id mismatch") + if submission.get("runner_error"): + raise RuntimeError(str(submission["runner_error"])) + solution = submission.get("solution") + if not isinstance(solution, dict) or not _finite(solution): + raise TypeError("candidate output must be a finite JSON object") + _require_schema(solution, OUTPUT_SCHEMA, "candidate output") + instance = case["instance"] + if not isinstance(instance, dict) or not _finite(instance): + raise TypeError("prepared instance must be a finite JSON object") + _require_schema(instance, INPUT_SCHEMA, "prepared instance") + valid, reason = _valid(problem, instance, solution) + if not valid: + raise ValueError(reason or "invalid candidate solution") + baseline_metric = _metric_value(case["baseline_metric"], "prepared baseline metric") + candidate_metric = _raw_metric(problem, instance, solution, "candidate") + record.update( + valid=True, + score=_score(baseline_metric, candidate_metric), + baseline_metric=baseline_metric, + candidate_metric=candidate_metric, + ) + except Exception as exc: + record.update(valid=False, score=INVALID_SCORE, reason=f"{type(exc).__name__}: {exc}") + record["runtime_s"] = time.monotonic() - started + records.append(record) + all_valid = bool(records) and all(record["valid"] for record in records) + scores = [float(record["score"]) for record in records] + combined = ( + (statistics.fmean(scores) if AGGREGATION == "mean" else statistics.median(scores)) + if all_valid + else INVALID_SCORE + ) + metrics = { + "valid": 1.0 if all_valid else 0.0, + "combined_score": combined, + "instances_total": len(records), + "instances_valid": sum(bool(record["valid"]) for record in records), + "runtime_s": sum(float(record["runtime_s"]) for record in records), + } + return {"metrics": metrics, "artifacts": {"instance_results": records}} + + +def _parse_json_result(result: BoundedResult, label: str) -> dict[str, Any]: + if result.timed_out: + raise TimeoutError(f"{label} timed out") + if result.output_truncated: + raise ValueError(f"{label} exceeded output limit") + if result.returncode != 0: + raise RuntimeError(f"{label} exited with {result.returncode}: {result.stderr[-2000:]}") + value = json.loads(result.stdout) + if not isinstance(value, dict) or not _finite(value): + raise TypeError(f"{label} output must be a finite JSON object") + return value + + +def _candidate_local( + candidate: Path, + instance: dict[str, Any], + *, + timeout_s: float = TIMEOUT_S, + max_output_bytes: int = MAX_OUTPUT_BYTES, +) -> dict[str, Any]: + result = run_bounded( + [sys.executable, str(candidate)], + json.dumps(instance, allow_nan=False), + timeout_s=timeout_s, + max_output_bytes=max_output_bytes, + ) + return _parse_json_result(result, "candidate") + + +def _docker_base(name: str) -> list[str]: + return [ + "docker", + "run", + "--rm", + "--interactive", + "--name", + name, + "--network", + "none", + "--read-only", + "--user", + "65532:65532", + "--cap-drop", + "ALL", + "--security-opt", + "no-new-privileges", + "--pids-limit", + str(PIDS_LIMIT), + "--memory", + f"{MEMORY_MB}m", + "--cpus", + str(CPUS), + "--tmpfs", + "/tmp:rw,noexec,nosuid,nodev,size=64m", + ] + + +def _docker_json( + command: list[str], + input_payload: dict[str, Any] | None, + *, + timeout_s: float, + label: str, + max_output_bytes: int = 2_000_000, +) -> dict[str, Any]: + name = f"benchgen-{label}-{uuid.uuid4().hex[:12]}" + full_command = _docker_base(name) + command + result = run_bounded( + full_command, + json.dumps(input_payload, allow_nan=False) if input_payload is not None else "", + timeout_s=timeout_s, + max_output_bytes=max_output_bytes, + ) + if result.timed_out: + run_bounded( + ["docker", "rm", "--force", name], + "", + timeout_s=10, + max_output_bytes=10_000, + ) + return _parse_json_result(result, label) + + +def _verifier_container(mode: str, payload: dict[str, Any] | None = None) -> dict[str, Any]: + if "," in str(ROOT) or "\n" in str(ROOT): + raise ValueError("benchmark path contains unsupported Docker mount characters") + command: list[str] = [] + for relative in ("verification", "baseline", "reference", "data", "references"): + source = ROOT / relative + if source.exists(): + command.extend( + ( + "--mount", + f"type=bind,src={source},dst=/workspace/benchmark/{relative},readonly", + ) + ) + command.extend( + ( + "--workdir", + "/workspace/benchmark", + RUNTIME_IMAGE, + "python", + "/workspace/benchmark/verification/evaluator.py", + mode, + ) + ) + return _docker_json(command, payload, timeout_s=130.0, label="verifier") + + +def _candidate_container( + candidate: Path, + case: dict[str, Any], + *, + timeout_s: float = TIMEOUT_S, + max_output_bytes: int = MAX_OUTPUT_BYTES, +) -> dict[str, Any]: + if "," in str(candidate) or "\n" in str(candidate): + raise ValueError("candidate path contains unsupported Docker mount characters") + trusted_solver = False + try: + relative = candidate.relative_to(ROOT) + trusted_solver = bool(relative.parts and relative.parts[0] in {"baseline", "reference"}) + except ValueError: + relative = Path(candidate.name) + if trusted_solver: + command = [ + "--mount", + f"type=bind,src={ROOT},dst=/workspace/benchmark,readonly", + "--workdir", + "/workspace/benchmark", + RUNTIME_IMAGE, + "python", + f"/workspace/benchmark/{relative.as_posix()}", + ] + else: + command = [ + "--mount", + f"type=bind,src={candidate},dst=/workspace/candidate.py,readonly", + "--workdir", + "/tmp", + RUNTIME_IMAGE, + "python", + "/workspace/candidate.py", + ] + return _docker_json( + command, + case["instance"], + timeout_s=timeout_s, + label="candidate", + max_output_bytes=max_output_bytes, + ) + + +def evaluate( + candidate: Path, + *, + local: bool = False, + candidate_timeout_s: float = TIMEOUT_S, + candidate_max_output_bytes: int = MAX_OUTPUT_BYTES, +) -> tuple[dict[str, Any], dict[str, Any]]: + candidate = candidate.resolve() + if not candidate.is_file(): + raise FileNotFoundError(f"candidate not found: {candidate}") + if local: + prepared = _prepare_payload() + submissions = [] + for case in prepared["cases"]: + try: + solution = _candidate_local( + candidate, + case["instance"], + timeout_s=candidate_timeout_s, + max_output_bytes=candidate_max_output_bytes, + ) + submissions.append({"case_id": case["case_id"], "solution": solution}) + except Exception as exc: + submissions.append( + {"case_id": case["case_id"], "runner_error": f"{type(exc).__name__}: {exc}"} + ) + result = _score_payload({"prepared": prepared, "submissions": submissions}) + else: + prepared = _verifier_container("--prepare") + submissions = [] + for case in prepared["cases"]: + try: + solution = _candidate_container( + candidate, + case, + timeout_s=candidate_timeout_s, + max_output_bytes=candidate_max_output_bytes, + ) + submissions.append({"case_id": case["case_id"], "solution": solution}) + except Exception as exc: + submissions.append( + {"case_id": case["case_id"], "runner_error": f"{type(exc).__name__}: {exc}"} + ) + result = _verifier_container( + "--score", {"prepared": prepared, "submissions": submissions} + ) + return result["metrics"], result["artifacts"] + + +def _atomic_json(path: Path, payload: dict[str, Any]) -> None: + from tempfile import NamedTemporaryFile + + path.parent.mkdir(parents=True, exist_ok=True) + with NamedTemporaryFile("w", encoding="utf-8", dir=path.parent, delete=False) as handle: + json.dump(payload, handle, indent=2, sort_keys=True, allow_nan=False) + handle.write("\n") + handle.flush() + os.fsync(handle.fileno()) + temporary = Path(handle.name) + temporary.replace(path) + + +def main() -> int: + if len(sys.argv) == 2 and sys.argv[1] == "--smoke": + try: + payload = _verifier_container("--smoke-local") + except Exception as exc: + message = f"{type(exc).__name__}: {exc}" + print(json.dumps({"ok": False, "error": message}, sort_keys=True)) + print(message, file=sys.stderr) + return 1 + print(json.dumps(payload, sort_keys=True, allow_nan=False)) + return 0 + if len(sys.argv) == 2 and sys.argv[1] == "--smoke-local": + try: + payload = _smoke_payload() + except Exception as exc: + message = f"{type(exc).__name__}: {exc}" + print(json.dumps({"ok": False, "error": message}, sort_keys=True)) + print(message, file=sys.stderr) + return 1 + print(json.dumps(payload, sort_keys=True, allow_nan=False)) + return 0 + if len(sys.argv) == 2 and sys.argv[1] == "--prepare": + print(json.dumps(_prepare_payload(), sort_keys=True, allow_nan=False)) + return 0 + if len(sys.argv) == 2 and sys.argv[1] == "--score": + payload = json.load(sys.stdin) + print(json.dumps(_score_payload(payload), sort_keys=True, allow_nan=False)) + return 0 + parser = argparse.ArgumentParser(description="Evaluate one structured-solution candidate.") + parser.add_argument("candidate", type=Path) + parser.add_argument("--local", action="store_true") + parser.add_argument( + "--_benchgen-probe-timeout-s", + type=float, + default=TIMEOUT_S, + help=argparse.SUPPRESS, + ) + parser.add_argument( + "--_benchgen-probe-max-output-bytes", + type=int, + default=MAX_OUTPUT_BYTES, + help=argparse.SUPPRESS, + ) + arguments = parser.parse_args() + if not 0 < arguments._benchgen_probe_timeout_s <= TIMEOUT_S: + parser.error("probe timeout must be positive and cannot relax the benchmark limit") + if not 0 < arguments._benchgen_probe_max_output_bytes <= MAX_OUTPUT_BYTES: + parser.error("probe output limit must be positive and cannot relax the benchmark limit") + metrics, artifacts = evaluate( + arguments.candidate, + local=arguments.local, + candidate_timeout_s=arguments._benchgen_probe_timeout_s, + candidate_max_output_bytes=arguments._benchgen_probe_max_output_bytes, + ) + _atomic_json(Path.cwd() / "metrics.json", metrics) + _atomic_json(Path.cwd() / "artifacts.json", artifacts) + print(json.dumps(metrics, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/verification/problem.py b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/verification/problem.py new file mode 100644 index 00000000..17a2af55 --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/verification/problem.py @@ -0,0 +1,262 @@ +import math +import random + + +def generate_instances(seed): + rng = random.Random(seed) + n = 14 + seed % 5 + groups = ["consumer", "merchant", "enterprise"] + queries = [] + eps_min_sum = 0.0 + for i in range(n): + group = groups[i % len(groups)] + tier = i % 4 + sensitivity = round(0.35 + 0.18 * tier + rng.random() * 0.22, 6) + if i % 6 == 0: + sensitivity = round(sensitivity + 0.75, 6) + business_value = round(5.0 + rng.random() * 15.0 + (4.0 if i % 6 == 0 else 0.0), 6) + population_coverage = round(0.18 + rng.random() * 0.72, 6) + epsilon_min = round(0.055 + 0.018 * (i % 3), 6) + epsilon_max = round(epsilon_min + 0.62 + rng.random() * 0.72, 6) + max_error = round(sensitivity / epsilon_min * 1.0001, 6) + eps_min_sum += epsilon_min + queries.append({ + "id": "q_%02d" % i, + "sensitivity": sensitivity, + "business_value": business_value, + "population_coverage": population_coverage, + "epsilon_min": epsilon_min, + "epsilon_max": epsilon_max, + "max_error": max_error, + "group": group, + }) + extra = 0.47 * n + rng.random() * 0.35 + max_extra = sum(q["epsilon_max"] - q["epsilon_min"] for q in queries) + epsilon_total = round(eps_min_sum + min(extra, max_extra * 0.72), 6) + return [{ + "queries": queries, + "epsilon_total": epsilon_total, + "fairness": {"max_group_error_ratio": 2.35}, + }] + + +def _error(q, eps): + return float(q["sensitivity"]) / float(eps) + + +def _objective(instance, allocations): + total = 1.0 + for q in instance["queries"]: + eps = float(allocations[q["id"]]) + value_weight = float(q["business_value"]) * float(q["population_coverage"]) + sensitivity = float(q["sensitivity"]) + error = sensitivity / eps + total += value_weight * math.log1p(2.4 * eps) - 0.18 * value_weight * error / (1.0 + sensitivity) + if total <= 0.0: + total = 1e-12 + return float(total) + + +def _group_average_errors(instance, allocations): + group_errors = {} + group_counts = {} + for q in instance["queries"]: + group = q["group"] + group_errors[group] = group_errors.get(group, 0.0) + _error(q, allocations[q["id"]]) + group_counts[group] = group_counts.get(group, 0) + 1 + return {group: group_errors[group] / group_counts[group] for group in group_errors} + + +def _fairness_ratio(instance, allocations): + avgs = list(_group_average_errors(instance, allocations).values()) + if not avgs: + return 1.0 + smallest = min(avgs) + if smallest <= 0.0: + return float("inf") + return max(avgs) / smallest + + +def _repair_fairness(instance, allocations, remaining, step=0.01): + queries = list(instance["queries"]) + q_by_id = {q["id"]: q for q in queries} + limit = float(instance["fairness"]["max_group_error_ratio"]) + iterations = 0 + while remaining > 1e-12 and _fairness_ratio(instance, allocations) > limit + 1e-9: + averages = _group_average_errors(instance, allocations) + if not averages: + break + worst_group = max(averages, key=averages.get) + candidates = [] + for q in queries: + qid = q["id"] + if q["group"] != worst_group: + continue + cap = float(q["epsilon_max"]) - allocations[qid] + if cap <= 1e-12: + continue + # d(sensitivity / epsilon) / d epsilon = -sensitivity / epsilon^2. + reduction = float(q["sensitivity"]) / (allocations[qid] * allocations[qid]) + candidates.append((reduction, qid, cap)) + if not candidates: + break + _reduction, qid, cap = max(candidates) + add = min(step, remaining, cap) + allocations[qid] += add + remaining -= add + iterations += 1 + if iterations > 10000: + break + return remaining + + +def _allocate_feasible_greedy(instance, allocations, remaining, score_delta, step=0.01): + queries = list(instance["queries"]) + while remaining > 1e-12: + best_q = None + best_gain = -1e100 + for q in queries: + qid = q["id"] + cap = float(q["epsilon_max"]) - allocations[qid] + if cap <= 1e-12: + continue + delta = min(step, remaining, cap) + before = allocations[qid] + allocations[qid] = before + delta + ok, _ = validate_solution(instance, {"allocations": dict(allocations)}) + allocations[qid] = before + if not ok: + continue + gain = score_delta(q, allocations, delta) + if gain > best_gain: + best_gain = gain + best_q = q + if best_q is None: + break + qid = best_q["id"] + add = min(step, remaining, float(best_q["epsilon_max"]) - allocations[qid]) + if add <= 1e-12: + break + allocations[qid] += add + remaining -= add + return remaining + + +def validate_solution(instance, solution): + if not isinstance(solution, dict) or set(solution.keys()) != {"allocations"}: + return False, "solution must be a dict with only an allocations field" + allocations = solution.get("allocations") + if not isinstance(allocations, dict): + return False, "allocations must be a dict" + queries = list(instance["queries"]) + expected = {q["id"] for q in queries} + if set(allocations.keys()) != expected: + return False, "allocations must cover exactly the query ids" + total = 0.0 + group_errors = {} + group_counts = {} + for q in queries: + qid = q["id"] + eps = allocations[qid] + if not isinstance(eps, (int, float)) or isinstance(eps, bool) or not math.isfinite(eps): + return False, "allocation for %s is not a finite number" % qid + eps = float(eps) + if eps < float(q["epsilon_min"]) - 1e-9: + return False, "allocation for %s is below epsilon_min" % qid + if eps > float(q["epsilon_max"]) + 1e-9: + return False, "allocation for %s is above epsilon_max" % qid + total += eps + err = _error(q, eps) + if err > float(q["max_error"]) + 1e-8: + return False, "allocation for %s exceeds max_error" % qid + group = q["group"] + group_errors[group] = group_errors.get(group, 0.0) + err + group_counts[group] = group_counts.get(group, 0) + 1 + if total > float(instance["epsilon_total"]) + 1e-8: + return False, "total epsilon exceeds epsilon_total" + avgs = [group_errors[g] / group_counts[g] for g in group_errors] + if avgs: + smallest = min(avgs) + largest = max(avgs) + if smallest <= 0.0: + return False, "group average error must be positive" + ratio = largest / smallest + if ratio > float(instance["fairness"]["max_group_error_ratio"]) + 1e-9: + return False, "group error fairness ratio exceeded" + return True, "ok" + + +def evaluate_solution(instance, solution): + ok, _ = validate_solution(instance, solution) + if not ok: + return -1e18 + value = _objective(instance, solution["allocations"]) + if not math.isfinite(value) or value <= 0.0: + return 1e-12 + return float(value) + + +def _baseline_allocations(instance): + queries = list(instance["queries"]) + allocations = {q["id"]: float(q["epsilon_min"]) for q in queries} + remaining = float(instance["epsilon_total"]) - sum(allocations.values()) + if remaining <= 0.0: + return allocations + remaining = _repair_fairness(instance, allocations, remaining) + remaining = min(remaining, 0.25 * float(instance["epsilon_total"])) + rounds = 0 + while remaining > 1e-12 and rounds < 10000: + used = 0.0 + for q in sorted(queries, key=lambda x: x["id"]): + qid = q["id"] + cap = float(q["epsilon_max"]) - allocations[qid] + if cap <= 1e-12: + continue + add = min(0.02, cap, remaining) + allocations[qid] += add + ok, _ = validate_solution(instance, {"allocations": dict(allocations)}) + if ok: + remaining -= add + used += add + else: + allocations[qid] -= add + if remaining <= 1e-12: + break + if used <= 1e-12: + break + rounds += 1 + return allocations + + +def solve_baseline(instance): + return {"allocations": _baseline_allocations(instance)} + + +def solve_random(instance): + queries = list(instance["queries"]) + allocations = {q["id"]: float(q["epsilon_min"]) for q in queries} + remaining = float(instance["epsilon_total"]) - sum(allocations.values()) + remaining = _repair_fairness(instance, allocations, remaining) + return {"allocations": allocations} + + +def solve_reference(instance): + queries = list(instance["queries"]) + allocations = {q["id"]: float(q["epsilon_min"]) for q in queries} + remaining = float(instance["epsilon_total"]) - sum(allocations.values()) + if remaining <= 0.0: + return {"allocations": allocations} + remaining = _repair_fairness(instance, allocations, remaining) + + def gain(q, current_allocations, delta): + qid = q["id"] + before = current_allocations[qid] + old = _objective(instance, current_allocations) + current_allocations[qid] = before + delta + try: + return (_objective(instance, current_allocations) - old) / delta + finally: + current_allocations[qid] = before + + _allocate_feasible_greedy(instance, allocations, remaining, gain) + return {"allocations": allocations} diff --git a/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/verification/process_runner.py b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/verification/process_runner.py new file mode 100644 index 00000000..3498497b --- /dev/null +++ b/benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/verification/process_runner.py @@ -0,0 +1,90 @@ +from __future__ import annotations + +import os +import signal +import subprocess +import threading +from dataclasses import dataclass + + +@dataclass(frozen=True) +class BoundedResult: + returncode: int + stdout: str + stderr: str + timed_out: bool + output_truncated: bool + + +def run_bounded( + command: list[str], input_text: str, *, timeout_s: float, max_output_bytes: int +) -> BoundedResult: + process = subprocess.Popen( + command, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + start_new_session=True, + ) + stdout = bytearray() + stderr = bytearray() + truncated = [False, False] + readers = [ + threading.Thread( + target=_drain, + args=(process.stdout, stdout, max_output_bytes, truncated, 0), + daemon=True, + ), + threading.Thread( + target=_drain, + args=(process.stderr, stderr, max_output_bytes, truncated, 1), + daemon=True, + ), + ] + for reader in readers: + reader.start() + writer = threading.Thread(target=_write, args=(process, input_text.encode("utf-8")), daemon=True) + writer.start() + timed_out = False + try: + process.wait(timeout=timeout_s) + except subprocess.TimeoutExpired: + timed_out = True + try: + os.killpg(process.pid, signal.SIGKILL) + except (ProcessLookupError, PermissionError, OSError): + process.kill() + process.wait() + for reader in readers: + reader.join() + return BoundedResult( + returncode=process.returncode, + stdout=stdout.decode("utf-8", errors="replace"), + stderr=stderr.decode("utf-8", errors="replace"), + timed_out=timed_out, + output_truncated=any(truncated), + ) + + +def _drain(stream, target: bytearray, limit: int, truncated: list[bool], index: int) -> None: + if stream is None: + return + while True: + chunk = stream.read(64 * 1024) + if not chunk: + return + remaining = limit - len(target) + if remaining > 0: + target.extend(chunk[:remaining]) + if len(chunk) > remaining: + truncated[index] = True + + +def _write(process: subprocess.Popen[bytes], content: bytes) -> None: + if process.stdin is None: + return + try: + process.stdin.write(content) + process.stdin.close() + except (BrokenPipeError, OSError): + pass diff --git a/benchmarks/PrivacyEngineering/README.md b/benchmarks/PrivacyEngineering/README.md new file mode 100644 index 00000000..e146d92e --- /dev/null +++ b/benchmarks/PrivacyEngineering/README.md @@ -0,0 +1,17 @@ +# Privacy Engineering + +This domain collects executable privacy-engineering optimization tasks with explicit +privacy-loss budgets, utility objectives, policy constraints, and independently +recomputed verification. + +## Tasks + +- `DifferentialPrivacyBudgetAllocation` + - Unified benchmark: `task=unified task.benchmark=PrivacyEngineering/DifferentialPrivacyBudgetAllocation` + - Quick run: `python -m frontier_eval task=unified task.benchmark=PrivacyEngineering/DifferentialPrivacyBudgetAllocation task.runtime.isolation_mode=process algorithm=openevolve algorithm.iterations=0` + - Description: allocate a fixed differential-privacy budget across analytics queries with heterogeneous sensitivity, business value, population coverage, fairness requirements, and estimation-accuracy constraints. + +The task models offline privacy-budget planning for business analytics portfolios. +NIST SP 800-226 and Dwork and Roth's differential privacy text provide the privacy +parameter, privacy loss, sensitivity, and utility-tradeoff context; the benchmark +instances are synthetic and are recomputed by the frozen verifier. diff --git a/benchmarks/PrivacyEngineering/README_zh-CN.md b/benchmarks/PrivacyEngineering/README_zh-CN.md new file mode 100644 index 00000000..17b352b7 --- /dev/null +++ b/benchmarks/PrivacyEngineering/README_zh-CN.md @@ -0,0 +1,14 @@ +# Privacy Engineering + +本领域收集可执行的隐私工程优化任务,强调明确的隐私损失预算、效用目标、策略约束和由验证器独立重算的结果。 + +## 任务列表 + +- `DifferentialPrivacyBudgetAllocation` + - `frontier_eval` 任务:`task=unified task.benchmark=PrivacyEngineering/DifferentialPrivacyBudgetAllocation` + - 快速运行:`python -m frontier_eval task=unified task.benchmark=PrivacyEngineering/DifferentialPrivacyBudgetAllocation task.runtime.isolation_mode=process algorithm=openevolve algorithm.iterations=0` + - 简介:在查询敏感度、业务价值、覆盖人群、公平性要求和估计精度约束不同的情况下,为一组分析查询分配固定的差分隐私预算。 + +该任务研究离线业务分析组合中的隐私预算规划。NIST SP 800-226 以及 Dwork 和 Roth +的差分隐私教材提供隐私参数、隐私损失、敏感度和效用权衡背景;benchmark 实例是合成数据, +并由冻结验证器独立重算。