Improve AI data privacy telemetry gates

[zdfgu113]

Pull Request Checklist

• Skill follows the format specification in CONTRIBUTING.md
• At least one real framework is cited with correct control IDs
• All framework references verified against primary sources (not blogs or AI output)
• Prompt Injection Safety Notice section included
• injection-hardened: true set in frontmatter
• allowed-tools scoped to minimum necessary permissions (Read, Grep, Glob)
• Tested with at least one AI coding agent (which one: Codex CLI / GPT-5 Codex)
• No prohibited patterns per SECURITY.md
• index.yaml updated with new skill entry (not applicable; this updates an existing skill)

What This PR Does

This updates skills/ai-security/ai-data-privacy/SKILL.md to cover two privacy review gaps that appear in modern AI telemetry systems:

• small-cohort leakage in AI usage analytics, dashboards, model-quality reports, and prompt-category exports;
• reversible or cross-domain pseudonymous identifiers, including stable unsalted hashes or reused user/account IDs across AI lifecycle stores.

It also adds false-positive guidance so reviewers do not report safe ephemeral local prompt classification when no raw text, user identifier, or linkable redacted payload is persisted and only k-thresholded aggregate metrics are exported.

Resubmission of #2130 with maintainer-requested usage evidence. Refs #2128.

Framework References

• NIST AI RMF 1.0: MAP 5.1, MEASURE 2.9, MANAGE 2.4, GOVERN 1.1
• OWASP Top 10 for LLM Applications 2025: LLM02 Sensitive Information Disclosure
• GDPR: Articles 5, 6, 13, 17, 25, 35
• EU AI Act: Articles 10, 11, 13
• CCPA/CPRA: California Civil Code Sec. 1798.100-199

Testing

Tested with Codex CLI / GPT-5 Codex against a real public AI observability repository:

• Target: langfuse/langfuse
• Target commit: 53433e881d2c8aeaf948eccfe4bc0dd89444ce85
• Method: applied the updated skill's read-only Grep checks for AI telemetry, dashboard aggregation, user/session identifiers, retention controls, and small-cohort suppression language.

Evidence from the skill run:

1. AI telemetry dashboards expose reviewer-relevant small-cohort questions.

   • worker/src/constants/langfuse-dashboards.json:62 describes aggregated model cost by trace.userId.
   • worker/src/constants/langfuse-dashboards.json:64 sets dimensions: [{ field: "userId" }].
   • worker/src/constants/langfuse-dashboards.json:129-132 defines Max Latency by User Id (Traces) with userId as the dimension.
   • Follow-up grep for cohort, k-anonym, minimum cell, small cell, suppression, and threshold across dashboard seeds, Mixpanel/PostHog transformers, and DashboardService paths returned no matches. This is not a confirmed vulnerability by itself, but it is exactly the review prompt this PR adds: reviewers should verify minimum-cell suppression after filters and exports.

2. Pseudonymous/linkable identifier checks produced concrete review evidence.

   • worker/src/services/IngestionService/index.ts:292-293 stores ingested userId and sessionId as user_id and session_id.
   • packages/shared/src/domain/traces.ts:27-28 and packages/shared/src/domain/observations.ts:117-118 model trace/observation user and session identifiers.
   • worker/src/features/mixpanel/transformers.ts:64-78 uses trace.langfuse_user_id as Mixpanel distinct_id / $user_id and exports session identifiers.
   • worker/src/features/posthog/transformers.ts:31-39 uses trace.langfuse_user_id as PostHog distinctId and maps posthog_session_id to $session_id.
   • The updated skill turns this into a structured question: are these identifiers purpose-scoped, non-dictionary-attackable, retention-bounded, and not reused across AI telemetry, support, billing, and evaluation domains without legal basis?

3. The run also produced a useful non-finding.

   • Retention controls are present: worker/src/ee/dataRetention/handleDataRetentionProcessingJob.ts re-fetches current project retention and deletes ClickHouse/S3 data older than the current setting; tests cover trace, observation, score, media, and disabled-retention cases.
   • This supports the PR's false-positive guidance: the new telemetry gates should not over-report when there is evidence of bounded retention or ephemeral-only processing.

Representative commands used:

rg -n -i "prompt_category|completion_category|ai_event|model_event|telemetry|analytics|dashboard|cohort|trace|observation" web worker packages ee --glob "*.{ts,tsx,js,jsx,py,yaml,yml,json,md}"
rg -n -i "sha256|sha1|md5|hash|digest|pseudonym|anonymous_id|subject_id|hmac|userId|user_id|sessionId|session_id" web worker packages ee --glob "*.{ts,tsx,js,jsx,py,yaml,yml,json,md}"
rg -n -i "cohort|k-anonym|minimum cell|small cell|suppression|threshold" worker/src/constants/langfuse-dashboards.json worker/src/features/mixpanel worker/src/features/posthog packages/shared/src/server/services/DashboardService

Bounty Info

• Preferred payment method: Crypto, preferably Base USDC; payout details can be provided privately after acceptance.

[github-actions]

Thanks for the submission! 🙏 SecuritySkills is now issue-first: contributions need a linked issue that a maintainer has marked approved before a PR is opened.

Please open an issue describing the skill, wait for the approved label, then reopen this PR with Closes #<issue> in the description. The PR template lists everything we'll look for (including an independently runnable reproduction).