Shared: Improve sensitive data heuristics

[geoffw0]

Improve sensitive data heuristics, by adding new matching heuristics and new exclusions. This is very, very loosely based on an initial version by Copilot, though I had to cut or refine the majority of those suggestions to get good results (testing with Rust).

I will start DCA runs on all affected languages. I do anticipate some further refinements may be required as common patterns vary slightly between languages.

• check DCA (Javascript).
• check DCA (Python).
• check DCA (Ruby).
• check DCA (Rust).
• check DCA (Swift).
• add change notes.

There are three things I'd like opinions about:

• are you happy with the changes to heuristics in this PR?
• would you be happy if I made a follow-up PR enabling the e(mail|_mail) heuristic? Its labelled with a comment "seems too noisy", but I don't understand what motivated that - it always worked very well in my tests (Swift + Rust).
• would you be happy if I made a follow-up disabling or narrowing the uid heuristic (([uU]|^|_|[a-z](?=U))([uU][iI][dD]))? I believe its responsible for a lot of false positive results because UID might stand for "User ID" (likely sensitive) or just "Unique ID" (which may or may not be sensitive, often they aren't). What are your experiences? (@andersfugmann FYI as we discussed this recently)

[Copilot]

Pull Request Overview

This PR improves sensitive data heuristics across multiple programming languages by enhancing the regular expressions used to identify sensitive data patterns. The changes add new matching patterns for various types of sensitive information while also improving exclusions to reduce false positives.

• Expands detection patterns for authentication data (oauth, api keys, mfa), financial information (cvv, iban, routing numbers), health data (gender, patient records), and device information (IP addresses, MAC addresses, fingerprints)
• Adds "confidential" to secret detection patterns and improves the trusted pattern exclusion
• Enhances exclusion patterns to reduce false positives for file paths, URLs, and accounting-related terms

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
swift/ql/lib/codeql/swift/security/internal/SensitiveDataHeuristics.qll	Updates sensitive data detection patterns for Swift
rust/ql/lib/codeql/rust/security/internal/SensitiveDataHeuristics.qll	Updates sensitive data detection patterns for Rust
ruby/ql/lib/codeql/ruby/security/internal/SensitiveDataHeuristics.qll	Updates sensitive data detection patterns for Ruby
python/ql/lib/semmle/python/security/internal/SensitiveDataHeuristics.qll	Updates sensitive data detection patterns for Python
javascript/ql/lib/semmle/javascript/security/internal/SensitiveDataHeuristics.qll	Updates sensitive data detection patterns for JavaScript
rust/ql/test/library-tests/sensitivedata/test.rs	Updates test expectations to reflect improved heuristics

[geoffw0]

Based on DCA results across all affected languages I think I'm going to:

• remove the new pattern ip.?addr (too often the matched value is an external "public" IP address, not our own or a users that might be considered identifying information).
• remove the new pattern finger.?print (sometimes this is a fingerprint identifying our machine (or credit card!), but often it's just a hash of some impersonal data).
• allow a bit of time for your feedback / change requests.
• re-run DCA on all languages.