Skip to content

Commit f64967f

Browse files
author
Fahad Alghanim
committed
Final QA: fix benchmarks, snippets, scorer quality, FastAPI docs, verification loop clean pass
1 parent 84ef8db commit f64967f

File tree

10 files changed

+722
-91
lines changed

10 files changed

+722
-91
lines changed

‎README.md‎

Lines changed: 133 additions & 75 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,11 @@ Add suspicious-login detection, auth-risk scoring, and step-up decisions to your
44

55
![MIT License](https://img.shields.io/badge/license-MIT-green.svg)
66
![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue.svg)
7-
![Tests Passing](https://img.shields.io/badge/tests-35%20passing-brightgreen.svg)
7+
![Tests Passing](https://img.shields.io/badge/tests-36%20passing-brightgreen.svg)
8+
![pip installable](https://img.shields.io/badge/pip-installable-blue.svg)
89

910
## Why This Exists
10-
Every fintech and crypto app builds auth-risk scoring internally because generic fraud tooling rarely models authentication flow context well. Teams need to distinguish legitimate users from attackers before full identity verification and before adding friction. This project packages those patterns into an open-source toolkit with synthetic-data-first workflows so you can run everything locally.
11+
Fintech and crypto teams must decide risk at authentication time, before transaction monitoring can help. Most teams rebuild this stack internally, and existing OSS tools usually cover only one slice (device, geo, or behavior). `identity-risk-engine` packages multi-signal auth risk scoring, policy decisions, and synthetic benchmarking into one local-first toolkit.
1112

1213
## Architecture
1314
```text
@@ -23,7 +24,7 @@ Auth Events
2324
v
2425
+---------------------------+
2526
| Risk Scoring |
26-
| Composite + signal fusion |
27+
| signal fusion + ensemble |
2728
+---------------------------+
2829
|
2930
v
@@ -37,6 +38,8 @@ Action: allow | step-up | review | block | revoke
3738
```
3839

3940
## Install
41+
> **Requires Python 3.9+** — check with `python3 --version`
42+
4043
```bash
4144
git clone https://github.com/KOKOSde/identity-risk-engine.git
4245
cd identity-risk-engine
@@ -45,37 +48,114 @@ python3 -m pip install -e .
4548

4649
## Quickstart
4750
```python
48-
from identity_risk_engine.simulator_ire import generate_synthetic_auth_events
4951
from identity_risk_engine.policy_engine import PolicyEngine
5052
from identity_risk_engine.risk_engine_ire import score_event
53+
from identity_risk_engine.simulator_ire import generate_synthetic_auth_events
5154

52-
events = generate_synthetic_auth_events(num_users=50, num_sessions=1000, attack_ratio=0.2, seed=42)
53-
event = events.iloc[50].to_dict()
54-
history = events.iloc[:50]
55-
result = score_event(event=event, history_df=history, policy_engine=PolicyEngine())
56-
57-
print(result["risk_score"], result["decision"]["action"])
55+
events = generate_synthetic_auth_events(num_users=20, num_sessions=200, attack_ratio=0.2, seed=42)
56+
result = score_event(event=events.iloc[40].to_dict(), history_df=events.iloc[:40], policy_engine=PolicyEngine())
57+
print(round(result["risk_score"], 4), result["decision"]["action"])
5858
print(result["explanation"]["human_summary"])
5959
```
6060

61-
`PolicyEngine` exposes `decide()` (not `evaluate()`), and per-event explanations are returned by `risk_engine_ire.score_event()` under `result["explanation"]`.
61+
`PolicyEngine` uses `decide()` (not `evaluate()`), and explanations are available via `result["explanation"]` from `risk_engine_ire.score_event(...)` or `explainer_ire.explain_scored_event(...)`.
6262

6363
## CLI Quickstart
6464
```bash
6565
python3 -m identity_risk_engine.cli_ire simulate --users 500 --sessions 20000 --attack-ratio 0.2 --out synthetic.csv
6666
python3 -m identity_risk_engine.cli_ire score --events synthetic.csv --policy configs/default_policy.yaml --out scored.csv
6767
python3 -m identity_risk_engine.cli_ire report --events scored.csv --out report.html
68-
# fast demo mode for large files:
69-
python3 -m identity_risk_engine.cli_ire score --events synthetic.csv --policy configs/default_policy.yaml --fast --out scored_fast.csv
70-
# force full scoring mode even on large files:
71-
python3 -m identity_risk_engine.cli_ire score --events synthetic.csv --policy configs/default_policy.yaml --full --out scored_full.csv
68+
```
69+
70+
`simulate` example output:
71+
```text
72+
Generated 37926 events -> /tmp/ire_verify.csv
73+
Attack mix:
74+
account_takeover: 159
75+
bot_behavior: 1014
76+
credential_stuffing: 1705
77+
impossible_travel: 318
78+
mfa_fatigue: 1496
79+
multi_account_sybil: 1155
80+
new_account_fraud: 188
81+
normal: 29587
82+
passkey_registration_abuse: 760
83+
recovery_abuse: 1208
84+
session_hijack: 336
85+
```
86+
87+
`score` example output:
88+
```text
89+
Auto-selecting fast mode for 37926 events (use --full for complete signal extraction)
90+
Scored 37926 events -> /tmp/ire_verify_scored.csv
91+
Scoring mode: fast-auto (history_window=8)
92+
Elapsed seconds: 27.90
93+
Mean risk score: 0.1016
94+
Action counts:
95+
allow: 24285
96+
allow_with_monitoring: 8678
97+
step_up_with_passkey: 2044
98+
step_up_with_totp: 1623
99+
require_recovery_review: 411
100+
block: 318
101+
```
102+
103+
`report` example output:
104+
```text
105+
Report summary:
106+
total_events: 37926
107+
avg_risk_score: 0.101567
108+
p95_risk_score: 0.44
109+
positive_rate: 0.219876
110+
Top actions:
111+
allow: 24285
112+
allow_with_monitoring: 8678
113+
step_up_with_passkey: 2044
114+
step_up_with_totp: 1623
115+
require_recovery_review: 411
116+
block: 318
117+
Report written -> /tmp/ire_verify_report.html
72118
```
73119

74120
## FastAPI Quickstart
75121
```bash
76122
uvicorn examples.fastapi_demo.app_ire:app --reload
77123
curl -s http://127.0.0.1:8000/health
78-
curl -s -X POST http://127.0.0.1:8000/simulate -H "Content-Type: application/json" -d '{"num_users":10,"num_sessions":100}'
124+
```
125+
126+
`POST /events` request schema:
127+
```json
128+
{
129+
"dry_run": false,
130+
"event": {
131+
"event_id": "evt_demo_001",
132+
"event_type": "login_success",
133+
"user_id": "user_demo_01",
134+
"session_id": "sess_demo_01",
135+
"timestamp": "2026-03-22T12:30:00Z",
136+
"ip": "34.23.11.9",
137+
"country": "US",
138+
"city_coarse": "San Francisco",
139+
"lat_coarse": 37.77,
140+
"lon_coarse": -122.42,
141+
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/122.0",
142+
"device_hash": "dev_demo_01",
143+
"device_type": "desktop",
144+
"browser": "Chrome",
145+
"os": "Windows",
146+
"auth_method": "password",
147+
"success": true,
148+
"tenant_id": "tenant_1",
149+
"metadata": {"ip_asn": "AS12345"}
150+
}
151+
}
152+
```
153+
154+
Example curl:
155+
```bash
156+
curl -s -X POST http://127.0.0.1:8000/events \
157+
-H "Content-Type: application/json" \
158+
-d '{"dry_run":false,"event":{"event_id":"evt_demo_001","event_type":"login_success","user_id":"user_demo_01","session_id":"sess_demo_01","timestamp":"2026-03-22T12:30:00Z","ip":"34.23.11.9","country":"US","city_coarse":"San Francisco","lat_coarse":37.77,"lon_coarse":-122.42,"user_agent":"Mozilla/5.0","device_hash":"dev_demo_01","device_type":"desktop","browser":"Chrome","os":"Windows","auth_method":"password","success":true,"tenant_id":"tenant_1","metadata":{"ip_asn":"AS12345"}}}'
79159
```
80160

81161
## Supported Auth Flows
@@ -84,88 +164,66 @@ curl -s -X POST http://127.0.0.1:8000/simulate -H "Content-Type: application/jso
84164
- MFA challenge flows (sent/passed/failed)
85165
- Password reset and account recovery
86166
- Session creation/revocation
87-
- Profile credential changes (email/phone)
88-
- OAuth/magic-link style events via normalized event schema fields
167+
- Credential profile changes (`email_changed`, `phone_changed`)
168+
- OAuth/magic-link style flows normalized through the same event schema
89169

90170
## Risk Signals
91171
| Category | Signals |
92172
|---|---|
93173
| Device | `new_device`, `device_dormant`, `multi_account_device`, `device_velocity`, `session_churn`, `emulator_heuristic` |
94174
| Geo/Network | `impossible_travel`, `geo_velocity`, `new_country`, `new_asn`, `tor_vpn_proxy`, `ip_velocity`, `residential_vs_datacenter` |
95-
| Behavior | `failure_burst`, `success_after_burst`, `unusual_hour`, `auth_method_switch`, `mfa_fatigue`, `recovery_abuse`, `login_cadence_anomaly`, `account_fanout` |
175+
| Behavior | `failure_burst`, `success_after_burst`, `unusual_hour`, `auth_method_switch`, `mfa_fatigue`, `recovery_abuse`, `login_cadence_anomaly`, `account_fanout`, `new_account_high_value`, `metadata_attack_hints` |
96176
| Passkey | `new_passkey_unfamiliar_device`, `passkey_registration_burst`, `passkey_after_password_failure`, `authenticator_churn`, `credential_novelty` |
97177
| Recovery | `recovery_unfamiliar_env`, `recovery_after_lockout`, `recovery_plus_credential_change`, `recovery_fanout`, `recovery_impossible_travel` |
98178

99179
## Policy Engine
100-
Default config lives at `configs/default_policy.yaml` and supports score thresholds, per-auth-method overrides, per-tenant overrides, and dry-run mode.
180+
Default config is at `configs/default_policy.yaml`.
101181

102182
```yaml
103183
dry_run: false
104184
thresholds:
105-
- max_score: 0.15
106-
action: allow
107-
- max_score: 0.30
108-
action: allow_with_monitoring
109-
- max_score: 0.45
110-
action: step_up_with_passkey
111-
- max_score: 0.60
112-
action: step_up_with_totp
113-
- max_score: 0.72
114-
action: step_up_with_email_code
115-
- max_score: 0.82
116-
action: require_recovery_review
117-
- max_score: 0.90
118-
action: manual_review
119-
- max_score: 0.97
120-
action: block
121-
- max_score: 1.00
122-
action: revoke_session
185+
- { max_score: 0.15, action: allow }
186+
- { max_score: 0.30, action: allow_with_monitoring }
187+
- { max_score: 0.45, action: step_up_with_passkey }
188+
- { max_score: 0.60, action: step_up_with_totp }
189+
- { max_score: 0.72, action: step_up_with_email_code }
190+
- { max_score: 0.82, action: require_recovery_review }
191+
- { max_score: 0.90, action: manual_review }
192+
- { max_score: 0.97, action: block }
193+
- { max_score: 1.00, action: revoke_session }
123194
```
124195
125196
Supported actions: `allow`, `allow_with_monitoring`, `step_up_with_passkey`, `step_up_with_totp`, `step_up_with_email_code`, `require_recovery_review`, `manual_review`, `block`, `revoke_session`.
126197

127198
## Benchmark Results
128-
Run details: `num_users=100`, `num_sessions=2400`, `attack_ratio=0.22`, `seed=21`, time split from `demo_outputs/benchmark_output_ire.txt`.
129-
Reproducible with `seed=42` in model training and deterministic model settings.
199+
Generated from code with fixed seed using:
130200

131-
| Cohort | AUC | Precision@0.95Recall | Recall@0.95Precision |
132-
|---|---:|---:|---:|
133-
| Global | 1.000 | 1.000 | 1.000 |
134-
| account_takeover | 0.916 | 0.246 | 0.000 |
135-
| bot_behavior | 0.906 | 0.138 | 0.000 |
136-
| credential_stuffing | 0.908 | 0.262 | 0.000 |
137-
| impossible_travel | 0.910 | 0.177 | 0.000 |
138-
| new_account_fraud | 0.910 | 0.177 | 0.000 |
139-
140-
## Demo Output
141-
CLI report summary snippet (from `/tmp/ire_report.html` generated by CLI):
142-
143-
```text
144-
total_events: 19101
145-
avg_risk_score: 0.533134
146-
p95_risk_score: 1.0
147-
positive_rate: 0.222711
201+
```bash
202+
python3 scripts/generate_benchmark_table_ire.py --num-users 100 --num-sessions 4000 --attack-ratio 0.2 --seed 42
148203
```
149204

150-
FastAPI `/simulate` response snippet (from `demo_outputs/fastapi_simulate_ire.txt`):
205+
Script: `scripts/generate_benchmark_table_ire.py`
206+
Outputs: `demo_outputs/benchmark_table_ire.md`, `demo_outputs/benchmark_table_ire.json`
151207

152-
```json
153-
{
154-
"generated_events": 509,
155-
"scored_events": 509,
156-
"mean_risk_score": 0.7395597724160389,
157-
"action_counts": {
158-
"revoke_session": 208,
159-
"manual_review": 125,
160-
"allow": 107
161-
},
162-
"attack_counts": {
163-
"normal": 368,
164-
"credential_stuffing": 37,
165-
"recovery_abuse": 32
166-
}
167-
}
168-
```
208+
| Cohort | AUC | Precision@0.95Recall | Recall@0.95Precision |
209+
|---|---:|---:|---:|
210+
| Global | 0.987167 | 0.865171 | 0.712741 |
211+
| account_takeover | 0.838010 | 0.024493 | 0.000000 |
212+
| bot_behavior | 0.883121 | 0.078209 | 0.000000 |
213+
| credential_stuffing | 0.872556 | 0.182295 | 0.000000 |
214+
| impossible_travel | 0.695006 | 0.000000 | 0.000000 |
215+
| mfa_fatigue | 0.827267 | 0.183863 | 0.000000 |
216+
| multi_account_sybil | 0.934892 | 0.113269 | 0.000000 |
217+
| new_account_fraud | 0.939105 | 0.027721 | 0.000000 |
218+
| passkey_registration_abuse | 0.952384 | 0.123056 | 0.000000 |
219+
| recovery_abuse | 0.964671 | 0.326278 | 0.000000 |
220+
| session_hijack | 0.953187 | 0.084667 | 0.000000 |
221+
222+
## Scorer Quality Check (Seed 42)
223+
- AUROC: `0.9818`
224+
- Near-zero attack scores (`<0.1`): `189/8339` (`2.3%`)
225+
- `session_hijack`: mean score `1.000`, near-zero `0.0%`
226+
- `passkey_registration_abuse`: mean score `0.996`, near-zero `0.0%`
169227

170228
## Who This Is For
171229
- Crypto exchanges

‎demo_outputs/benchmark_table_generation_ire.txt‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
Global AUROC: 0.987167
2+
Markdown table: /scratch/fkalghan/circuit_discovery_and_supression/graphs_ai_psych/identity-risk-engine/demo_outputs/benchmark_table_ire.md
3+
JSON metrics: /scratch/fkalghan/circuit_discovery_and_supression/graphs_ai_psych/identity-risk-engine/demo_outputs/benchmark_table_ire.json
4+
5+
| Cohort | AUC | Precision@0.95Recall | Recall@0.95Precision |
6+
|---|---:|---:|---:|
7+
| Global | 0.987167 | 0.865171 | 0.712741 |
8+
| account_takeover | 0.838010 | 0.024493 | 0.000000 |
9+
| bot_behavior | 0.883121 | 0.078209 | 0.000000 |
10+
| credential_stuffing | 0.872556 | 0.182295 | 0.000000 |
11+
| impossible_travel | 0.695006 | 0.000000 | 0.000000 |
12+
| mfa_fatigue | 0.827267 | 0.183863 | 0.000000 |
13+
| multi_account_sybil | 0.934892 | 0.113269 | 0.000000 |
14+
| new_account_fraud | 0.939105 | 0.027721 | 0.000000 |
15+
| passkey_registration_abuse | 0.952384 | 0.123056 | 0.000000 |
16+
| recovery_abuse | 0.964671 | 0.326278 | 0.000000 |
17+
| session_hijack | 0.953187 | 0.084667 | 0.000000 |

‎demo_outputs/benchmark_table_ire.json‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
{
2+
"config": {
3+
"num_users": 100,
4+
"num_sessions": 4000,
5+
"attack_ratio": 0.2,
6+
"seed": 42,
7+
"model_random_state": 42
8+
},
9+
"global_auc": 0.9871674467754769,
10+
"rows": [
11+
{
12+
"cohort": "Global",
13+
"auc": 0.9871674467754769,
14+
"precision_at_95_recall": 0.8651709401709402,
15+
"recall_at_95_precision": 0.7127409496944053
16+
},
17+
{
18+
"cohort": "account_takeover",
19+
"auc": 0.8380095687031318,
20+
"precision_at_95_recall": 0.024493020805899393,
21+
"recall_at_95_precision": 0.0
22+
},
23+
{
24+
"cohort": "bot_behavior",
25+
"auc": 0.8831208569156869,
26+
"precision_at_95_recall": 0.07820927723840346,
27+
"recall_at_95_precision": 0.0
28+
},
29+
{
30+
"cohort": "credential_stuffing",
31+
"auc": 0.8725561797870893,
32+
"precision_at_95_recall": 0.1822953948938201,
33+
"recall_at_95_precision": 0.0
34+
},
35+
{
36+
"cohort": "impossible_travel",
37+
"auc": 0.6950064627024247,
38+
"precision_at_95_recall": 0.0,
39+
"recall_at_95_precision": 0.0
40+
},
41+
{
42+
"cohort": "mfa_fatigue",
43+
"auc": 0.8272670036563118,
44+
"precision_at_95_recall": 0.18386329223447978,
45+
"recall_at_95_precision": 0.0
46+
},
47+
{
48+
"cohort": "multi_account_sybil",
49+
"auc": 0.9348919756667037,
50+
"precision_at_95_recall": 0.11326860841423948,
51+
"recall_at_95_precision": 0.0
52+
},
53+
{
54+
"cohort": "new_account_fraud",
55+
"auc": 0.9391048405058279,
56+
"precision_at_95_recall": 0.027721433400946585,
57+
"recall_at_95_precision": 0.0
58+
},
59+
{
60+
"cohort": "passkey_registration_abuse",
61+
"auc": 0.9523841291867123,
62+
"precision_at_95_recall": 0.12305611899932387,
63+
"recall_at_95_precision": 0.0
64+
},
65+
{
66+
"cohort": "recovery_abuse",
67+
"auc": 0.9646705289945313,
68+
"precision_at_95_recall": 0.32627774909654106,
69+
"recall_at_95_precision": 0.0
70+
},
71+
{
72+
"cohort": "session_hijack",
73+
"auc": 0.9531868828219887,
74+
"precision_at_95_recall": 0.08466701084150749,
75+
"recall_at_95_precision": 0.0
76+
}
77+
]
78+
}

‎demo_outputs/benchmark_table_ire.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
| Cohort | AUC | Precision@0.95Recall | Recall@0.95Precision |
2+
|---|---:|---:|---:|
3+
| Global | 0.987167 | 0.865171 | 0.712741 |
4+
| account_takeover | 0.838010 | 0.024493 | 0.000000 |
5+
| bot_behavior | 0.883121 | 0.078209 | 0.000000 |
6+
| credential_stuffing | 0.872556 | 0.182295 | 0.000000 |
7+
| impossible_travel | 0.695006 | 0.000000 | 0.000000 |
8+
| mfa_fatigue | 0.827267 | 0.183863 | 0.000000 |
9+
| multi_account_sybil | 0.934892 | 0.113269 | 0.000000 |
10+
| new_account_fraud | 0.939105 | 0.027721 | 0.000000 |
11+
| passkey_registration_abuse | 0.952384 | 0.123056 | 0.000000 |
12+
| recovery_abuse | 0.964671 | 0.326278 | 0.000000 |
13+
| session_hijack | 0.953187 | 0.084667 | 0.000000 |

0 commit comments

Comments
 (0)