Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 4 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ as structured, serializable data.

An investigation contains observables and their relationships, Findings,
supporting Evidence, threat intelligence, enrichments, and tags. Cyvest maintains
deterministic keys, calculates scores and security levels, and records mutations
in an audit log.
deterministic keys and calculates scores and security levels.

## Main capabilities

Expand Down Expand Up @@ -79,8 +78,7 @@ cv.io_save_json("investigation.json")
```

Public model objects are exposed through read-only proxies. Mutations use the
facade or fluent methods so score propagation, reverse links, and the audit log
remain consistent.
facade or fluent methods so score propagation and reverse links remain consistent.

For deterministic reports and comparisons, pass an explicit
`investigation_id` when creating the investigation.
Expand Down Expand Up @@ -361,16 +359,6 @@ cv = Cyvest(score_mode_obs=ScoreMode.SUM) # accumulative children
- `Investigation.investigation_id` is a stable ULID included in exports.
- Findings keep a *canonical origin* (`origin_investigation_id`) for LOCAL_ONLY propagation; it is compared against the current investigation id.

**Audit log**

- All meaningful changes (including score/level changes) are recorded in the investigation-level audit log.
- Per-object histories are not stored; use `cv.investigation_get_audit_log()` to review changes.
- For compact, deterministic JSON output (useful for testing/diffing), exclude the audit log:
```python
cv.io_save_json("output.json", include_audit_log=False) # audit_log: null
cv.io_to_invest(include_audit_log=False) # schema.audit_log is None
```

To force cross-investigation propagation for a specific link, use a GLOBAL link:

```python
Expand Down Expand Up @@ -712,9 +700,8 @@ The repo includes a PNPM workspace under `js/` with three packages:
- `@cyvest/cyvest-app`: Vite demo that bundles the JS packages with sample investigations.

The JS packages track the generated schema; serialized investigations should include fields like
`investigation_id`, `investigation_name`, `audit_log`, `score_display`, `finding_links`, and
`observable_links`. The investigation start time is recorded as an `INVESTIGATION_STARTED` event
in the `audit_log`.
`investigation_id`, `investigation_name`, `score_display`, `finding_links`, and
`observable_links`.

See `docs/js-packages.md` for workspace commands and usage snippets.

Expand Down
32 changes: 1 addition & 31 deletions docs/getting-started/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Each observable has:
> instances rather than raw dataclasses. These proxies provide live scores/levels but raise an error if you
> attempt to assign attributes. All mutations flow through the Investigation layer, so use the facade helpers
> (`cv.observable_add_threat_intel`, `cv.observable_set_level`, …) or the fluent methods on the proxies themselves
> (`with_ti`, `relate_to`, `link_observable`, `set_level`, etc.) so the score engine and audit log remain consistent.
> (`with_ti`, `relate_to`, `link_observable`, `set_level`, etc.) so the score engine remains consistent.
> Safe metadata fields (`comment`, `extra`, `internal`, etc.) can be updated via the dedicated `update_metadata()`
> helpers on each proxy. Use `set_level()` to update the level without changing the score:
>
Expand Down Expand Up @@ -371,36 +371,6 @@ explicit direction leaves both scores untouched. This is intended: a correlation
with no established deduction mechanism has no reason to make a score propagate.
`EXTRACTION` and `PIVOT` default to `OUTBOUND` and do propagate.

### Audit Log

All meaningful changes are recorded in a centralized, append-only audit log at the investigation level:

```python
# Observable score changes
obs = cv.observable_create(cv.OBS.IPV4, "10.0.0.1")
cv.observable_add_threat_intel(obs.key, "source1", score=Decimal("5.0"))
cv.observable_add_threat_intel(obs.key, "source2", score=Decimal("8.0"))

events = cv.investigation_get_audit_log()
obs_score_events = [
event
for event in events
if event.object_key == obs.key and event.event_type.startswith("SCORE")
]
for event in obs_score_events:
print(event.timestamp, event.details["old_score"], "→", event.details["new_score"])
print("Level:", event.details["old_level"], "→", event.details["new_level"])
print("Reason:", event.reason)
```

**Audit Event Fields (score changes):**

- Timestamp
- Old/new score values
- Old/new level values
- Reason for change (threat intel, propagation, merge, manual, etc.)
- Contributing investigation IDs when relevant (e.g., merges)

Investigation names are optional, human-readable labels. They are serialized separately from `investigation_id` and are never used for scoring or propagation.

### Hierarchical Score Propagation
Expand Down
5 changes: 1 addition & 4 deletions docs/getting-started/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,9 +166,6 @@ cv.io_save_json("investigation.json")
cv.io_save_markdown("report.md")
# Hide observables while keeping aggregate stats/whitelists
cv.io_save_markdown("redacted_report.md", include_observables=False)

# For compact, deterministic JSON (useful for testing/diffing):
cv.io_save_json("deterministic.json", include_audit_log=False)
```

!!! tip "Filtering findings by severity"
Expand All @@ -181,7 +178,7 @@ cv.io_save_json("deterministic.json", include_audit_log=False)
The docs assume you write to the project root, but automation pipelines typically point to `dist/` (JSON) and `reports/` (Markdown/PDF). Adjust paths to match your workflow.

!!! note "Provenance fields in JSON"
Exports include `investigation_id`, optional `investigation_name`, and the investigation-level `audit_log`, plus finding origins (`origin_investigation_id`) and link fields (`observable_links`, `finding_links`) needed for scoring after merges.
Exports include `investigation_id`, optional `investigation_name`, plus finding origins (`origin_investigation_id`) and link fields (`observable_links`, `finding_links`) needed for scoring after merges.

---

Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Build, score, and narrate cybersecurity investigations with a single fluent Pyth
| Area | Why it matters | What to look at |
| --- | --- | --- |
| **Structured objects** | Model observables, findings, TI, tags, and enrichments with typed helpers | `cyvest.model`, [Concepts](getting-started/concepts.md#observables) |
| **Deterministic scoring** | MAX/SUM propagation, centralized audit log, and automatic level classification | `cyvest.score`, [Scoring System](getting-started/concepts.md#scoring-system) |
| **Deterministic scoring** | MAX/SUM propagation and automatic level classification | `cyvest.score`, [Scoring System](getting-started/concepts.md#scoring-system) |
| **Fluent helpers** | Builder-style methods with deterministic keys and safe merges | `cyvest.cyvest`, [Quick Start](getting-started/quickstart.md#using-the-fluent-api) |
| **Shared context** | Thread-safe fragments that can reconcile into a single story | `cyvest.shared.SharedInvestigationContext`, [Guide](shared-investigation-context.md) |
| **Comparison** | Compare investigations with tolerance rules for regression testing | `cyvest.compare`, [Guide](comparing-investigations.md) |
Expand Down
5 changes: 2 additions & 3 deletions docs/js-packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,8 @@
Cyvest ships a small JavaScript/TypeScript workspace alongside the Python API. Use these packages to validate serialized investigations, power UI integrations, or explore the data model in a browser.

The JS packages follow the generated schema. Serialized investigations should include the
schema-required fields such as `investigation_id`, `investigation_name`, `audit_log`,
`score_display`, `finding_links`, and `observable_links`. The investigation start time is
recorded as an `INVESTIGATION_STARTED` event in the `audit_log`.
schema-required fields such as `investigation_id`, `investigation_name`,
`score_display`, `finding_links`, and `observable_links`.

## Packages

Expand Down
6 changes: 3 additions & 3 deletions docs/migration-v5-to-v6.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ This guide covers both serialized investigations and Python integrations.
| No required schema version | Exact `schema_version: "6.0.0"` |

The scoring model, relationship propagation, tags, enrichments, threat
intelligence, audit log, and immutable proxy behavior remain in place.
intelligence, and immutable proxy behavior remain in place.

## 1. Upgrade Cyvest

Expand Down Expand Up @@ -63,7 +63,7 @@ The migration command:
- renames the root `checks` collection to `findings`;
- renames `check_name` to `finding_name`;
- rewrites `chk:` keys to `fnd:` keys;
- updates Finding references in tags and audit events;
- updates Finding references in tags;
- recalculates observable and threat-intelligence keys;
- initializes `evidences` and every `evidence_links` collection;
- preserves existing `EMAIL` observables;
Expand Down Expand Up @@ -97,7 +97,7 @@ assert cv.finding_get_all()
```

Do not update keys with a global text replacement. Observable, threat
intelligence, tag, and audit references must remain consistent; the migration
intelligence, and tag references must remain consistent; the migration
command performs those updates as one validated operation.

## 3. Rename Check APIs to Finding APIs
Expand Down
14 changes: 3 additions & 11 deletions examples/04_email.py
Original file line number Diff line number Diff line change
Expand Up @@ -489,16 +489,8 @@ def run(self, cy: Cyvest) -> None:
@click_logger_params
@click.option("-w", "--workers", type=int, default=1)
@click.option("--stats", "stats", is_flag=True, default=False)
@click.option("--audit", "audit", is_flag=True, default=False)
@click.option(
"--no-audit-log",
"no_audit_log",
is_flag=True,
default=False,
help="Exclude audit log from JSON output for deterministic output",
)
@click.option("-o", "--output", type=click.Path(dir_okay=False, path_type=Path), default=None)
def main(workers, stats, audit, no_audit_log, output):
def main(workers, stats, output):
"""Main execution demonstrating multi-threaded investigation."""

# Prepare input data
Expand Down Expand Up @@ -553,13 +545,13 @@ def main(workers, stats, audit, no_audit_log, output):
# Display results
logger.info("Investigation complete - displaying summary - score should be 36.1")

cy.display_summary(show_audit_log=audit)
cy.display_summary()
if stats:
cy.display_statistics()

if output is not None:
logger.info("[bold cyan]Generating json...[/bold cyan]")
json_path = cy.io_save_json(output, include_audit_log=not no_audit_log)
json_path = cy.io_save_json(output)
size_kb = Path(json_path).stat().st_size / 1024
logger.info("[green]✓ Full json saved to: %s (%.2f KB)[/green]", json_path, size_kb)

Expand Down
11 changes: 2 additions & 9 deletions examples/05_graph_dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,8 @@

@click.command()
@click_logger_params
@click.option(
"--no-audit-log",
"no_audit_log",
is_flag=True,
default=False,
help="Exclude audit log from JSON output for deterministic output",
)
@click.option("-o", "--output", type=click.Path(dir_okay=False, path_type=Path), default=None)
def main(no_audit_log: bool = False, output: Path | None = None) -> None:
def main(output: Path | None = None) -> None:
# Create a comprehensive investigation with various observables
cv = Cyvest(
investigation_id="cyvest-visual-example",
Expand Down Expand Up @@ -159,7 +152,7 @@ def main(no_audit_log: bool = False, output: Path | None = None) -> None:
# Generate json
logger.info("[bold cyan]Generating json...[/bold cyan]")
output_path = output or (Path(tempfile.gettempdir()) / "cyvest_investigation.json")
json_path = cv.io_save_json(output_path, include_audit_log=not no_audit_log)
json_path = cv.io_save_json(output_path)
size_kb = Path(json_path).stat().st_size / 1024
logger.info("[green]✓ Full json saved to: %s (%.2f KB)[/green]", json_path, size_kb)

Expand Down
4 changes: 1 addition & 3 deletions js/packages/cyvest-app/src/App.tsx
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import type { CyvestInvestigation } from "@cyvest/cyvest-js";
import { getStartedAt } from "@cyvest/cyvest-js";
import {
CyvestGraph,
DARK_CYVEST_THEME,
Expand Down Expand Up @@ -75,8 +74,7 @@ export const App: React.FC = () => {
investigation.investigation_id}
</h2>
<p>
Started {getStartedAt(investigation) ?? "N/A"} · Schema{" "}
{investigation.schema_version}
Schema {investigation.schema_version}
</p>
</div>
<dl className="app-metrics">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
"level": "MALICIOUS",
"whitelisted": false,
"whitelists": [],
"audit_log": null,
"observables": {
"obs:artifact:root": {
"type": "artifact",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
"level": "MALICIOUS",
"whitelisted": false,
"whitelists": [],
"audit_log": null,
"observables": {
"obs:file:root": {
"type": "file",
Expand Down
23 changes: 0 additions & 23 deletions js/packages/cyvest-js/src/getters.ts
Original file line number Diff line number Diff line change
Expand Up @@ -379,29 +379,6 @@ export function getCounts(inv: CyvestInvestigation): InvestigationCounts {
};
}

/**
* Get the investigation start time from the event log.
*
* Looks for the INVESTIGATION_STARTED event and returns its timestamp.
*
* @param inv - The investigation
* @returns The start timestamp string or undefined if not found
*
* @example
* ```ts
* const startedAt = getStartedAt(investigation);
* if (startedAt) {
* console.log(`Started: ${startedAt}`);
* }
* ```
*/
export function getStartedAt(inv: CyvestInvestigation): string | undefined {
const event = inv.audit_log?.find(
(e) => e.event_type === "INVESTIGATION_STARTED"
);
return event?.timestamp;
}

// ============================================================================
// Tag Aggregation
// ============================================================================
Expand Down
28 changes: 0 additions & 28 deletions js/packages/cyvest-js/src/types.generated.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,15 +15,6 @@ export type Justification = string | null;
* List of whitelist entries applied to this investigation.
*/
export type Whitelists = InvestigationWhitelist[];
/**
* Append-only investigation audit log. Null when serialization disabled audit.
*/
export type AuditLog = AuditEvent[] | null;
export type Actor = string | null;
export type Reason = string | null;
export type Tool = string | null;
export type ObjectType = string | null;
export type ObjectKey = string | null;
export type Subtype = string | null;
export type Namespace = string | null;
export type Subtype1 = string | null;
Expand Down Expand Up @@ -89,7 +80,6 @@ export interface CyvestInvestigation {
*/
whitelisted: boolean;
whitelists: Whitelists;
audit_log?: AuditLog;
observables: Observables;
findings: Findings;
evidences: Evidences;
Expand All @@ -112,24 +102,6 @@ export interface InvestigationWhitelist {
justification?: Justification;
[k: string]: unknown;
}
/**
* Centralized audit event for investigation-level changes.
*/
export interface AuditEvent {
event_id: string;
timestamp: string;
event_type: string;
actor?: Actor;
reason?: Reason;
tool?: Tool;
object_type?: ObjectType;
object_key?: ObjectKey;
details?: Details;
[k: string]: unknown;
}
export interface Details {
[k: string]: unknown;
}
/**
* Observables keyed by their unique key.
*/
Expand Down
29 changes: 0 additions & 29 deletions js/packages/cyvest-js/tests/getters-finders.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ import {
getAllTags,
getAllObservables,
getCounts,
getStartedAt,
getTagChildren,
getTagDescendants,
getTagAggregatedScore,
Expand Down Expand Up @@ -51,15 +50,6 @@ function createTestInvestigation(): CyvestInvestigation {
score_display: "7.50",
level: "MALICIOUS",
whitelisted: false,
audit_log: [
{
event_id: "01HXYZTESTEVENT001",
timestamp: "2024-01-01T00:00:00Z",
event_type: "INVESTIGATION_STARTED",
object_type: "investigation",
object_key: "01HXYZTESTINVESTIGATION",
},
],
whitelists: [
{
identifier: "wl-1",
Expand Down Expand Up @@ -362,25 +352,6 @@ describe("Getters", () => {
expect(counts.whitelists).toBe(1);
});
});

describe("getStartedAt", () => {
it("returns timestamp from INVESTIGATION_STARTED event", () => {
const startedAt = getStartedAt(inv);
expect(startedAt).toBe("2024-01-01T00:00:00Z");
});

it("returns undefined when no audit_log", () => {
const invWithoutAuditLog = { ...inv, audit_log: undefined };
const startedAt = getStartedAt(invWithoutAuditLog);
expect(startedAt).toBeUndefined();
});

it("returns undefined when no INVESTIGATION_STARTED event", () => {
const invWithEmptyLog = { ...inv, audit_log: [] };
const startedAt = getStartedAt(invWithEmptyLog);
expect(startedAt).toBeUndefined();
});
});
});

describe("Finders", () => {
Expand Down
9 changes: 0 additions & 9 deletions js/packages/cyvest-js/tests/graph.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,6 @@ function createGraphTestInvestigation(): CyvestInvestigation {
score_display: "5.00",
level: "MALICIOUS",
whitelisted: false,
audit_log: [
{
event_id: "01HXYZTESTEVENT001",
timestamp: "2024-01-01T00:00:00Z",
event_type: "INVESTIGATION_STARTED",
object_type: "investigation",
object_key: "01HXYZGRAPHINVESTIGATION",
},
],
whitelists: [],
observables: {
"obs:file:msg1": {
Expand Down
Loading
Loading