add icebug-memory converter

.github/workflows/ci.yml

@@ -0,0 +1,26 @@
+name: CI
+
+on:
+  push:
+    branches: ["**"]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: astral-sh/setup-uv@v6
+        with:
+          python-version: "3.13"
+
+      - name: Install dependencies
+        run: |
+          uv pip install --system duckdb "pyarrow>=21"
+          uv pip install --system --no-deps -e .
+          uv pip install --system pytest
+
+      - name: Run tests
+        run: pytest

README.md

@@ -1,144 +1,111 @@
 # Icebug Format
 
-> **Note**: This project was formerly called **graph-std**.
+Icebug is a standardized graph format designed for efficient graph data interchange. It comes in two flavours:
 
-Icebug is a standardized graph format designed for efficient graph data interchange. It comes in two formats:
+| Format | Storage | Use case |
+|---|---|---|
+| **icebug-disk** | Parquet files | Object storage, persistence |
+| **icebug-memory** | Apache Arrow tables | In-process, zero-copy access |
 
-- **icebug-disk**: Parquet-based format for object storage
-- **icebug-memory**: Apache Arrow-based format for in-memory processing
+Both represent graphs in [CSR (Compressed Sparse Row)](https://en.wikipedia.org/wiki/Sparse_matrix#Compressed_sparse_row_(CSR,_CRS_or_Yale_format)) format, which enables fast adjacency-list traversal.
 
-This project provides tools to convert graph data from simple DuckDB databases or Parquet files containing `nodes_*` and `edges_*` tables, along with a `schema.cypher` file, into standardized graph formats for efficient processing.
+---
 
-## Sample Usage
+## icebug-disk v1
+
+### CLI
+
+Convert a DuckDB source database containing `nodes_*` / `edges_*` tables into Parquet files and a `schema.cypher` that a graph database can mount directly:
 
 ```bash
-uv run icebug-format.py \
---source-db karate/karate_random.duckdb \
---output-db karate/karate_csr.duckdb \
---csr-table karate \
---schema karate/karate_csr/schema.cypher
+uv run icebug-format \
+  --source-db examples/karate/duckdb/karate_random.duckdb \
+  --schema examples/karate/duckdb/schema.cypher      // input schema for rel tables
 ```
 
-This will create a CSR representation with multiple tables depending on the number of node and edge types:
+### Output structure
 
-- `{table_name}_indptr_{edge_name}`: Array of size N+1 for row pointers (one per edge table)
-- `{table_name}_indices_{edge_name}`: Array of size E containing column indices (one per edge table)
-- `{table_name}_nodes_{node_name}`: Original nodes table with node attributes (one per node table)
-- `{table_name}_mapping_{node_name}`: Maps original node IDs to contiguous indices (one per node table)
-- `{table_name}_metadata`: Global graph metadata (node count, edge count, directed flag)
-- `schema.cypher`: A cypher schema that a graph database can mount without ingesting
+For each node table `nodes_<name>` and edge table `edges_<name>`, the following files/tables are produced:
 
-## More information about Icebug and Apache GraphAR
+| Name | Description |
+|---|---|
+| `nodes_<name>.parquet` | Original node table with attributes |
+| `indices_<name>.parquet` | Target node for each edge, sorted by source (size E) |
+| `indptr_<name>.parquet` | Row-pointer array of size N+1 |
+| `schema.cypher` | Cypher schema for mounting in a graph database |
 
-[Blog Post](https://adsharma.github.io/graph-archiving/)
+NOTE: Each parquet file stores `icebug_disk_version` in its metadata
 
-## Recreating demo-db/icebug-disk
+### Example
 
-Start from a simple demo-db.duckdb that looks like this
+Starting from a `demo-db.duckdb` with `nodes_user`, `nodes_city`, `edges_follows`, and `edges_livesin` tables:
 
-```
-Querying database: demo-db.duckdb
-================================
-
---- Table: edges_follows ---
-┌────────┬────────┬───────┐
-│ source │ target │ since │
-│ int32  │ int32  │ int32 │
-├────────┼────────┼───────┤
-│    100 │    250 │  2020 │
-│    300 │     75 │  2022 │
-│    250 │    300 │  2021 │
-│    100 │    300 │  2020 │
-└────────┴────────┴───────┘
-================================
-
---- Table: edges_livesin ---
-┌────────┬────────┐
-│ source │ target │
-│ int32  │ int32  │
-├────────┼────────┤
-│    100 │    700 │
-│    250 │    700 │
-│    300 │    600 │
-│     75 │    500 │
-└────────┴────────┘
-================================
-
---- Table: nodes_city ---
-┌───────┬───────────┬────────────┐
-│  id   │   name    │ population │
-│ int32 │  varchar  │   int64    │
-├───────┼───────────┼────────────┤
-│   500 │ Guelph    │      75000 │
-│   600 │ Kitchener │     200000 │
-│   700 │ Waterloo  │     150000 │
-└───────┴───────────┴────────────┘
-================================
-
---- Table: nodes_user ---
-┌───────┬─────────┬───────┐
-│  id   │  name   │  age  │
-│ int32 │ varchar │ int64 │
-├───────┼─────────┼───────┤
-│   100 │ Adam    │    30 │
-│   250 │ Karissa │    40 │
-│    75 │ Noura   │    25 │
-│   300 │ Zhang   │    50 │
-└───────┴─────────┴───────┘
-================================
-
---- Schema: schema.cypher --
-CREATE NODE TABLE User(id INT64, name STRING, age INT64, PRIMARY KEY (id));
-CREATE NODE TABLE City(id INT64, name STRING, population INT64, PRIMARY KEY (id));
-CREATE REL TABLE Follows(FROM User TO User, since INT64);
-CREATE REL TABLE LivesIn(FROM User TO City);
+```bash
+uv run icebug-format \
+  --directed \
+  --source-db demo-db.duckdb \
+  --schema demo-db/schema.cypher
 ```
 
-and run:
+Verify the result with `test_csr_duckdb.py`:
 
+```bash
+uv run ./icebug-format/test_csr_duckdb.py --input demo-db_csr
 ```
-uv run icebug-format.py \
---directed \
---source-db demo-db.duckdb \
---output-db demo-db_csr.duckdb \
---csr-table demo \
---schema demo-db/schema.cypher
+
 ```
+Metadata: 7 nodes, 8 edges, directed=True
 
-You'll get a demo-db_csr.duckdb AND the object storage ready representation aka icebug-disk.
+Node Tables:
+Table: demo_nodes_user
+(100, 'Adam', 30) ...
 
-## Verification
+Edge Tables (reconstructed from CSR):
+Table: follows (FROM user TO user)
+(100, 250, 2020) ...
+```
 
-You can verify that the conversion went ok by running `scan.py`. It's also a good way to understand the icebug-disk format.
+---
 
-```
-uv run scan.py --input demo-db_csr --prefix demo
-Metadata: 7 nodes, 8 edges, directed=True
+## icebug-memory v1
 
-Node Tables:
+### Python API
 
-Table: demo_nodes_user
-(100, 'Adam', 30)
-(250, 'Karissa', 40)
-(75, 'Noura', 25)
-(300, 'Zhang', 50)
+Convert Arrow tables directly into an in-memory CSR graph
 
-Table: demo_nodes_city
-(500, 'Guelph', 75000)
-(600, 'Kitchener', 200000)
-(700, 'Waterloo', 150000)
+```python
+from icebug_format import IcebugMemGraph, convert_arrow_tables_to_csr
 
-Edge Tables (reconstructed from CSR):
+graph: IcebugMemGraph = convert_arrow_tables_to_csr(
+    from_node_arrow_table=users,   # pa.Table, first column is the primary key
+    to_node_arrow_table=cities,    # pa.Table, first column is the primary key
+    rel_arrow_table=livesin,       # pa.Table with 'source' and 'target' columns
+    directed=True,
+)
 
-Table: follows (FROM user TO user)
-(100, 250, 2020)
-(100, 300, 2020)
-(250, 300, 2021)
-(300, 75, 2022)
-
-Table: livesin (FROM user TO city)
-(75, 500)
-(100, 700)
-(250, 700)
-(300, 600)
+# Node tables are passed through unchanged
+graph.src    # pa.Table — source nodes
+graph.dest   # pa.Table — destination nodes
+
+# CSR adjacency structure
+graph.indices  # pa.Table — 'target' column (+ any edge properties), sorted by source
+graph.indptr   # pa.Table — 'ptr' column of length len(src) + 1
 ```
+
+The `rel_arrow_table` source and target columns are resolved by name in priority order, with a positional fallback:
+
+| Role | Accepted names (in order) | Fallback |
+|---|---|---|
+| Source | `source`, `src`, `from` | 0th column |
+| Target | `target`, `destination`, `dest`, `to` | 1st column |
+
+Any remaining columns are preserved as edge properties in `graph.indices`.
+
+Set `directed=False` to automatically add reverse edges (undirected graph).
+
+---
+
+## Further reading
+
+[Blog post: Graph Archiving with Apache GraphAR](https://adsharma.github.io/graph-archiving/)
+