Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
6f88e41
Save difference log regardless of errors
jucor Dec 1, 2025
6ac18ae
Account for flip sign in numerical difference reporting
jucor Dec 1, 2025
85cd2e4
Reformat help message
jucor Dec 3, 2025
4839347
Add cluster centers to PCA-related paths for sign flip handling
jucor Jan 5, 2026
6137b61
Allow for a few components with higher error tolerance
jucor Jan 6, 2026
09e9c5e
Fix per-component PCA sign flip detection
jucor Jan 6, 2026
5b2d8a0
Replace element-wise tolerance with projection metrics for regression…
jucor Jan 6, 2026
248c4ef
Address Copilot review feedback on regression comparer
jucor Feb 25, 2026
2a45f8f
Add PCA benchmark script
jucor Dec 1, 2025
0bc0303
Replace handwritten PCA by SKLearn's implementation
jucor Dec 1, 2025
851cc78
Replace projection by Sklearn's fit_transform for better performance.
jucor Dec 1, 2025
d59689a
Remove unused start_vectors argument
jucor Dec 1, 2025
52b58e7
Remove function wrapped_pca
jucor Dec 1, 2025
1b7d40e
Update biodiversity and vw golden snapshots with sklearn PCA
jucor Jan 6, 2026
c57ec94
Fix PCA warnings in test_conversation.py
jucor Jan 6, 2026
ec45a70
Address Copilot review feedback on PCA implementation
jucor Jan 6, 2026
c25ccd0
Address Copilot review feedback on sklearn PCA migration
jucor Feb 25, 2026
ff19ec1
Replace PCA Clojure comparison test with pass
jucor Jan 6, 2026
79d6b2f
Consolidate Clojure comparison tests into test_legacy_clojure_regress…
jucor Jan 6, 2026
05a7cd9
FIx path of mock file for test run_math_pipeline_e2e
jucor Jan 7, 2026
2e8e6cf
Clarify test name
jucor Jan 7, 2026
5f9d1bc
Consolidate and reorganize Clojure comparison tests
jucor Jan 13, 2026
8956137
Implement two-level hierarchical clustering matching Clojure architec…
jucor Jan 14, 2026
3e7e641
Add script to generate cold-start Clojure math blobs for fair comparison
jucor Jan 14, 2026
1e2760c
Implement conversation replay approach for cold-start generation and …
jucor Jan 14, 2026
10c8e9d
Improve cold-start generator and cluster visualizer CLI
jucor Jan 14, 2026
080d763
Compute cold-start math blobs for kmeans
jucor Jan 14, 2026
abf5096
Interrupt upon Clojure error
jucor Jan 15, 2026
0aaf98e
Mark known Clojure discrepancies as xfail and re-record golden snapshots
jucor Feb 25, 2026
0d0c786
Address Copilot review feedback and fix pre-existing test failures
jucor Feb 25, 2026
d6a728c
Unfold group-cluster base-cluster IDs to participant IDs for downstre…
jucor Feb 25, 2026
3c8d039
Tighten Clojure comparison tests and fix outdated docstrings
jucor Feb 26, 2026
e35f31a
Fix serialization outputting base-cluster IDs instead of participant IDs
jucor Mar 3, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
414 changes: 414 additions & 0 deletions delphi/docs/CLOJURE_COMPARISON.md

Large diffs are not rendered by default.

204 changes: 204 additions & 0 deletions delphi/docs/CLOJURE_TWO_LEVEL_CLUSTERING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@
# Clojure Two-Level Clustering Architecture

## Overview

**YES**, Clojure **ALWAYS** uses two-level hierarchical clustering, not conditionally.

## Architecture

### Level 1: Base Clusters (Participants → ~100 Base Clusters)

**Location**: `polismath/math/conversation.clj` lines 400-407

```clojure
:base-clusters
(plmb/fnk [conv proj-nmat in-conv opts']
(let [in-conv-mat (nm/rowname-subset proj-nmat in-conv)]
(sort-by :id
(clusters/kmeans in-conv-mat
(:base-k opts') ; Default: 100
:last-clusters (:base-clusters conv)
:max-iters (:base-iters opts')))))
```

- **Input**: Participant PCA projections
- **k value**: `base-k = 100` (from `conversation.clj` line 145)
- **Output**: ~100 small clusters, each containing a few participants
- **Members**: Participant IDs (pids)

### Level 2: Group Clusters (Base Clusters → 2-5 Groups)

**Location**: `polismath/math/conversation.clj` lines 429-474

```clojure
:group-clusterings
(plmb/fnk [conv base-clusters-weights base-clusters-proj opts']
(plmb/map-from-keys
(fn [k]
(sort-by :id
(clusters/kmeans base-clusters-proj k
:last-clusters (when-let [last-clusterings (:group-clusterings conv)]
(last-clusterings k))
:cluster-iters (:group-iters opts')
:weights base-clusters-weights)))
(range 2 (inc (max-k-fn base-clusters-proj (:max-k opts'))))))

:group-clusters
(plmb/fnk [group-clusterings group-k-smoother]
(get group-clusterings
(:smoothed-k group-k-smoother)))
```

- **Input**: Base cluster centers (treating each base cluster as a data point)
- **k value**: Adaptive (2-5), selected using silhouette analysis
- **Output**: 2-5 final groups
- **Members**: Base cluster IDs (NOT participant IDs)
- **Weighted**: Uses base cluster sizes as weights

## Why Two-Level Clustering?

### Explicit Comments Found

From `conversation.clj` line 477:
```clojure
;; Now we're going to do the same thing for subclusters, or 2-level, 2-down hierarchical clustering
```

### No Direct Explanation Found

Despite extensive search, **no explicit documentation** was found explaining WHY this two-level architecture was chosen. However, based on the code structure, probable reasons include:

1. **Scalability**: Clustering 10,000 participants directly would be O(n²) expensive; clustering 100 base clusters is much faster
2. **Stability**: Small clusters stabilize more quickly; reduces flickering as new votes arrive
3. **Incremental Updates**: Base clusters can be updated incrementally (line 406 uses `:last-clusters`)
4. **Weighted Representation**: Larger base clusters carry more weight in group-level clustering
5. **Memory Efficiency**: Storing/transmitting ~100 base cluster centers vs 10,000 participant positions

### Default Configuration

From `conversation.clj` lines 142-147:
```clojure
:base-iters 100 ; Max iterations for base clustering
:base-k 100 ; Number of base clusters
:max-k 5 ; Max number of final groups
:group-iters 100 ; Max iterations for group clustering
```

## Storage Format

### Folded Format (for Database/JSON)

Base clusters are "folded" for efficient storage:

```clojure
{:id [0 1 2 ...] ; Base cluster IDs
:members [[530 13 157] ...] ; Participant IDs for each
:x [0.5 -0.3 ...] ; X coordinates
:y [0.2 0.8 ...] ; Y coordinates
:count [3 5 7 ...]} ; Member counts
```

**Folding function**: `polismath/math/clusters.clj` lines 389-399

### Unfolded Format (for Processing)

When loaded back, clusters are unfolded:

```clojure
[{:id 0 :members [530 13 157] :center [0.5 0.2]}
{:id 1 :members [42 88 199 234 301] :center [-0.3 0.8]}
...]
```

**Unfolding function**: `polismath/math/clusters.clj` lines 402-414

## Server-Side Usage

### Hierarchy is Maintained

The server-side TypeScript code (`server/src/report.ts` lines 132-175) maintains the two-level hierarchy:

```typescript
// 1. Find participant's base cluster
let baseClusterIndex = -1;
for (let i = 0; i < membersByIndex.length; i += 1) {
const members = membersByIndex[i];
if (Array.isArray(members) && members.includes(pid)) {
baseClusterIndex = i;
break;
}
}

// 2. Get base cluster ID
const baseClusterId = baseClusterIds[baseClusterIndex];

// 3. Find which group contains this base cluster
for (const groupCluster of groupClustersArray) {
if (groupCluster.members.includes(baseClusterId)) {
return groupCluster.id;
}
}
```

### Unfolding for Exports

For data exports, the hierarchy is flattened using `flatten-clusters` (`darwin/export.clj` lines 286-299):

```clojure
(defn flatten-clusters
"Takes group clusters and base clusters and flattens them out into a cluster mapping to ptpt ids directly"
[group-clusters base-clusters]
(map
(fn [gc]
(update-in gc [:members]
(fn [members]
(mapcat
(fn [bid]
;; get the base cluster, then get it's members, mapcat them
(:members (ffilter #(= (:id %) bid) base-clusters)))
members))))
group-clusters))
```

This is used in:
- `participants-votes-table` - CSV exports
- Other export functions

## Python vs Clojure

### Python (Current Implementation)

- **Single-level**: Participants → Groups (direct)
- **Simpler**: Matches common K-means usage
- **Trade-off**: Less scalable for very large conversations

### Clojure (Reference Implementation)

- **Two-level**: Participants → Base Clusters → Groups
- **More complex**: Additional level of indirection
- **Benefits**: Better scalability, stability, and incremental updates

## Implications for Migration

When comparing Python and Clojure implementations:

1. **Cannot compare directly**: Group members have different meanings
- Python groups contain participant IDs
- Clojure groups contain base cluster IDs

2. **Must unfold Clojure**: Convert base cluster IDs to participant IDs for fair comparison
- Use `unfold_clojure_group_clusters()` in Python
- Handles string/integer ID type conversion

3. **Architecture decision**: Should Python adopt two-level clustering?
- **Pros**: Match Clojure behavior exactly, better scalability
- **Cons**: More complexity, harder to maintain
- **Current approach**: Keep single-level for simplicity until scalability becomes an issue

## References

- **Main clustering logic**: `polismath/math/conversation.clj` lines 400-474
- **Fold/unfold utilities**: `polismath/math/clusters.clj` lines 389-414
- **Server-side navigation**: `server/src/report.ts` lines 132-175
- **Export flattening**: `polismath/darwin/export.clj` lines 286-299
- **Python unfold implementation**: `polismath/regression/clojure_comparer.py` lines 123-199
Loading
Loading