Is there a faster alternative to Python's built-in set.intersection() for large datasets?

[codehaven175]

??

[Data-Unix]

For general Python objects, set.intersection() is already about as fast as it gets: it's implemented in C, and when both operands are sets CPython loops over the smaller one and does a hash lookup in the larger one, so it costs roughly O(min(len(a), len(b))). The gains usually come from how you call it, or from changing the data representation:

1. Make sure both sides are sets

a.intersection(some_list) has to walk through the whole list. Convert once and reuse it, instead of rebuilding sets inside a loop:

b = set(big_list)          # once
for a in many_sets:
    common = a & b

2. Many sets: start with the smallest

common = set.intersection(*sorted(sets, key=len))

The intermediate result can only shrink, so starting with the smallest keeps every step cheap.

3. Don't build the result if you don't need it

If you only need to know whether they overlap, a.isdisjoint(b) stops at the first common element.

4. Change the representation for big numeric data

• NumPy: if the data is already in arrays, np.intersect1d(x, y, assume_unique=True) avoids creating millions of Python objects. It sorts (O(n log n)), so measure it against sets for your data.
• Integers in a known range: bitmaps are much faster and smaller. Use pyroaring (Roaring bitmaps, BitMap(a) & BitMap(b)), or even a plain Python int as a bit set for small ranges.
• Already sorted data / streams: a two-pointer merge does it in O(n + m) without hashing and with constant memory.
• Tabular data: pandas.Index(a).intersection(b) or an inner merge.

Benchmark with timeit on your real data before switching. For most cases, points 1–3 are all you need.