filters = {}
candidates = []
for i in range(len(pi)):
block = pi[i:i+10]
hash_prefix = block[:4]
try:
bloom = filters[hash_prefix]
except KeyError:
filters[hash_prefix] = BloomFilter().add(block)
candidates.append((i, block))
continue
if block in bloom:
candidates.append((i, block))
else:
bloom.add(block)
Basically, make one pass through to build a list of statistically likely candidates to evaluate later. Then use full string matching on those candidates to find the real matching sequences.That might be helpful here because Bloom filters can say "we might have seen this string before" or "we definitely have not seen it before". That may greatly reduce the amount of storage you need.