Hacker News Reader

Comment by simonw | Hacker News Reader

Would it make sense for this to offer a chunking strategy that doesn't need a tokenizer at all? I love the goal to keep it small, but "tokenizers" is still a pretty huge dependency (and one that isn't currently compatible with Python 3.13).

I've been hoping to find an ultra light-weight chunking library that can do things like very simple regex-based sentence/paragraph/markdown-aware chunking with minimal additional dependencies.

View on HN

Across a broad enough dataset (char count / 4) is very close to the actual token count in english -- we verified across millions of queries. We had to switch to using an actual tokenizer for chinese and other unicode languages, as that simple formula misses the mark for context stuffing.

The more complicated stuff is the effective bin-packing problem that emerges depending on how much different contextual sources you have.

For a Regex approach take a look at the work from Jina.ai who among other things have a chunk/tokenizer [1] and now it's part of a bigger API service [2] also they developed an interesting late interaction (aka ColBERT like) chunking system that fits certain use cases. But the Regex is enough all by itself:

[1] https://gist.github.com/LukasKriesch/e75a0132e93ca989f8870c4...

[2] https://jina.ai/segmenter/

I made a rudimentary semantic chunking in just a few lines of code.

I just removed one sentence at a time from the left until there was a jump in the embedding distance. Then repeated for the right side.

Reply on news.ycombinator.com