Hacker News Reader

Comment by seabass | Hacker News Reader

When they recommend using smaller chunks, how small in general are we talking? One sentence? One paragraph? 100 tokens or 1000? While I understand that the context of the data is important, it’s hard for me to ground vague statements like that without a concrete realistic example. I’m curious what chunk sizes people have found the most success with for various tasks in the wild.

View on HN

think this way: you are going to return the chunk to your user as 'proof'.

the size therefore depends of your content style.

E.g. for a HN discussion, I would go 'paragraph' of each comment.

in a contract, each clause.

in a non-fiction book, maybe each section...

you can also decide to do some kind of reverse adaptive tree: you chunk at sentences level, then compare, if 'close enough', you merge them into a bigger chunk

Suppose you don't know the shape of the content ahead of time. This would be the case for most apps that allow users to upload their own sources.

It sounds like (at the expense of more computation and time) the reverse adaptive tree approach you described would be ideal for those scenarios.

that's why the article talks about using machine learning to decide of best strategy: to deal with unknown content 'shape'

When using vector search, you want chunks in the ~1k tokens range. If you're using full text search then chunks should probably be smaller, say a few paragraphs at most. If you use trigrams, you want chunks to be short, maybe even sentence level.

I’ll share my strategy. I usually keep chunks at the max of 0.5% of a document or 270 tokens. Multiple that by three and those are the size of the sliding windows that are then worried.

Why not overlapping sizes? (1), (1,2), (1,2,3) Sometimes the match is in a single sentence. Sometimes in the full paragraph. Sometimes across two paragraphs. If, in your top n results, some of these items overlap, you use the greater unit. And you slide these windows.

Also, I wouldn’t necessarily use a “sentence” as the lower bound, since that can be something like “Yes.”

Reply on news.ycombinator.com