HNHacker News
TopNewBestAskShowJobs

snyy

247 karma · joined February 11, 2024

Founder, Feyn AI. www.usefeyn.com
submissionscomments
snyy··on Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking
We don't yet, but our library comes with a visualization tool that you can use to compare chunkers directly. https://docs.chonkie.ai/python-sdk/utils/visualizer
snyy··on Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking
Pretty much what you described. Convert the PDF to Markdown, join content across pages so that its all one string, then chunk it. Our evals show this approach works best.
snyy··on Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking
> you may want to remove errors while retaining the correction

Double clicking on this, are these messages you’d want to drop from memory because they’re not part of the actual content (e.g. execution errors or warnings)? That kind of cleanup is something Chonkie can help with as a pre-processing step.

If you can share an example structure of your message threads, I can give more specific guidance. We've seen folks use Chonkie to chunk and embed AI chat threads — treating the resulting vector store as long-term memory. That way, you can RAG over past threads to recover context without redoing the conversation.

P.S. If HN isn’t ideal for going back and forth, feel free to send me an email at shreyash@chonkie.ai.

snyy··on Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking
Awesome! Keep us posted :)
snyy··on Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking
Great questions!

Chunking fundamentals remain the same whether you're doing traditional semantic search or agentic retrieval. The key difference lies in the retrieval strategy, not the chunking approach itself.

For quality agentic retrieval, you still need to create a knowledge base by chunking documents, generating embeddings, and storing them in a vector database. You can add organizational structure here—like creating separate collections for different document categories (Physics papers, Biology papers, etc.)—though the importance of this organization depends on the size and diversity of your source data.

The agent then operates exactly as you described: it queries the vector database, retrieves relevant chunks, and synthesizes them into a coherent response. The chunking strategy should still optimize for semantic coherence and appropriate context window usage.

Regarding your concern about large DB records: you're absolutely right. Even individual research papers often exceed context windows, so you'd still need to chunk them into smaller, semantically meaningful pieces (perhaps by section, abstract, methodology, etc.). The agent can then retrieve and combine multiple chunks from the same paper or across papers as needed.

The main advantage of agentic retrieval is that the agent can make multiple queries, refine its search strategy, and iteratively build context—but it still relies on well-chunked, embedded content in the underlying vector database.

snyy··on Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking
In addition to size and speed we also offer the most variety of chunking strategies!

Typically, our current users fall into one of two categories:

- People who are running async chunking but need access to a strategy not supported in langchain/llamaIndex. Sometimes speed matters here too, especially if the user has a high volume of documents

- people who need real time chunking. Super useful for apps like codegen/code review tools.

snyy··on Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking
Thank you :)
snyy··on Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking
Not sure what HN uses :)

If you want agents/LLMs to be able to find relevant data based on similarity to queries, vectorDBs like Chroma (or even pgVector) are great.

snyy··on Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking
Node and Bun should work. Haven't tested on Deno yet.

We rely on the huggingface/transformers library which might be too heavy for a react-native app.

snyy··on Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking
As mentioned in the other reply, we have a cloud/on-prem offering that comes with a managed ETL pipeline built on top of our OSS offering.
snyy··on Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking
Awesome! Keep us posted!
snyy··on Show HN: Advanced Chunking in JavaScript/TypeScript with Chonkie
We don't have this yet but we will soon. Finding the right setup for your data is definitely tougher than it needs to be.
snyy··on Show HN: Advanced Chunking in JavaScript/TypeScript with Chonkie
We're aiming to launch it by Monday
snyy··on Show HN: Advanced Chunking in JavaScript/TypeScript with Chonkie
Glad you like it!
snyy··on Show HN: Chonkie Cloud – No-nonsense chunking now on the the cloud
When you sign up for Chonkie, you’ll receive a bundle of free credits to get started. These credits are used as you process documents, based on the pricing listed at chonkie.ai/pricing.

Once your credits run out, no worries—Chonkie keeps working! You’ll continue to be billed based on usage, so there’s no interruption in service.

Credits are refreshed at the start of every billing cycle.

snyy··on Show HN: Chonkie Cloud – No-nonsense chunking now on the the cloud
Glad it helps! :)
← PreviousPage 2 of 2