Boriscii

Also, please feel free to take a look at the blog post if you are interested in more technical details!

https://blog.bynedocs.com/layoutlm-byne-v0.1-the-new-sota-in...

There is a growing trend of applying Multimodal LLMs directly to PDF pages without doing extraction first (i.e. classic RAG). We've had this request from multiple customers whose products are supposed to handle visually rich documents like "GPT for pitch decks" or "GPT for investment bankers". The problem is that you must first find the relevant page to feed into the LLM. This helps to avoid the overhead of running an expensive model on each single page.

We first attempted to extract text from pages and do a semantic/hybrid search. Still, it felt massively suboptimal since much positional information was lost, so we decided to train our model that could help with this task!

With ZeroX that launched a month ago and grew to 1.2K stars, it's clear that using multimodal LLMs to parse documents as images is the new way to go. We were trying to add a pipeline like this to our service, but were quite challenged by the most important step: retrieval. MiniCPM-Llama3-V-2_5 can answer about 95% of questions correctly based on a document page, but it first needs to be fed the right pages.

We attempted to parse the pages into text and run embedding models on them. While it worked, the results were suboptimal since the models often missed important context inside pages, especially in visually rich documents. So we decided to train the first embedding model that ingests not only the text, but also positional information to improve its understanding of content hierarchy on the page. It's still in alpha and we haven't fully trained it yet, but we are looking for feedback and ideas!