LLaVA-1.6: Improved reasoning, OCR, and world knowledge

Wow, this is pretty good. My sort of benchmark is a photo of someone holding my sweet little Ollie. Well, he's not so little and he's not mine anymore, but he'll always be my widdle Ollie!

Anyways, GPT4-vision wasn't always able to tell me that he doesn't really look the most comfortable being held, cause that's a lot of gravity pulling him down. Neither was Llava in the past. But with Llava 1.6 34B, it can, with no further questions asked besides "Please describe this image" as the first user message along with the image. So yeah this is really amazing! Its OCR has also definitely improved. Before, it'd just say text is in another language, but now it just shows the text. Can't wait to have a good enough computer to quickly run this locally.

To me this is the money shot:

>LLaVA-1.6 is trained with 32 GPUs for ~1 day, with 1.3M data samples in total. The compute / training data cost is 100-1000 times smaller than others.

Damn, literally a day after I wrote up my experiments[0] with LLaVA 1.5 and computing image embeddings. Interesting to see the performance with the fine-tuned Mistral-7B variant being pretty close to the one with Vicuna-13B - using Mistral 7B is what BakLLaVA did back with LLaVA 1.5.

[0] https://mildbyte.xyz/blog/llama-cpp-python-llava-gpu-embeddi...

Anyone got any fun stories to share of trying to use LLaVA e.g. to make toy robots navigate? How good is it at outputting directions in structured data, guess distances, angles, etc.?

My weekend hacking goal would be something like an RC car that can "drive to the largest plant in the room" or "go hide under the dining table" when prompted by voice. Slowly, by combining some sort of basic SLAM with still image promoting.

This looks quite promising: https://i.imgur.com/DnYWYPl.jpeg

Of course an alternative to doing it one-shot would be to collect lots of pictures + orientation for each, have LLaVA only caption them, then prompt a more generic LLM with that collected world info to pick where to go, etc.

What I like most about this AI stuff is how many neat things it makes achievable in a weekend by a motivated hobbyist that in the past required entire companies to tackle :). DIY/maker life in the AI age has been amazing fun do far.

My current test of image models is generating React code from a screenshot of the top of a HackerNews comment page. Llava-1.6 gave me this (over two responses), which is honestly not bad:

```ts

const CommentForm = () => { // State to hold the user's input const [comment, setComment] = useState(''); // State to hold the list of comments const [comments, setComments] = useState([]);

  // Function for posting a new comment
  const postComment = (e) => {
    e.preventDefault();
    // Add logic here to handle the POST request and update state
    setComments([...comments, { content: comment }]); // Assuming you want the entire object in your state
    setComment(''); // Reset the input field after posting
  };

... ```

```ts

import React from 'react'; import { CommentForm } from './CommentForm';

const App = () => { return ( <div> <h1>Comments</h1> <CommentForm /> {comments && <ul> {comments.map(comment => <li>{comment.content}</li>)} </ul>} </div> ); };

export default App; ```

Wow! You folks are making huge strides for open-source multimodal models. Thank you for all the time and effort on these as they will open up many opportunities for researchers and developers. Also, the emergent zero-shot capabilities when LLaVA-1.6 is tested against Chinese benchmarks with only English multi-modal training data are interesting and that may be a good direction for future research.

Demo: https://llava.hliu.cc/

My main interest with VLM is their ability to caption images, and this one seems very good honestly, this is going to be super useful to caption datasets.

This thought just occurred to me: would it make sense to train a model to recognise vector encoded video frames?

I completely forgot how video encoders work, but I do remember that some encode the differences between one frame and the next with vectores of the "motion" of pixels. The training data would be comprised of frames from a video with labels assigned to time ranges across the length of the video.

The we could feed a video stream to a model and it would learn to not only recognise still images, but also motion across time.

Make it fast enough and we would have near real time inference of video.

If this works, maybe an extension to this model would be to accept its previous inference result as an input to the next frame inference request. Then we'd have results like "a person entered the bright light seen of a sunny day in the country side".

Is there an OSS model that can do OCR on the level of Google Cloud Vision or Amazon Textract?

Does anyone have any information about LLaVA-1.6 being bundled as a LLamafile? https://github.com/Mozilla-Ocho/llamafile

Anyone know how much it costs to run this yourself vs paying money for GPT 4 vision?

The article says the training costs are far lower, but it doesn't say how the inference costs compare (unless I'm missing it)

I tested GPT-4V and Llava 1.6 on a Chinese text and they both hallucinate like crazy. LVMs can still barely recognize caracters while traditional OCR nails it. Do someone know why ?

Can anyone comment on the practical use of a model like this versus traditional libraries for OCR?

I’m specifically interested in processing smartphone photos of pages from an out of print book.

There's no reasoning involved with LLMs. Please. Words have meaning.