valine

It gets progressively bigger and louder the longer you ignore it. After 30ish seconds it sounds an alarm and kicks you out.

All of it can be reused. Their plan is for the tower to set the booster back down and immediately relaunch it.

The landing legs are heavy. SpaceX would rather have more payload than carry legs to space on every launch.

Also landing on the launch pad means you don’t have to transport the rocket. Just have the arms set it down and you’re ready to launch again.

Not really the first paper is just fine-tuning on synthetic data. The second paper doesn’t optimize the model weights.

This period of model scaling at all cost is going to be a major black eye on the industry in a couple years. We already know that language models are few shot learners at inference time, and yet OpenAI seems to be happy throwing petaflops of compute training models the slow way.

The question is how can you use in-context learning to optimize the model weights. It’s a fun math problem and it certainly won’t take a billion dollar super computer to solve it.

Demonstrating good performance from a non-transformer based architecture is cool. I agree though these particular models aren’t that useful given the current landscape.

I think the intent here is probably to justify training a larger 400B model with their architecture.

That’s a strange perspective. AEB isn’t 100% reliable but we still allow it on the roads because it’s a net increase in safety. When used correctly FSD seems to be a net increase in safety as well. You can make the case they should do a better job with the labeling, maybe stop calling it FSD to start.

I did, was about to delete it but then you commented haha.

Edit: replied to wrong comment

Reflection 70B is a scam. The creator was just routing requests to Claude.

https://old.reddit.com/r/LocalLLaMA/comments/1fc98fu/confirm...

The model performance is driven by chain of thought, but they will not be providing chain of thought responses to the user for various reasons including competitive advantage.

After the release of GPT4 it became very common to fine-tune non-OpenAI models on GPT4 output. I’d say OpenAI is rightly concerned that fine-tuning on chain of thought responses from this model would allow for quicker reproduction of their results. This forces everyone else to reproduce it the hard way. It’s sad news for open weight models but an understandable decision.

A single document doesn’t contribute much to the loss during base training. LLMs can absolutely memorize text if it’s duplicated in the training set, not so much if the document in question is obscure.

NASA wanted to understand if the malfunctioning thrusters could operate within a margin error for safety. Boeing spent a month running a study that produced non-definitive answer as to the root cause of the problem. Had Boeing definitively demonstrated the root cause of the thruster issue NASA probably would have sent the Astronauts home on Starliner. It’s not that NASA necessarily thought that Starliner was unsafe. It’s more like they lack confidence that Boeing actually understands the problem.

This is such a natural extension to LLMs. I’m shocked it hasn’t been tried before.

When I ask a diffusion model to generate a chessboard, I’d expect the pieces to be placed randomly. We are getting closer to image generators that not only know what chess pieces look like but also where to place them.

Just because you have the dataset doesn't mean you can generate a reference. Let's say I hand you a potato salad recipe and a copy of the entire internet. Say you somehow extract all potato salad recipes from the dataset (non trivial btw) and none of them are an exact match for the recipe the model generated. Now what?

If you have the model weights you have roughly the same opportunities as the company that trained the model. The code you need to run inference on the Llama weights is very much open source. The only thing you're missing out on is the training code, which is prohibitively expensive to run for most anyways. Open source training isn't going to give you any unique insights into the "digital brain library" of your model.

Also just to be clear, if you want to set up a RAG with an open weight model and a large dataset there's nothing stopping you. Download Red Pajama and Llama and give it a try.

https://github.com/togethercomputer/RedPajama-Data

Not sure what this has to do with anything. The paper I was commenting on is using diffusion models to parse raw light hitting the sensor as an alternative to a glass lens. No one wants an image generation model hooked up to weather data— that’s kind of ridiculous.

Not sure I understand the question. This paper is about using diffusion models to reconstruct usable images from raw sensor data. The diffusion model in essence replaces the lens.

I get the feeling that lens free cameras are the future. Obviously the results here are no where near good enough, but given the rapid improvement of diffusion models lately the trajectory seems clear.

Would love to lose the camera bump on the back of my phone.

The codebase to do the training is way less valuable than the weights for the vast majority of people. Releasing the training code would be nice, but it doesn't really help anyone but Meta's direct competitors.

If you want to train on top of Llama there's absolutely nothing stopping you. Plenty of open source tools to do parameter optimization.

LLMs are bad at counting things just in general. It’s hard to say whether the failures here are vision based or just an inherent weakness of the language model.

We don’t store color data in full precision usually. People aren’t sensitive to all colors equally, the less sensitive the eye is to a particular color the more efficiently you can store it.

You can also typically discard high frequency data in your color channels. So not only are you not storing full precision data, you also don’t need to store roughly a third of the frequency bins. You can think of this like cropping the image but in the frequency domain. The data saving are identical to cropping the image but visually imperceptible.

Or alternatively a lot of energy is wasted answering simple questions.

The whole point of the transformer is to take words and iteratively, layer by layer, use the context to refine their meaning. The vector you get out is a better representation of the true meaning of the token. I’d argue that’s loosely akin to ‘understanding’.

The fact that the transformer architecture can memorize text is far more surprising to me than the idea that it might understand tokens.

It’s a neat approach. VRAM is obviously the major concern here since it sounds like the parameter count grows as you insert more facts.

I’m also curious how well the facts communicate with each other. A major problem with RAG is that the model can’t draw novel inferences between documents. You have a model with a broad knowledge base but a very shallow understanding of the connections between datapoints. Let’s say I insert two facts separately using this method: “the cookie is under cup A” and “cup A is moved to position 2”. If I ask where the cookie is can this new model tell me? This is trivial for in-context learning and non-trivial for RAG. Curious where this solution falls.

No one knows how it will all shake out. I'm personally skeptical scaling laws will hold beyond GPT4 sized models. GPT4 is likely severely undertrained given how much data facebook is using to train their 8B parameter models. Unless OpenAI has a dramatic new algorithmic discovery or a vast trove of previously unused data, I think GPT5 and beyond will be modest improvements.

Alternatively synthetic data might drive the next generation of models, but that's largely untested at this point.

Llama 3 8B captures that 'magic' fairly well and runs on a modest gaming PC. You can even run it on an iPhone 15 if you're willing to sacrifice floating point precision. Three years from now I full expect GPT4 quality models running locally on an iPhone.

Happy to see the control center change. Look up for control center is a bad experience.

Llava1.6, IntenVL, CogVLM2 can all do OCR with nothing but tiled image embeddings and an LLM. Feeding in OCR results from tesseract improves the reliability of the transcript, especially for long strings of random characters, but it’s not strictly necessary for the model to read the text out of the image.

Clip embeddings can absolutely “read” text if the text is large enough. Tiling enables the model to read small text.

Very curious how it performs on OCR tasks compared to InternVL. To be competitive at reading text you need tiling support, and InternVL does tiles exceptionally well.

The final layer of the transformer prior to the logits pretty much does what you want already. The KV of the final layers are taken into account when generating the final hidden state for your new token. The first layers of the model are really just contextualizing the token within the sentence, forcing hidden state from a higher layer through lower layers isn't going to help you much. Replacing the last 5 or so layers with an RNN could certainly be interesting though.