Used Kino yesterday head to head with claude-code directly to make a simple video. Kino was way easier to nudge and control. Props to you guys for building something that "just works". You rarely see that when you try anything in software.
324 karma · joined July 20, 2020
sahil@linum.ai
Empirically they learn an approximation to the derivative of the likelihood (Ie the score function). So we have a bound on the true likelihood distribution, but don’t have exact likelihoods (which we do have in LLMs).
In practice this bound is too loose to be used for train with GRPO effectively (though folks have tried).
The linear patchificaion is per patch, so if it works on 360p 2 seconds (really short low res) it should adapt naturally (given ROPE3D). The final layers of the DiT start moving us from latent space to pixel space.
Net net the compression is as much as H3 (32x32xTime) so it should be comparable to the best VAEs in terms of expense
Very related; our models are quite literally cousins, as we’re both interesting on the work of JiT from last fall.
Architecture is roughly the same up to some minor differences (they use MM-DiT from FLUX, we use single stream from Z-Image).
The biggest delta is that we’re going with 32x32 token reduction and they go for 16x16. Our goal here was to cut attention windows 4x to speed up training inference. And have been publishing work (this included) on how to match models with less compression to than us.
In this blog specifically there are 3 things we stack on top of each other to get good results:
1) transformer blocks per modality before they enter shared single stream transformer blocks
2) very different noise schedules during training than those proposed in prior work
3) predicting the output at k different resolutions along the depth of the diffusion transformer to provide gradient signal earlier in the network to learn better (and faster)
The other work does a great job scaling up the JiT to 3B parameter; this is purely an experimental release highlighting how we make up the gap.
The jumping point for this no-VAE work was three-fold:
1) Our goal here is to get 32x32 token reduction to make video training and inference downstream cheaper. To date, the best open-weight Image VAEs like Flux-2 seem to cap out at 16x16 token reduction (8x8 VAE + 2x2 linear patchification). Others like H3 have pushed to 32x32 reduction but requires them swapping out the small VAE decoder with a 2B parameter decoder. So, this is a foray to get 32x32 compression without compromising quality.
2) We believe that end-to-end trained networks will tend to perform better than modularly trained networks (e.g. VAE + DiT). This hypothesis comes from work like REPA-E, where authors are able to get much better results by backpropagating through the pretrained VAE. The latent space for perception / reconstruction seems to have a different "optimal" configuration than a latent space specifically built for generation. That's why we liked the idea of trained E2E here.
3) There's work with VAEs that show that providing additional modalities (e.g. text captions) can help the VAEs improve as well. That's natural to this construction, so we thought language might actually help with the compression, not hurt. To be honest, this this is the most hypothetical of three ideas; and definitely warrants specific ablations.
From our POV, this quite an experimental checkpoint. That's why we've dubbed it a research release. Our goal is to share our findings (and checkpoints) along the way to a legit image and video release for Linum v3.
Image release should be out in the coming weeks, so stay tuned for that. Video will take a little longer (we're doing on-going research atm on pixel-space video training).
If we have extra compute lying around in the coming weeks, we’ll try this out and report back. It’s a good idea :)
Can you confirm I'm understanding correctly?
1) Linear unpatchify as usual to go from hidden states to pixel space
2) Attention within a local window (e.g. 3x3, 5x5) to "blend" pixel space data and come up with a better image (as an alternative to MLP or Convolution)
And follow up questions:
1) How do you handle boundaries between your "attention windows"? Do you move the window just like a convolution does or are the "attention windows" all mutually exclusive from one another?
2) How much faster/slower is this operation vs. a linear layer + MLP?
GenEval and GenEval2 prompts are very terse (~10 tokens) while these models are primarily trained on long-dense captions. So, it's pretty common to upsample prompts before running them through benchmarks. That way you're testing the model as it was trained, rather than testing it on prompts that are out of distribution.
If you look at the Qwen-Image Report before they enhanced it for their 12-25 release, upsampled prompts score 0.87 on GenEval* In this paper they take it from 0.74 to 0.81.
To me that reads to me that they're basically getting the model to adapt to the benchmarks' terse prompts. And it's unclear to me if simply finetuning the model on shorter prompts would work just as well as the RL solution.
For context, we're a 2-person lab training generative video models. Goal is a new set of controllable, animation tools (you can read more about that here https://www.linum.ai/about if you're curious).
The biggest bottleneck for our last text-to-video model in terms of training and inference cost is attention. Video models are incredibly token dense (e.g. 110K tokens for a several second clip). If we can condense that context window more aggressively, we can train bigger models for a lot less $$ and offer them to prosumers at reasonable price points (unlike the big models today like Seedance, which cost an arm and a leg to run).
Traditionally, image and video models have two disjoint components: VAE (Variational Autoencoder) and Diffusion Transformer (DiT). They're trained separately, and empirically VAEs seems to struggle to get past 16x16 token reduction.
Here, we're switching to pixel-space, throwing away the VAE, and achieving 32x32 token reduction (4x smaller context windows) while learning a better overall model in a fraction of the training samples.
The central thesis is "simpler is better". If we can put the compression problem into the more powerful Diffusion Transformer (DiT) would should be able to learn a "latent space" optimized for generation and get better compression without hurting generation quality.
I'll be checking this post off and on the next couple of hours, so feel free to drop questions below. And I'll try to answer them to the best of my ability.
P.S. The model checkpoints from this blog are Apache 2.0, so feel to try playing with it yourself on a GPU!
For us specifically, we want to target prosumers who aren’t gonna pay $1 for a few seconds of footage. It has to be 1/50th of the cost of the big guys. It helps that our ultimate focus/niche is animation, so we can get away with smaller models.
But to answer question more broadly, I think you have to ask the questions:
1) How will this lab front run the hyperscalers? Is there someone thing they’re doing (or if executed correctly) could break right such that they could have a hook for customers to use the product over hypscaler (eg crazy low cost of inference)?
2) If this lab lands the hook and gets small lead how could they maintain and grow it? This doesn’t have to be technical per say. It could be a clear land and expand sales motion at the enterprise level, for example.
Startup history is littered with examples of companies that seemed too close to a big guy to sprout up in the first place (eg stripes adjacency to PayPal).
From the outside looking in it might seem likes there no space for the lab to bloom; so you’ll have to bring the specifics of the lab and your on the ground expertise of the space / company’s situation to come to an answer for yourself.
When it comes to multi-resolution training (e.g. matryoshka training), there are precedents that don't require this AR formulation.
They cite MAR (https://arxiv.org/pdf/2406.11838), which I think is a much clearer articulation of "autoregressive diffusion". MAR uses an autoregressive base (like an LLM) and staples on a small MLP on-top that's trained as a diffusion head. That makes more sense to me, since you can leverage the "knowledge prior" from an LLM and have it generate images. That's probably how nano-bannana and GPT-Image broadly work.
Zooming out, it's not clear to me from any of the work WHY autoregressive diffusion on it's own is better than regular diffusion. Most autoregressive diffusion models use speculative decoding, because pure autoregressive diffusion is too slow to run at inference time.
The one ~magical~ thing about auto-regressive diffusion (IMO) has nothing to do with autoregressive vs. fully-bidirectional diffusion. But simply, the face that you can do it on-top of a LLM. That means the LLM can look at it's generation, assess it's quality, think on what's broken, and then call itself to edit the image and fix it. With methods like RLVR, that means you can essentially guarantee the correctness of your image (along the axes you've RLVR-ed).
When it comes to inference it should also be cheaper (since we’ve cut attention sequence length by 4x), but I don’t have a hard number for you how much cheaper for inference
There’s been more recent work on continuous space diffusion models for language this year. Sander Dielman has a good blog post on that.
But if I’d predict where diffusion lands in LLMs, it’ll be used in looped models like Astra. Once reasoning is happening in hidden states, we’re in a good continuous domain, perfect for diffusion. We’re going to end up swapping “looping” for predicting the models hidden states at the next “timestep” with diffusion. And that way the time to generate traces (no longer human intelligible though) will become 10x faster.
I think ultimately the folks with the purses won’t care enough about a for it to be taken seriously, even if it’s an engineering bottleneck.
B definitely has scope but still smaller than I’d expect. When I was an intern at yelp, I was migrating us off internal credit card management to Braintree/stripe. No one would have imagined outsourcing that in early 2000s. There’s a long tail of stuff that you can’t sent to a 3rd party; but for most use cases it’ll suffice.
For c, true; but this is expensive for everyone, including the Chinese model companies that are trying to undercut Open Ai / anthropic. In the nth degree, i think the field will bring the cost down to the place where it’s manageable (see the existence of the cheap Chinese models). The real question is the $$ spent on pushing the research forward at scale.
This feels reminiscent of the big push to RAG a few years ago. And, more broadly the skunkworks projects that big companies tout in the press before they end up killing, when the operational overhead becomes too much for their liking.
Ultimately, the narrative is good for the consumer and the enterprise. It’ll mean OpenAI and anthropic will have to keep prices low. But ultimately, in the course of the next 10 years, I don’t see enterprises wanting to do this themselves. It’ll just be simpler (and eventually safer in their eyes) to send traffic to the big labs.
Since DeepSeeks GRPO, they’ve been improvements as well like AliBabas GSPO that have gotten wide adoption. Again iterations
Moreover,I think it’s impossible to know if you’re hitting a portion of the sigmoid, because there will often be an idea that changes the trajectory altogether.
In 2024, there was a ton of talk about the plateau. Reasoning was an iteration on chain of thought, but it didn’t really work. Deepseek proposes RLVR as a way to get around the lack of $ they have to produce human reasoning trace data. That small iteration catches the eye of OpenAI and Anthropic, turns out to be way more important than even DeepSeek could have ever expected when it comes to improving LLMs for coding, and last 18 months have been an exercise on riding that insight to the nth degree.
That one small iteration brought us a lot of progress. Now we’re seemingly exhausting the impact of that one insight, but there may be another soon enough.
Only reason I can think why the LLM might still work better here is that it’s trained to solve a bunch of different image/video related questions, so it’s perceptual modules may be more robust adaptive for this aesthetic grading task versus something like LingBot
In the original Stable Diffusion, CLIP doubled as part of this LAION Aesthetic Filter as well as the text control for the generative model. But in the last couple of years, the field has drifted away from CLIP towards LLM hidden states (e.g. Qwen, Mistral, etc.). It turns out that text-to-image and text-to-video models do not need the visual prior from CLIP; instead the hidden states of multimodal LLMs act as better conditioning mechanisms.
But you are on-to something when it comes to leveraging pre-trained vision models as priors in generative image/video.
Nowadays, models like Dino-v3 (which have the strong spatial priors, you're talking about with LingBot) are being used in 3 distinct ways:
1) REPA: The DiT in the generative model minimizes cosine distance to another vision prior, like Dinov3. This paper from earlier this year, iREPA (https://arxiv.org/pdf/2512.10794) digs into WHAT specifically makes a good regression target for generative image.
2) VA-VAE: Most generative image/video models work in a latent space that's pre-trained separately. We wrote a lot about this in an earlier blog post (https://www.linum.ai/field-notes/vae-reconstruction-vs-gener...). But basically, VAEs that overfit to the reconstruction task create brittle latent spaces for generative models. One solution folks have figured out is to regress to a strong spatial prior like Dino, during VAE training. This is discussed at length in the VA-VAE paper(https://arxiv.org/abs/2501.01423). Recent image models like Flux 2, likely achieve their improvements through this type of VAE training.
3) RAE (Representation Autoencoder): This is the latest/greatest right now. Where generative image models are trained directly in Dino-v3 space; and then they just train a decoder to read back out into pixel space. (https://arxiv.org/pdf/2605.18324)
Hopefully this helps!
For the LAION classifier specifically, it's trained on-top of CLIP. The bottleneck for accuracy isn't the linear/non-linear readout, it's CLIP's specific knowledge of the world.
CLIP was trained with a contrastive learning objective, where all it has to learn is that images of the same category are closer to each other than images of other categories (e.g. two items are both dogs so they should be closer semantically, while say a dog and a computer are different and should be farther semantically.
As a result CLIP is really good at object definitions, but struggles at pretty much everything else. Modern multimodal LLM's perceptual modules are much stronger than CLIP; that's why we chose fine-tuning a LLM rather then training a MLP on-top of CLIP (which would be a direct iteration on the LAION aesthetic classifier).
In the past year, I’ve had 3 or 4 times when someone has “called me out” for using AI when I’m just an em dash organically.There’s no discussion of what I wrote; just immediate anger.
It feels like giving in, but I might have to edit my writing style just because of the visceral reactions