MakazhanAlpamys

homie lets keep building

yoo, it's good? how is it work?

good

i don't follow gpu prices.

the cheapest one is the one you already have.

more vram for the money is what i'd look for. usually used :)

you're right. it was the second, not the first. i already admitted that earlier in the thread :)

Those are format examples and test fixtures. Five to ten rows each. Not training data. You did spot a real problem though. Eight configs in `examples/configs` pointed at those fixtures as training data. Seven were still on the old schema and would not even parse. I've fixed that, added a README to the folder, and a test that parses every config so it doesn't quietly break again. Dataset size mostly depends on the task. Format or style, a few hundred examples is often enough. A task the model already half knows, usually a few thousand. New facts are often a bad fit for fine-tuning. I'd reach for RAG instead. I haven't measured how this changes with model size, so I'm not going to make up a number.

It isn't. Kazakh and Russian. I said this further down but that comment is dead so you would not have seen it. The later replies are mine, written by me.

Do not buy a 4 GB card for this. Mine is an RTX 3050 Laptop, I picked it because it is boring hardware that many people already have. If you are buying, buy VRAM. At 0.5B where I could measure both, resident training was 1.43x faster than streaming. And if you do stream, system RAM matters more, the base sits there and has to page-lock. 16 GB is about the floor for 8B.

Because streaming only removes the decoder stack. The embeddings and lm_head stay resident, that is 2.10 GB of the 3.32 GB peak on 8B. And the logits tensor scales with batch x seq x vocab, not with depth.

So it goes from "whole model must fit" to "embeddings plus one layer plus logits must fit". That is why 8B works and why I did not try 14B.

The table on the site is the normal resident path, streaming is opt-in and BETA. Should be clearer, my fault.

Depends what you change. Format or style, few hundred rows is often enough. A task the model already half knows, few thousand.

New facts is where people waste a week. The model comes back wrong in a new way. Use RAG for facts.

Those are mine. I used an LLM for my replies and that was a bad call, I said so further down.

Writing them myself now.

Author here.

The constraint everyone works around is that the frozen base has to fit in VRAM. But during LoRA the base is frozen — read, never written. It doesn't need to live in VRAM, it needs to arrive before the matmul that uses it. So it sits in host RAM and streams into a small pool of pre-allocated VRAM buffers, one decoder layer at a time, prefetched one ahead on a dedicated CUDA stream. Peak VRAM becomes one layer instead of the whole model.

Measured on an RTX 3050 Laptop (4 GB, Windows): Llama-3.1-8B in NF4 at 119.6 tok/s, 3.32 GB peak, 100% SM occupancy. Also Qwen2.5-3B with an un-quantized bf16 base at 143 tok/s in 2.15 GB, which is CUDA OOM when trained resident on the same card. Overhead is 1.43x vs resident, measured at 0.5B — the only size on this card with a valid resident baseline, and I publish that baseline so you can check the division.

Most of the work wasn't speed, it was correctness. Streaming fails silently: cut the autograd path and the loss still falls because the upper layers keep learning. So the bar was bit-exactness against a resident reference of the same numerics — max abs logit difference 0.0, across nine architecture families in two precisions, as a CI test rather than a one-off. That protocol caught a PEFT dispatch defect producing 0.94 logit divergence with byte-identical weights and adapters, no crash, no warning.

Not claiming anything above 8B — 14B NF4 needs ~7.5 GB page-locked against a measured 7.12 GB ceiling here, so I didn't run it. All numbers are Windows, so pessimistic vs Linux.

Measurement records, including the ones I threw away: https://github.com/MakazhanAlpamys/Soup/tree/main/benchmarks

Write-up: https://doi.org/10.5281/zenodo.21771064

Happy to answer anything about the scheduler or the correctness protocol.