idonotknowwhy

Unsloth. Check their colab notebooks

You mean conversations? Just the jsonl of the standard hf dataset format to import into other systems?

I didn't realise the 5070 is slower than the 3090. Thanks.

If you want a bit more context, try -ctv q8 -ctk q8 (from memory so look it up) to quant the kv cache.

Also an imatrix gguf like iq4xs might be smaller with better quality

BetaMax had better fidelity than VHS, but initially could only hold 60 minutes of A/V while VHS could do 120m (ie, almost a full movie without swapping tapes)

I feel like this is being pushed to get more of the system controlled by the provider's side. After a few years, Anthropic, google, etc might start turning off the api. Similar to how Google made it very difficult to use IMAP / SMTP with Gmail

> Or are you saying you'll retroactively hate a song once you figure out it was AI generated?

Can't speak for OP, but for me personally, I don't mind AI augmented content, as long as it's done well. Eg. I recently played a game called "Slay the Princess", where they very clearly used Claude to write a lot of the dialogue/narration, but that didn't detract from the experience.

On the other hand, I hate it when I open a youtube video and the script is 100% chatgpt slop.

Music -- a lot of it is already made using software. If I enjoy listening to it, I don't care how it was made.

What about you?

Have you tried Deepseek-R1?

I run it locally and read the raw thought process, find it very useful (can be ruthless at times) seeing this before it tags on the friendliness.

Then you can see it's planning process to tag on the warmth/friendliness "but the user seems proud of... so I need to acknowledge..."

I don't think Gemini's "thoughts" are the raw CoT process, they're summarized / cleaned up by a small model before returned to you (same as OpenAI models).

> It's not even per token. The routing happens once per layer, with the same token bouncing between layers.

They don't really "bounce around" though do they (during inference)? That implies the token could bounce back from eg. layer 4 -> layer 3 -> back to layer 4.

> tariff on our closest allies

I saw that your president said:

> “We imported $3b of Australian beef last year alone… they won’t take any of our beef because they don’t want it to affect their farmers.”

Just FYI, the reason we don't import much beef is because we have about as many cows as we do people here :)

It depends what you're trying to do.

Coding - Mistral-Small-2503 or Qwen2.5-32b-Coder Reasoning - QwQ-32b Writing - Gemma-3-27b is good at this. etc

> Like there must be a size too small to hold "all the information" in.

We're already there. If you running a Mistral-Large-2411 and Mistral-Small-2409 locally, you'll find the larger model is able to recall more specific details about works of fiction. And Deepseek-R1 is aware of a lot more.

Then you ask one of the Qwen2.5 coding models, and they won't even be aware of it, because they're:

> small, specialized models.

> But maybe training compute will just get to the point where we can run a full-featured model on our desktop (or phone)?

Training time compute won't allow the model to do anything out of distribution. You can test this yourself if you run one of the "R1 Distill" models. Eg. If you run the Qwen R1 distill and ask it about niche fiction, no matter how long you let it <think> for, it can't tell you something the original Qwen didn't know.

Holy shit, did they just make every photo taken from an iPhone (AI enhancement) public domain? And spellchecker for text?

What about movies like Deadpool3, where AI wrote part of the script?

The multimodal models aren't good for this. Refusals aren't the issue (they're fine with BERSERK, though occasionally they'll refuse for copyright). The issue is the tech isn't there yet.

You'll want to use custom models to segment the manga (panels, speech bubbles), OCR the text, translate (gemma punches above it's weights for this part).

That said, I've been experimenting with using Pixtral to do the analysis part with okay-ish results (providing individual panels with the character names) but it'll still mix up the characters when they're drawn differently.

> I'm not a big believer in abliteration, it seems to always hurt performance.

Agreed, it's fun to play with but it increases halucinations. And for creative writing, it makes the model write more compliant characters (they'll give in too easily during negotiations, rather than refuse, etc)

Could probably be improved with more targeted abliteration.

Replace "google" with "unsloth" in the browser address bar if you want to download them without signing up to hf

Thanks a lot for the v2.5! I'll give that a whirl. Hopefully it's as coherent as v3.5 when quantized so small.

> I can't quantify the difference between these and the full FP8 versions of DSR1, but I've been playing with these ~Q2 quants and they're surprisingly capable in their own right.

I run the Q2_K_XL and it's perfectly good for me. Where it lacks vs FP8 is in creative writing. If you prompt it with for a story a few times, then compare with FP8, you'll see what I mean.

For coding, the 1.58bit clearly makes more errors than the Q2XXS and Q2_K_XL

Oh right, yeah I've done things like this (phone calls to ChatGPT) or the openwebui Whisper -> LLM -> TTS setup. I thought there might be something more than this by now

Yeah, I've had similar experiences. I still hesitate if it's a field I don't know too well of course (never trust an LLM), but R1 has been able to solve things I've been stuck on. And watching it's <think></think> process has been insightful. Only issue is that it ties up all my GPUs while I run it.

Hopefully Mistral can copy their technique and give us a 123b reasoning model.

> This is missing the most interesting changes in generative AI space over the last 18 months

I agree, though personally I'm liking the "big thing" as well. R1 is able to one-shot a lot of work for me, churning away in the background while I do other things.

> Multi-modal

IMO this is still early days and less reliable. What are some of your daily use cases?

> Context lengths

This is the biggest thing IMO (Models remaining coherent at > 32k contexts)

And whatever improvements have caused models like Qwen2.5 to be able to write valid code reliably vs the GPT-4 and earlier days.

There are a whole lot of useful smaller niche projects HF like extracting vocals/drums/piano from music, etc

Yeah I'm surprised by all the negativity as well. I'm listening to the post right now (using xtts-v2 finetuned on a voice I like lol). Sounds like these companies are overvalued / over hyped. Maybe they are / some of these companies go the way of myspace, but LLMs are incredibly useful for me.

I'm able to do a so much more using LLMs (Mistral-Large, Qwen2.5 and R1 locally, Claude via API) than without them.

I have to get the IDE setup properly now.

Did you just ls my /workspace dir? Lol

Intel had a great P/E a couple of years ago as well :)

> Unless something radically changed in the last couple years, I am not sure where you got this from?

This was the first thing that stuck out to me when I skimmed the article, and the reason I decided to invest the time reading it all. I can tell the author knows his shit and isn't just parroting everyone's praise for AMD Linux drivers.

> (I am specifically talking about GPUs for computer usage rather than training/inference)

Same here. I suffered through the Vega 64 after everyone said how great it is. So many AMD-specific driver bugs, AMD driver devs not wanting to fix them for non-technical reasons, so many hard-locks when using less popular software.

The only complaints about Nvidia drivers I found were "it's proprietary" and "you have to rebuild the modules when you update the kernel" or "doesn't work with wayland".

I'd hesitate to ever touch an AMD GPU again after my experience with it, haven't had a single hick-up for years after switching to Nvidia.

Deepseek was there side project. They had a lot of GPUs from their crypto mining project.

Then Ethereum turned off PoW mining, so they looked into other things to do with their GPUs, and started DeepSeek.

Most people running local inference do so thorough quants with llamacpp (which runs on everything) or awq/exl2/mlx with vllm/tabbyAPI/lmstudio which are much faster to than using pytorch directly

I've bought 5 used and they're all perfect. But that's what buyer protection on ebay is for. Had to send back an Epyc mobo with bent pins and ebay handled it fine.

For me personally, hacking together projects as a hobbiest, 2 reasons :

1. It just works. When i tried to build things on Intel Arcs, i spent way more hours bikeshedding ipex and driver issues than developing

2. LLMs seem to have more cuda code in their training data. I can leverage claude and 4o to help me build things with cuda, but trying to get them to help me do the same things on ipex just doesn't work.

I'd very much love a translation layer for Cuda, like a dxvk or wine equivalent.

Would save a lot of money since Arc gpus are in the bargain bin and nvidia cloud servers are double the price of AMD.

As it stands now, my dual Intel Arc rig is now just a llama.cpp inference server for the family to use.

He could probably do 12b (Nemo) up to 14b (Qwen 2.5) at 4bpw with exllamav2

What did you switch to?

I switched to Manjaro from Arch in 2017 because I don't have time to debug / fix broken updates, and the same Manjaro install has been completely stable since then.

Is there another distro which can do this (and I also don't want to have full reinstalls / upgrades every few years like Ubuntu/Mint)

It's because oai are going for low latency. I've got pause free working locally by running the tts model after the llm has finished generating a huge chunk of the response.

And fine tuning on 15 minutes of the voice you're targeting makes it sound natural

And they don't interrupt each other