HNHacker News
TopNewBestAskShowJobs

brucethemoose2

7,874 karma · joined February 13, 2023

submissionscomments
brucethemoose2··on Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
In theory quantized weights of smaller models are under a gigabyte.

If you are looking for megabytes, yeah, those "chat" llms are pretty unusable at that size.

brucethemoose2··on Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
Yeah, especially with how much Google is hyping it.

It could have been long context? Or a little bigger, to fill the relative gap in the 13B-30B area? Even if the model itself was mediocre (which you can't know until after training), it would have been more interesting.

brucethemoose2··on Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
This is not necessarily a production backend, as it mentions in the readme.

There are some very interesting efforts in JAX/TPU land like https://github.com/erfanzar/EasyDeL

brucethemoose2··on Jim Keller criticizes Nvidia's CUDA, x86
If you are implying we are at the mercy of a few CUDA experts because we can't efficiently write it ourselves, you would be correct.

Triton does seem way more approachable as something I'd like to pick up.

brucethemoose2··on Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
The code is a basically irrelevant fraction of the model weights. The raw FP16 is like 17GB.

In practice your priority would be fancy quantization, and just any library that compiles down to an executable (like this, MLC-LLM or llama.cpp)

brucethemoose2··on Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
...Also, we have eval'd Gemma 7B internally in a deterministic, zero temperature test, and its error rate is like double Mistral Instruct 0.2. Well below most other 7Bs.

Was not very impressed with the chat either.

So maybe this is neat for embedded projects, but if it's Gemma only, that would be quite a sticking point for me.

brucethemoose2··on Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
Not to be confused with llama.cpp and the GGML library, which is a seperate project (and almost immediately worked with Gemma).
brucethemoose2··on Jim Keller criticizes Nvidia's CUDA, x86
> "Basically nobody writes CUDA,"

Quite a thing to say as head of an AI hardware startup.

Most of the machine learning libraries we use/test in production (and that I use personally on my desktop) have hand written, extensively optimized CUDA kernels. It would be problem #1 switching to Tenstorrent hardware.

Not that I like that one bit, or that I really understand the optimization of these libraries. I do see triton code as well, but it still seems to be the suboptimal path.

brucethemoose2··on Phind-70B: Closing the code quality gap with GPT-4 Turbo while running 4x faster
Being accesible is huge.

I still see post of people running ollama on H100s or whatever, and that's just because its so easy to set up.

brucethemoose2··on Phind-70B: Closing the code quality gap with GPT-4 Turbo while running 4x faster
Yeah I will have to test it out, though TBH I am more inclined to run models locally.

As I mentioned, being such an extensive continuation train can (sometimes) totally change the capabilities of a model.

brucethemoose2··on Phind-70B: Closing the code quality gap with GPT-4 Turbo while running 4x faster
Yeah, and I'd go so far as to call it a continued pretrain with that many tokens. More like a whole new model than a traditional finetune.
brucethemoose2··on Phind-70B: Closing the code quality gap with GPT-4 Turbo while running 4x faster
Your options are:

- Hybrid offloading with llama.cpp, but with slow inference.

- Squeezing it in with extreme quantization (exllamav2 ~2.6bpw, or llama.cpp IQ3XS), but reduced quality and a relatively short context.

30B-34B is more of a sweetspot for 24GB of VRAM.

If you do opt for the high quantization, make sure your laptop dGPU is totally empty, and that its completely filled by the weights. And I'd recommend doing your own code focused exl2/imatrix quantization, so it doesn't waste a megabyte of your vram.

brucethemoose2··on Phind-70B: Closing the code quality gap with GPT-4 Turbo while running 4x faster
I have not had luck with codellama 70B models for coding, nor have I had it with the mistral leak.

If I were Phind, I'd be looking at Deepseek 33B instead. While obviously dumber for anything else, it feels much better at coding. Its just begging for a continued pretrain like that, and it will be significantly faster on 80GB cards.

brucethemoose2··on Phind-70B: Closing the code quality gap with GPT-4 Turbo while running 4x faster
I agree...

Except for the leaderboard. Its all but useless, not just because of the data contamination/cheating but because the benchmarks themselves are flawed. They are full of ambiguity/errors, and they dont even use instruct formatting.

brucethemoose2··on A peek at Intel's future foundry tech
I think so.

But Intel pretty much bet everything AI on Falcon Shores (2025). If its crumby or late, I think it will be difficult for them to catch up. They'll have inference hardware in laptops, sure, but still...

brucethemoose2··on Gemma: New Open Models
That is awesome!

I hope y'all consider longer context models as well.

Also, are ya'll looking alternative architectures like Mamba? Being "first" with a large Mamba model would cement your architectural choices/framework support like llama did for Meta.

brucethemoose2··on Gemma: New Open Models
There are some exceptions, like Mistral 0.1 (which is technically 32K according to the config but practically 8K because the sliding window is awful) and InternLM (which (at least initially) used auto rope scaling to extend the context as part of the model's architecture).
brucethemoose2··on Gemma: New Open Models
You should be looking at Deepseek's coding models, and finetunes of those.

I run 33B on my desktop, and find it to be sufficient for many tasks.

brucethemoose2··on Gemma: New Open Models
> Also the inference doesn't always stop generating resulting in garbage being added to the response.

That sounds like a chat format misconfiguration.

This could partially be Google's fault, as they used yet another novel prompting format.

Also, for sane inference speed on H100s, you'll have to wait for architecture support from the optimized frameworks. Vanilla transformers is beyond awful even with FA2.

brucethemoose2··on Gemma: New Open Models
Alignment is all but a non issue with open weight base model releases, as they can be finetuned to "de align" them if prompt engineering is not enough.
brucethemoose2··on Gemma: New Open Models
This is a good point actually, and an underappreciated fact.

I think so many people (including me) effectively ignored Mistral 0.1's sliding window that few realized 0.2 instruct is native 32K.

brucethemoose2··on Gemma: New Open Models
Miqu was (allegedly) an internal continued pretrain Mistral did as a test, that was leaked as a GGUF.

Maybe its just semantics, it is technically a finetune... But to me theres a big difference between expensive "continuation training" (like Solar 10.7B or Mistral 70B) and a much less intense finetuning. The former is almost like releasing a whole new base model.

It would be awesome if Mistral did that with their data, but thats very different than releasing a Gemma Instruct finetune.

brucethemoose2··on Gemma: New Open Models
TBH the community has largely outrun Mistral's own finetuning. The 7B model in particular is such a popular target because its so practical to train.
brucethemoose2··on Gemma: New Open Models
Will there be "extended context" releases like 01.ai did for Yi?

Also, is the model GQA?

brucethemoose2··on Gemma: New Open Models
Only 8K context as well, like Mistral.

Also, as always, take these benchmarks with a huge grain of salt. Even base model releases are frequently (seemingly) contaminated these days.

brucethemoose2··on Hetzner GPU Server
Are you going to be training constantly?

TBH renting a big GPU (A100 or better) and getting the run over with quick is usually best. VRAM is everything.

brucethemoose2··on SSDs have become fast, except in the cloud
Yeah this was my impression.

I am but an end user, but I noticed that disk IO for a certain app was glacial compared to a local test deployment, and I chalked it up to networking/VM overhead

brucethemoose2··on Big media publishers are inundating the web with subpar product recommendations
But people need to use it.

Critical internet/app browsing should be taught in school, like critical reading. I feel lucky to have been a nerd in the 2000s where people picked up this skill, but honestly I have no idea how kids, older folks just getting into tech and such are acquiring sources/skills trapped inside of Discord, YouTube, Facebook or whatever.

brucethemoose2··on Ask HN: Nitter officially declared "over" today – alternatives?
And yet Discord has become a de-facto forum replacement for niche topics/communities.

...Which is awful.

As is often said here, Discord is where information goes to die.

brucethemoose2··on World model on million-length video and language with RingAttention
Its not that cost prohibitive! You can run GPT 3.5-replacing AIs (aka 34Bs) on a $1500 PC... If you can find a used 3090 now.
← PreviousPage 3 of 34Next →