Hacker News Reader

Comment by tatef | Hacker News Reader

[flagged]

Don't post generated/AI-edited comments. HN is for conversation between humans

https://news.ycombinator.com/item?id=47340079

Noted, thanks. I had LLM help positioning this message but I did the initial draft along with edits. Will keep in mind for the future.

That doesn't read like an AI-generated comment to me. He did mention he vibe-coded the project but that's not against the guidelines.

It's either written by an LLM, or written by someone who learned to write by reading LLM output

Vibe-coded project is fine.

At least prompt your LLM to dodge the obvious tells when commenting!

gptzero says 99% chance it’s AI-generated

It certainly has a lot of telltale signs

> The core insight:

That's a telltale sign of ai written text.

You need to change the title or actually include 1T parameter model content.

This is interesting work, thank you for sharing. What hardware would you buy today for experimenting? Seems like the new gen of macbook pros are pretty powerful?

Yes definitely. I use a M1 Max with 32gb of RAM daily and it's about on par from a performance standpoint with the new base M5 Pro 24gb. You can check the benchmarks in the repo if you're interested in seeing specific performance metrics, but investing in Apple hardware with as much memory as possible will generally get you furthest in this game.

Have you ever generated access frequency statistics for the experts in these models, something like a histogram?

ktransformers can do dynamic placement of experts and could presumably produce such a histogram, though currently its activation statistics are just a ".pt" file. https://github.com/kvcache-ai/ktransformers/blob/main/doc/en...

FWIW I never got it to work and did not dig into it much.

Why would llama with --mmap crash?

This doesn't surprise me all that much, mmap support gets little attention in general and interacts poorly with GPU-side inference. (And that's with it being default, you don't even really need to specify it as a CLI option.) OP has raised a discussion with the llama.cpp folks https://github.com/ggml-org/llama.cpp/discussions/20852 but little interest so far

But if mmap already works why would there be any interest?

Besides, discussions are for users. He didn't open PRs or issues.

Reply on news.ycombinator.com