HNHacker News
TopNewBestAskShowJobs

andy_xor_andrew

2,096 karma · joined August 11, 2022

submissionscomments
andy_xor_andrew··on Groq surpasses 1,200 tokens/sec with Llama 3 8B
When reading Hacker News you develop a signal/noise filter, where lots of headlines make bold claims but you filter them out as embellishment or exaggeration.

My bullshit detector went off when I first saw Groq posted on HN - a startup is making their own chips (doubt) that performs faster than anything Nvidia has for inference (doubt) and accelerates LLMs to hundreds/thousands of tokens per second?? Mega doubt.

But... then I tried their demo, and... yeah, it's that good. Such an amazing company of talented individuals.

andy_xor_andrew··on Microsoft Phi-3 Cookbook
in the screenshot you shared in the twitter link [2], the model appears to do everything correctly - it terminated its message with <|end|> which is correct according to the Phi-3 prompt format. It seems whatever environment you are hosting it in does not understand that <|end|> should be considered the EOS string, and not <|endoftext|> ??
andy_xor_andrew··on Dear Tim Cook: Be a Decent Human Being and Delete This Revolting Apple Ad
am i literally shaking with anger, pacing around my room, crying, cuddling my emotional support dog, because i'm so upset at this advertisement?

no.

is it a terrible, possibly even insulting ad? yeah, I think so.

pretty clearly the original concept was "the new iPad is so thin, let's make an ad that shows all types of art compressed into a small size". And somehow that got lost in translation during production to "let's destroy objects of artistic expression, and show them all obliterated, only to be replaced by our mass-market product".

if the goal was to generate controversy, it worked. but if they actually thought this ad would go over well, they're insane, and surely there is someone in the ad meeting who spoke up and said it was terrible, and they got shut down, and right now they must feel so vindicated.

andy_xor_andrew··on Linux When?
> On macOS, for example, Zed makes direct use of Metal. We have our own shaders, our own renderer, and we put a lot of effort into understanding macOS APIs to get to 120FPS.

So they are taking the exact opposite approach of Electron (VS Code).

In my mind, if you're someone who types in HN comments in a rage because your text editor (VS Code) eats up 200+ MB ram, you don't get to cry about Zed not being supported on Linux from day one, because you can't have your cake and eat it too - if you want it on your platform, you gotta wait for the shaders to be written.

andy_xor_andrew··on Apple introduces M4 chip
The iPad Pros, though?

I'm very curious how much iPad Pros sell. Out of all the products in Apple's lineup, the iPad Pro confuses me the most. You can tell what a PM inside Apple thinks the iPad Pro is for, based on the presentation: super powerful M4 chip! Use Final Cut Pro, or Garageband, or other desktop apps on the go! Etc etc.

But in reality, who actually buys them, instead of an iPad Air? Maybe some people with too much money who want the latest gadgets? Ever since they debuted, the general consensus from tech reviewers on the iPad Pro has been "It's an amazing device, but no reason to buy it if you can buy a MacBook or an iPad Air"

Apple really wants this "Pro" concept to exist for iPad Pro, like someone who uses it as their daily work surface. And maybe some people exist like that (artists? architects?) but most of the time when I see an iPad in a "pro" environment (like a pilot using it for nav, or a nurse using it for notes) they're using an old 2018 "regular" iPad.

andy_xor_andrew··on Rust: Box Is a Unique Type
One of the biggest struggles I have (and others have, judging from Stack Overflow) is how to generically handle accepting types of Box<T>, Rc<T>, T, Pin<T>, &T, &mut T, etc etc.

Of course you can write a function that is generic on <I, T: AsRef<I>> or something, but the moment you introduce function-coloring stuff like async, object safety, etc, things explode.

andy_xor_andrew··on FCC votes to restore net neutrality rules
great, what are the odds they reverse the reversal next year in a hypothetical new administration?
andy_xor_andrew··on The V8 Sandbox
I'm confused about the fizzbuzz example they provide.

```js

let array = new Array(100);

let evil = { [Symbol.toPrimitive]() { array.length = 1; return 15; } };

array.push(evil);

// At index 100, the @@toPrimitive callback of |evil| is invoked in

// line 3 above, shrinking the array to length 1 and reallocating its

// backing buffer. The subsequent write (line 5) goes out-of-bounds.

array.fizzbuzz();

```

I'm probably missing the point, but I thought indexing into an array in Javascript outside its bounds would result in an exception or an error or something?

andy_xor_andrew··on Mamba Explained
> But Transformers have one core problem. In a transformer, every token can look back at every previous token when making predictions.

Lately I've been wondering... is this a problem, or a strength?

It might be a fallacy to compare how LLMs "think" with how humans think. But humor me for a second. When you are speaking, each time you emit a word, you are not attending to every previous word in your sentence (like transformers), rather you have a state in your mind that represents the grammar and concepts, which is continuously updated as you speak (more similar to SSMs).

Similarly, when you read a book, every time you read a word, you are not attending to every previous word in the book. Your model of "the book" is rather a fuzzy/approximate state that is updated with new information every time a new word appears. Right? (I'm sorry I know this is very handwavy and psuedoscientific but bear with me).

Ok, so if (big if) you feel like the above is true, then to match human-type language modelling, SSMs seem more human-like than transformers.

BUT... then aren't transformers strictly better in terms of accuracy? Because a transformer never "forgets" information, as long as it is within the context window, because it revisits that information every time it emits a new token.

So let's say we can remove the "quadratic attention" problem of transformers with SSMs. That's a nice training/inference performance boost. But... look at where we got with "naive" attention. GPT 4, Claude 3. It's not like we're hitting a wall with quadratic attention. It's absurdly more expensive than SSMs, but GPUs certainly aren't getting slower. If all AI work stops now, and only hardware improves, it wouldn't be long until GPT4 could run on local hardware, right, provided Moore's law?

/end rant, not really sure what my point was, I'm not against SSMs (they're cool) but rather I'm wondering if the SOTA will ever be SSM when attention is so damn good

andy_xor_andrew··on Towards 1-bit Machine Learning Models
Interesting, and it kinda makes sense. You quantize, which invariably means you lose some precision, but then you can finetune post-quantization to recover at least some of it. Neat idea.
andy_xor_andrew··on Can Demis Hassabis save Google?
Given his experience with AlphaGo/Star/Zero/etc, I'm sure if there's any way to apply tree search, or policy networks, to LLMs, Demis will find it.

I've always found it strange that a LLM is basically an action/value model that scores all next possible actions (tokens), and yet we DON'T traverse it like a tree. Well, "beam search" exists, but that's kind of the most naive approach.

Then again, I'm not smart, and lots of very smart people are thinking nonstop about LLMs, and no type of tree search has become widely used, so maybe there's really nothing there.

andy_xor_andrew··on Flipping Pages: New Linux vulnerability in nf_tables and exploitation techniques
I have some questions for you wizard shellcode-flinging hackers out there.

with modern mitigations like ASLR and beyond, how are exploits like this even possible?

I took a course in college which provided us with a dozen binaries, to be run on a specific Ubuntu version. Each binary had a different bug, use-after-free, buffer overflow, both, or more, and our task was to exploit it.[0]

It was HARD. Finding the flaw was hard. Then, writing the exact correct shellcode to do something useful with the flaw was even harder. Especially when you only have the compiled binary and GDB to work with.

In harder levels, we had to do this with mitigations enabled like ASLR, stack canaries, etc.

And my takeaway was, this is nearly impossible, because you need to 1. discover an exploitable flaw (or multiple, and chain them together!!) 2. find the exact binary shellcode payload that will do something useful without simply crashing the executable. And this was in a controlled environment as students, not the real world which is even more difficult.

So my question is, with all these mitigations, how is it possible?!

[0] I googled for the exact project but could not find it, though it was very very similar to this: https://web.stanford.edu/class/archive/cs/cs107/cs107.1194/a...

andy_xor_andrew··on Apple Turned on 'Discoverable by Others'. You Might Want It Off
> If you don’t want your device to be included in that tally of nearby contacts for others, you can toggle it off. Just remember that leaving it on won’t tell your friends—or Apple—anything about you personally.

That sentence seems like a self-contradiction?

If it didn't tell my friends--or Apple--anything about me, how can it result in my friends getting journaling suggestions that are different based on whether the feature is enabled or not?

Unless the author thinks "knowing who you were with and when" does not count as "knowing anything about you," which I would strongly disagree with, and I'm not even generally a paranoid person like some on HN are (no offense to them, glad they're paying attention)

andy_xor_andrew··on Fast Virtual Functions: Hacking the VTable for Fun and Profit
on this topic, it's interesting how my mindset changes when I'm writing Rust vs Python, with regards to vtables, dispatching, and allocation.

Writing Rust for a toy project: "I MUST avoid allocation, and the dyn keyword, at all cost!!"

Writing the same toy project in Python: "lol just copy the string who cares"

andy_xor_andrew··on Fixing Gemma Bugs
This gave me lots of confidence in Unsloth when I first read it.

I'll admit I was a little skeptical of Unsloth, since anything that boasts free perf improvement, just by dropping in some middleware, makes me suspicious. Especially from such a small team.

I assumed it was just introducing some hacks that create an inexact implementation of attention or some faster-but-inaccurate cuda kernels or something.

But now I believe this small team really knows their stuff :)

andy_xor_andrew··on Google to pause Gemini image generation of people after issues
> Quite the cultural flame war in this thread.

It's kind of the perfect storm, because both sides of the argument include a mixture of reasonable well-intentioned people, and crazy extremists. However you choose to be upset about this, you always have someone crazy to point at.

andy_xor_andrew··on The killer app of Gemini Pro 1.5 is using video as an input
> That 7 second video consumed just 1,841 tokens out of my 1,048,576 token limit.

is this simply an approximation done by Gemini in order to add some artificial limit on the amount of video?

Or do video frames actually equate directly to tokens somehow?

I guess my question is, is there a real relationship between videos and tokens as we understand them (i.e. "hello" is a token) or are they just using the term "tokens" because it's easy for a user to understand, and an image is not literally handled the same way a token is?

andy_xor_andrew··on Gemma: New Open Models
> The training token count is tripled (6T vs. Llama2's 2T)

Damn, 6T? That's a lot!

Given that this model seems to roughly match Mistral (according to the numbers from Google), this makes me think we have saturated the 7B parameter space, and couldn't possibly make it much better unless new techniques are discovered.

andy_xor_andrew··on iMessage with PQ3 Cryptographic Protocol
I'm way out of my depth in terms of the math here.

But my 'software engineer brain' likes the ideal of using the prime factoring problem, because it's so simple to understand, and feels like some kind of universal primitive. "It's easy to multiply but hard to factor." It just seems so intuitive.

But I'm reading the 'learning with errors' wiki page and it's beyond my comprehension.

There's a weird fear in my mind that all these "post quantum algorithms" are so complicated, with such a large surface area, that they may hide flaws. While prime factoring, or even the elliptic key stuff, is so simple to comprehend.

that said, obviously the experts know what they're doing, and I'll use what they suggest. just saying that this thought has crossed my mind.

andy_xor_andrew··on Show HN: NotesOllama – I added local LLM support to Apple Notes (through Ollama)
On this topic (using local LLM for analyzing local text on an iDevice) -

I highly suspect that the recent Journal app from Apple, which auto-installed via an iOS update, is intended to incentivize users to journal and write about their daily lives, so that when Apple inevitably ships a local LLM on iDevices, there is already a corpus of data for the model to RAG over and use to "understand" the user.

andy_xor_andrew··on Dead Internet Theory
I found out recently there's a place on the web, right now, that has all the charm and personality of the "homepages" of the 90s and early 00s.

A place where you can make your own little corner of the web, share it with friends, find others with overlapping interests, etc. It can be uniquely yours, and even if there are hundreds of others, yours is unique because it's yours.

And believe it or not, it's... Twitch. Seriously- small, personal Twitch streams, usually with fewer than ~50 views.

And I think this emergence is quite poetic, because the exact same thing that killed the "old" web - that is, nearly unlimited bandwidth for streaming rich media - is the thing that makes Twitch possible.

Just open Twitch, click around, search for something that interests you, and you'll find someone streaming in their cozy room, possibly on camera or possibly with a cute PNG or 3D avatar, talking about that topic, interacting with chat, etc. It's a really nice world there, if you know where to look and what to filter out.

andy_xor_andrew··on Groq runs Mixtral 8x7B-32k with 500 T/s
are your accelerator chips designed in-house? or they're some specialized silicon or FPGPU or something that you wrote very optimized code for inference?

it's really amazing! the first time I tried the demo, I had to try a few prompts to believe it wasn't just an animation :)

andy_xor_andrew··on There's a Reason McDonald's Coke Tastes Better Than All Others
Every day - every single day - my mom gets in her car, and drives 2.5 miles (each way) to the nearest McDonald's, to buy a diet coke. That's the only item she gets.

She has done this for over a decade.

This used to drive me insane. But somewhere around the 8th year of this, maybe because I matured a bit, I started to finding this extremely endearing.

andy_xor_andrew··on Inside the proton, the ‘most complicated thing you could possibly imagine’
I just had a really stupid thought, after finishing reading the article.

So, the electron is an elementary particle, right? Compared to the proton, the electron is "simple", yes?

Despite this difference in complexity, an electron has a charge of -e and a proton has a charge of +e. They are exactly complementary regarding charge (if I am understanding right, I am not a smart person).

my question is... why? why must protons and electrons be perfectly complementary regarding charge? if the proton is this insanely complex thing, by what rule does it end up equaling exactly the opposite charge of an electron? why not a charge of +1.8e, or +3e, or 0.1666e, etc? Certainly it is convenient that a proton and electron complement each other, but what makes that the case? Does this question even make sense?

so, there's a concept of a "positron", which I can understand - of course it has charge +e, it is the "opposite" of an electron. it is an anti-electron. at least that makes some kind of sense. but a proton is made up of this complex soup of other elementary particles following all these crazy rules, and yet it also ends up being exactly +e.

andy_xor_andrew··on World model on million-length video and language with RingAttention
> but there are some things one can do with a mega context that you just can't get with RAG.

Can you elaborate? In my mind, RAG and "mega context" are orthogonal - RAG is something done by adding documents to the context for the LLM to reference, and "mega context" is just having a big context. No?

andy_xor_andrew··on Handwriting but not typewriting leads to widespread brain connectivity
This smells true to me, though I couldn't exactly explain why.

(baseless speculation below)

One thing I've noticed is how drastically my handwriting (and maybe hand dexterity in general) varies wildly from hour to hour, mostly depending on how tired I am or how much caffeine I've had.

If I'm feeling tired and sluggish, my handwriting is a disaster. It doesn't look good. It doesn't feel good. But after some caffeine, or otherwise becoming more alert, suddenly it looks better, and it feels much better as I do it.

Is this just me? I guess it's not surprising that caffeine would have an impact like this. But the part that's surprising to me is how pronounced and reliable it is.

andy_xor_andrew··on Nvidia's Chat with RTX is an AI chatbot that runs locally on your PC
so they branded this "Chat with RTX", using the RTX branding. Which, originally, meant "ray tracing". And the full title of your 2080 Ti is the "RTX 2080 Ti".

So, reviewing this...

- they are associating AI with RTX (ray tracing) now (??)

- your RTX card cannot chat with RTX (???)

wat

andy_xor_andrew··on Seattle ordinance intended to help app delivery workers is 'hurting' them
Living in Seattle, every election the ballet was filled with items like these, which sound great and have good-faith intentions that are totally in line with my own personal convictions.

But I guess policy needs to be studied--really studied--examined from all fronts, tested with the full weight of reality.

If you ban X, what replaces it? Y? Is Y really better? If you add a charge of Z to support certain industries, how does that shift the economics? You can't just assume that a $5 charge means $5 more in the pocket of everyone in that industry.

andy_xor_andrew··on New cancer drug kinder than chemotherapy
dumb question... is it purely the demand that makes it this expensive? The "you need this or you die" aspect? Or is the cost of research and manufacturing for this stuff so astronomical that it warrants such a high price?

I almost don't want to even know... if I find out it costs only ~$5 to develop a dose, and they're charging $200k to dying people... ugh

andy_xor_andrew··on Rust to stabilize `async fn` and return-position `impl Trait` in traits
ah, ok. so this is for scenarios where you, the library author, define the trait, and users will implement it. But since different users have different requirements, you provide two+ trait definitions, so they can pick if they need Send or not.
← PreviousPage 3 of 5Next →