455 karma · joined July 15, 2014
Personal opinion: I think this means we've probably exhausted all the low hanging fruit in LLM land. This was the last thing I was reserving judgement for. When the most hyped up big idea openai has rn is basically "we're just gonna have the model dump out a massive wall of semi-optimized chain of thought every time and not send it over the wire" we're officially out of big ideas. Like I mean it obviously works... but that's more or less what we've _been_ doing for years now! Barring a total rethinking of LLM architecture, I think all improvements going forward will be baby steps for a while, basically moving at the same pace we've been going since gpt-4 launched. I don't think this is the path to AGI in the near term, but there's still plenty of headroom for minor incremental change.
By analogy, i feel like gpt-4 was basically the same quantum leap we got with the iphone 4: all the basic functionality and peripherals were there by the time we got iphone 4 (multitasking, facetime, the app store, various sensors, etc.), and everything since then has just been minor improvements. The current iPhone 16 is obviously faster, bigger, thinner, and "better" than the 4, but for the most part it doesn't really do anything extra that the 4 wasn't already capable of at some level with the right app. Similarly, I think gpt-4 was pretty much "good enough". LLMs are about as they're gonna get for the next little while, though they might get a little cheaper, faster, and more "aligned" (however we wanna define that). They might get slightly less stupid, but i don't think they're gonna get a whole lot smarter any time soon. Whatever we see in the next few years is probably not going to be much better than using gpt-4 with the right prompt, tool use, RAG, etc. on top of it. We'll only see improvements at the margins.
Personal opinion: I think this means we've probably exhausted all the low hanging fruit in LLM land. This was the last thing I was reserving judgement for. When the most hyped up big idea openai has rn is basically "we're just gonna have the model dump out a wall of semi-optimized chain of thought every time and not send it over the wire" we're officially out of big ideas. Like I mean it obviously works... but that's more or less what we've _been_ doing for years now! Barring a total rethinking of LLM architecture, I think all improvements going forward will be baby steps for a while, basically moving at the same pace we've been going since gpt-4 launched. I don't think this is the path to AGI in the near term, but there's still plenty of headroom for minor incremental change.
By analogy, i feel like gpt-4 was basically the same quantum leap we got with the iphone 4: all the basic functionality and peripherals were there by the time we got iphone 4 (multitasking, facetime, the app store, various sensors, etc.), and everything since then has just been minor improvements. The current iPhone 16 is obviously faster, bigger, thinner, and "better" than the 4, but for the most part it doesn't really do anything extra that the 4 wasn't already capable of at some level with the right app. Similarly, I think gpt-4 was pretty much "good enough". LLMs are about as they're gonna get for the next little while, though they might get a little cheaper, faster, and more "aligned" (however we wanna define that). They might get slightly less stupid, but i don't think they're gonna get a whole lot smarter any time soon. Whatever we see in the next few years is probably not going to be much better than using gpt-4 with the right prompt, tool use, RAG, etc. on top of it. We'll only see improvements at the margins.
You can use laser diodes just like you would a regular LED in an Arduino/rPi setup. They're basically the same thing electrically. Be aware, using those standard laser diodes in public, especially for an event or show, is against the law because of the risk of blinding someone. If you want to do a laser light show or something, you need to use the sub 5 milliwatt kind they have at concerts and such, and you need a permit.
Some of it's fun, but a lot of it really is just unpleasant exercise. I hate it, but I stick to it for the positive side effects. In my mind, the only way to it is to do it.
The UI looks a little ugly, but the UX is really great! I love the fun facts while loading, and how you're able to test the app right on the landing page. The graph is cool too, and I legitimately feel motivated to show this to other people.
I feel like a first class citizen. I don't fear getting car doored. The bike lane is relatively clean, unobstructed, and unweathered by heavy loads. The contract for how cars and bikes should behave is could not be more clear. The only thing drivers don't seem to understand about it is that they are no longer prioritized. On that note, left hand turns are no longer terrifying. There is still loads of parking available on both sides of the road that doesn't require driving over my path.
My only real complaint is that I wish it were longer as the transition to regular lanes is whacky. Also the guard rails do seem to take a bit more space than they need to.
Each attention node is essentially performing a simple search on the previous text given a single word. It's kind of like Google! The actual search or "query" is the word you're looking at, the page title or "key" is the other words in the sentence, and the page/document or "value" is... also the other words in the text (each word is like its own page title). So if you imagine the whole internet is just the sentence "Hi my name is Bob", and you googled "name", you should get a results page with "Bob" at the top, followed by "my" then "is" then "Hi".
As an optimization, the transformer will actually run this "search" operation for all the words in the sentence at the same time. It then spits out a matrix saying how relevant each word is too every other word. Basically each transformer in your neural network is like a little guy who says "Hey, if you're looking at X word, here's the other words Y and Z that might be relevant in the same sentence". The rest of the NN can then use that information using more typical feed forward techniques (I'm mostly glossing over use of embeddings and other various special techniques that are used in practice here). There will be multiple transformers in a given LLM that each perform a slightly different search.
But how does a transformer know how to rank these words? In a nutshell, as we train it to predict text, signals from the error in that training prediction back propagate to the transformer node, and it gradient descends to be the best search engine it can be. Each transformer will have a slightly different random starting state and different distribution of modules further forward in the network back propagating error to it, and thus they will each converge to a slightly different "search algorithm" so to speak.
Yeah, citation needed. I see this everywhere online, often used as a rhetorical device to blame toxic dating culture on irrational female behavior vs reasonable men willing to settle. But I don't think this claim even shakes out in the surveys. I remember having this conversation with a friend, and IIRC we found an okcupid study showing that both genders are generally hypergamous. Dating app dynamics make it so men act slightly less hypergamously than women, but both men and women ultimately try to date up and shoot for the top 20% if they can. People don't want to settle. Anecdotal data point: me - as a man, I still definitely have standards.
Edit: yeah in fact okcupid found basically the opposite to be true https://archive.ph/2017.01.21-154729/https://blog.okcupid.co... "As you can see from the gray line, women rate an incredible 80% of guys as worse-looking than medium. Very harsh. On the other hand, when it comes to actual messaging, women shift their expectations only just slightly ahead of the curve, which is a healthier pattern than guys’ pursuing the all-but-unattainable."
If there are any Apple execs in this thread: please listen to your customers! We are screaming for a flagship mini!
Having the one thing to do gives you structure and a guarantee that you don't waste the day. The rest gives you novelty and a chance to actually relax if you want to.
The only real long term dangers of psychedelics would be impaired driving while under the influence (obviously) and increased likelyhood of psychosis if you were already predisposed to schizophrenia or bipolar. "Dangerous to a stable mindset" what does that even mean? Is that supposed to sound scientific?
As with most brewery tours, we got to try some of their beers afterwards. One of the ones I tasted was the best sour I've ever had in my life. I asked where I could buy it and they said they were never going to sell the stuff.
I'd argue the only two real things that have happened since COVID are that FiDi retail failed to recover and the Tenderloin expanded a bit further into lower Nob and SOMA. The rest is pretty much as I remember it in 2019.