HNHacker News
TopNewBestAskShowJobs

andreyk

5,023 karma · joined December 28, 2015

www.andreykurenkov.com
submissionscomments
andreyk··on Why the original transformer figure is wrong, and some other tidbits about LLMs
The actual title "Why the Original Transformer Figure Is Wrong, and Some Other Interesting Historical Tidbits About LLMs" is way more representative of what this post is about...

As to the figure being wrong, it's kind of a nit-pick: "While the original transformer figure above (from Attention Is All Your Need, https://arxiv.org/abs/1706.03762) is a helpful summary of the original encoder-decoder architecture, there is a slight discrepancy in this figure.

For instance, it places the layer normalization between the residual blocks, which doesn't match the official (updated) code implementation accompanying the original transformer paper. The variant shown in the Attention Is All Your Need figure is known as Post-LN Transformer."

andreyk··on Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
Just a quick clarification, attention of the same sort transformers used was already being employed in RNNs for a while. Thus the name "attention is all you need", it turned out you can just remove the recurrent part which makes it hard to train the NN.
andreyk··on How we hack Hacker News and consistently hit the front page
This article is great insofar as it describes how to write an article that appeals to HN. But having been writing and submitting to HN for like 8 years with maybe a dozen front page entries, I think the hack is simpler: pick a topic people want to discuss/find interesting, name the article well (not clickbaity but not boring), and submit earlyish in the day PST. The content of the article is not super important for hitting the front page (as evidenced by a lot of the stuff that does hit front page). BUT if you want to stay on the front page/rank higher, I suspect article quality does matter a lot.
andreyk··on Turn ideas into music with MusicLM
FYI, this is MusicLM, which was just recently published and appears to be way ahead of anything out there so far. You can check out examples here https://google-research.github.io/seanet/musiclm/examples/

So this is not just empty PR by Google.

andreyk··on 100K Context Windows
Curious why you think this? PaLM2 looks great, and Google has been productizing cutting edge AI pretty fast for years.
andreyk··on Sorting waste and recyclables with a fleet of robots
Most if not all the research papers from EDR was co-authored with people from Google robotics, so it's not all that weird. I think at least some of the people from EDR wound up at Google robotics anyway.
andreyk··on Sorting waste and recyclables with a fleet of robots
From what I hear Levine is a beast, super hard worker with incredible memory and thinking speed. So I would say he most likely is aware of all the stuff his name is on, at least at a high level. Especially this sort of big project.
andreyk··on AI-enhanced development makes me more ambitious with my projects
I've definitely found this to be the case with writing. ChatGPT gets you a "valid" output, but usually a bland and lame one. I think it's better for editing and brainstorming, or for simple tweaks. Specialized tools like SudoWrite are actually more useful as part of a writing work flow.

Pardon the plug, but I co-wrote a little blog post about this with concrete examples, if it is of interest: https://open.substack.com/pub/storiesbyai/p/chatgpt-for-crea...

andreyk··on What we still don’t know about how A.I. is trained
Almost nothing is quite an exaggeration - we know a whole lot about GPT3 (their paper was quite detailed), and even if OpenAI made some tweaks beyond RLHF the underlying model and training objective are most likely the same.
andreyk··on What we still don’t know about how A.I. is trained
TLDR: we don't know the data some of these models (particularly GPT4) is trained in, and labor practices related to their moderation seem shady.

Pretty misleading title, this is more of an overview of bad things to be aware of wrt large commercial models like GPT4. It is mostly reasonable, though complaining about carbon emissions always seems overblown to me.

It is also not super accurate in describing what GPT4 is. Quote:

"The key word in that phrase is “pre-trained.” Using all kinds of digitized content scraped from the Internet, GPT employs deep-learning techniques to find patterns, including words that are likely to appear together, while also acquiring facts, absorbing grammar, and learning rudimentary logic. "

Pre-trained is only important to note for downstream fine tuning / transfer learning (what GPT and similar things were used for back in 2018/2019), but is largely irrelevant now - for most users it is just "trained".

And "deep learning techniques..." makes it sound way fancier than it is - GPT4 is still (presumably) a language model that is fundamentally trained to do autocomplete (followed by some training on human chat data).

It may be nit-picky to criticize these technical details, but given how massive the impact of ChatGPT and GPT4 has been (if only in terms of making people aware of this sort of tech), it sure would be nice if coverage was more technically informed by this point.

andreyk··on The teen mental illness epidemic is international – Part 1: The Anglosphere
World events and economy aside, there are also clear trends with teenagers (and people as a whole) having fewer close friends than ever - loneliness is more common than ever. Our culture and society as a whole is declining in its ability to foster community and friendship, which really is deadly in its own way.
andreyk··on If you're so good, why aren't you making 600k at BigTech?
FYI, the "you" here refers to the author, not the reader. Here's the gist:

"Why aren't I working in BigTech? Why isn't everyone, if the money's so good and the coasting so sweet? Rest and vest is a real thing that happens.

The short answer – I don't think I'd like it."

andreyk··on Palm-E: An Embodied Multimodal Language Model
FYI, translation to code was actually already done last year by Google (Code as Policies - https://lastweekin.ai/p/code-as-policies) - but this still assumes a set of low level primitives, not tight visuomotor control. You really need reactive policies with tight feedback loops in many robotics domains, and that's kind of hard with these super gigantic models.
andreyk··on Palm-E: An Embodied Multimodal Language Model
Like jerpint said, there's lots of limitations here still since this only addresses the high level decision making part, not the low-level control (sending signals to motors). Not tackling low-level control means the robot is reliant on human-engineered primitives such as 'place' or 'push', which can do a lot but is not fully generic (there are tasks that require dexterous control, tight visuomotor control loops, etc.). In other words, this kind of system is not even close to being general enough to drive a car.

Also, this has the typical limitations of LLMs - no (long term) memory, inherently limited context window size, no online learning via eg trial and error, no 'agency' in general (it can only do things when it is told to, there is no sense of exploring the world and learning about it). To really become AGI these sorts of systems would really need to be able to adapt to lots of niche tasks that they don't apriori know how to do, and there's not a great solution for that right now (prompting can sort of work, but again memory/context window size is still an issue).

andreyk··on Twitter was down
https://downdetector.com/status/twitter/

Looks like it's already being/has been fixed. Still, pretty impressive level of brokenness for a short time.

andreyk··on Show HN: Graph-based AI for longform writing
It looks neat - do you as the author need to manage the graph yourself though? If i'm writing something long form, i'd rather just write the text with AI generating the graph automatically...
andreyk··on ChatGPT for Robotics
The context is pretty different - Duplex was a public demo, this is a research paper with a whole paper behind it. There's no reason to script something if they have a working algorithm...
andreyk··on ChatGPT for Robotics
Looks pretty similar at a high level to Google's Code as Policies paper (translate instructions to API calls with a language model). Exciting to see this sort of thing happening in robotics!
andreyk··on The paper that made ChatGPT possible
I mean... Plenty of papers did? Might as well just cite the 1986 Hinton backpropagation paper or the 2012 ImageNet paper.
andreyk··on Prime Voice AI: AI speech software
Related - I've found BeyondWords to be really nice. Its generated speech is not quite this good, but it's close, and it has a library of fairly different voices. Plus, it's UI allows you to create audio with a mix of voices, which is not offered by most other such services.

Plug warning - I've been using it to create narration for short stories with it for a while, and the output is better than I would have expected. Here's a recent example involving two characters talking - https://storiesby.ai/p/melancholy-musings-over-drinks

andreyk··on Write simply (2021)
Write in a way that accomplishes the goal you have in mind
andreyk··on MusicLM: Generating music from text
FYI, they don't actually train end-to-end, they have three separate models that are each trained via self-supervised learning independently (I think, I haven't read the paper very carefully).
andreyk··on MusicLM: Generating Music from Text
Link to paper: https://arxiv.org/abs/2301.11325 (the one on the site seems broken)

Even with the knowledge that something like this is inevitabely coming, it's pretty crazy to see the examples. The fact it can create fitting music from painting descriptions (including ones such as or Guernica) is unexpected. And it seems to generate almost naturalistic vocals with reasonable-sounding lyrics, with no embedded structure for lyrics or anything.

The paper itself is pretty cool. While GPT-3 was just "GPT-2, but bigger", this seems to have required some clever combination of models. Wonder if similar ideas will be needed for longer-form text or video generation.

andreyk··on Work life balance is impossible
Here is the key bit:

"When you hear “balance,” you immediately think of a dichotomy. For two things to be balanceable, they must be at odds with each other. Lowering one side of the scale must raise the other side.

When we describe work and life as things to be balanced, we are suggesting that work and life are at odds with each other. More time or energy allocated to work means less time and energy allocated to life.

This is obviously absurd, though. Work is just another part of life like family, community, food, fitness, creativity, travel, fun, spirituality, etc.

The question isn’t how do you balance work and life, but how do you create a healthy relationship among work and the various other important areas of life? "

This is completely pedantic - obviously work is literally a part of life, so the phrase means exactly what the author states in the last paragraph.

Not worth reading IMO.

andreyk··on The 5% Rule
"between the years of 2009 and 2013, I talked to something like 13,000 to 15,000 people while I was doing retail and SMB sales for Verizon. I learned a lot from that period, but one of the things I learned that I did not expect was what I now call my "5% rule."

No matter how kind, warm, thoughtful, amazing, cheerful, consistent, and perfect I treated people roughly 5% of them would just be terrible (yes, I have the numbers to back this) ... this number has held up in other parts of my life too (dating, restaurant customers, etc).

...

the take here - 3% to 5% of people for whatever reason just plain suck."

Sounds easy enough to believe. Of course, social conventions and so on influence such things, these people are not always terrible - I am not sure I buy the latter part of the thesis. I certainly haven't had this experience dating or in grad school. But, there is a reason "karen" is now a commonly used noun.

andreyk··on I Will Write More
I can relate to this post, but it does seem to be a little idealistic. The plan is: consistently set aside time, write any ideas that seem fun, then see if it's valuable and does not exist (and don't release it otherwise). So, basically it's "write for the sake of writing." As someone who has written many long high effort blog posts (1000s of words long), the amount of work a solid post usually takes for me is too great for this write-and-see-what-happens plan; I simply would not want to put things out there that are redundant or uninteresting. But that's just me - maybe this'll work for the author of this post.
andreyk··on I have reached Vim nirvana
Personally I like Vim because I do a lot of coding while in an SSH session and it's easy to port my setup very quickly. My work involves using a workstation with a GPU so I am not able to do that at home, and debugging requires me to run things from command line a lot to know what to change in the code. No doubt I could create a set up with a fancier IDE to do that, but at the scale I'm working in (academic code with just 1000s of lines of code) there is really no need.
andreyk··on How Twitter moderated the Covid debate
Definitely agree wrt the last point. It's bizarre that one of the one hand as far as I know Twitter is still committed to moderating a wide range of content (hate speech, violent media, etc.) but on the other hand Musk has doubled down on bots and calling the shots in a top down manner (exactly what this article called out).
andreyk··on How Twitter moderated the Covid debate
Nowhere in this piece is the FBI mentioned...

And it's fine to make an argument against moderation in general. But that's not what this article is doing.

andreyk··on How Twitter moderated the Covid debate
"Why does a government has to "lean" on a private media company in a "democrat" and "liberal" democracy?"

There are people within the government whose job it was to respond to the covid pandemic. Part of that response involved educating the public about the disease and best practices related to it, which therefore included fighting misinformation. Misinformation is spread on platforms such as Twitter, so it made sense for the government to communicate with Twitter and let them know what it considered to be a good course of action. Do you disagree with this explanation?

← PreviousPage 2 of 21Next →