1,697 karma · joined June 29, 2017
This will only improve when o1's context windows grow large enough to maintain all its intermediate thinking steps, we're talking orders of magnitude beyond current limits. Until then, this isn't just a UX quirk, it's a fundamental constraint on the model's ability to develop thoughts over time.
the hardware lottery, well... imo it's really about leveraging fully parallel training to learn how to use a memory. attention is quadratic but it can be computed in parallel. it's an end to end learned memory. getting that kind of pattern into RNNs won't be easy but it's going to be crucial before we boil the ocean.
Really? ;) I guess you don't believe in the universal approximation theorem?
UAT makes a strong case that by reading all of our text (aka computational traces) the models have learned a human "state transition function" that understands context and can integrate within it to guess the next token. Basically, by transfer learning from us they have learned to behave like universal reasoners.
I think the discussion here suggests it's an issue of perspective. Is socialization a fact of life, or is it something you are allowed to opt out of?
It's okay to be tired. A day that leaves you exhausted should be celebrated and followed up with rest to prepare for the next day!
Yeah, there's clearly tendencies for the fish to return to where they were born. I'm sure that's driven by all kinds of complex genetic memories and probably more importantly selective advantage due to adaptation to the specific characteristics of the given stream, but genetic memories for a specific stream seems a little bit unlikely.
Tell claude or your favorite LLM to write a full plan to implement what you need in such a way that your coworker can implement it.
Copy the result into aider, and check the results!
That's how I'm writing this message to you.
Learning to use these speech-to-text systems will be a new kind of literacy.
I think pushing the transcription through language models is a fantastic way to deal with the complexity and frankly, disorganization of directly going from speech to text.
By doing this we can all basically type at 150-200 words a minute.