I agree that the moats are weak for applications, but I think there are possible strategies to capture users. One way is to make it difficult to switch, similar to Apple Music vs Spotify or iPhone vs Android. Although these platforms offer very similar features, switching has high friction. As an example, I can imagine an AI app that has been adapted to your use cases, maybe it has a lot of useful information stored about you and your projects which makes it much more effective, and switching to some other app would mean you have to move all this data or possibly start over from zero.
DeepSeek has demonstrated that there is no technical moat. Model training costs are plummeting, and the margins for APIs will just get slimmer. Plus model capabilities are plateauing. Once model improvement slows down enough, seems to me like the battle is to be fought in the application layer. Whoever can make the killer app will capture the market.
> The vast majority of our work is already automated to the point where most non-manual workers are paid for the formulation of problems (with people), social alignment in their solutions, ownership of decision-making / risk, action under risk, and so on.
I agree. That's why I think the next step is automating trivial physical tasks, i.e. robotics, not automating nontrivial knowledge tasks.
This was my first thought too. AFAIK each layer encodes different information, and it's not clear that the last layer would be able to communicate well with the first layer without substantial retraining.
Like in a CNN for instance, if you fed later representations back in to the first kernels they wouldn't be able to find anything meaningful because it's not the image anymore, it's some latent representation of the image that the early kernels aren't trained on.
I understand the complaints, but I don't think the Freakonomics podcast should necessarily be expected to meet the high standard of peer review. Is it really that bad for Levitt to take published research at face value, trusting that gatekeepers upriver have done their due diligence?
The biggest difference is when you feed a sequence into a decoder only model, it will only attend to previous tokens when computing hidden states for the current token. So the hidden states for the nth token is only based on tokens <n. This is where you hear the talk about "causal masking", as the attention matrix is masked to achieve this restriction. Encoder architectures on the other hand allow for each position in the sequence to attend to every other position in the sequence.
Encoder architectures have been used for semantic analysis, and feature extraction of sequences, and encoder only for generation (i.e. next token prediction).
Touché. I put the first paragraph in GPTZero, and it says 100% AI generated. Unfortunately, AI detectors are notoriously bad, giving all kinds of false positives, which goes to my initial point that it is largely a gut feeling. But if you look at other articles posted on the site by the same author they also come up as 100% AI generated on GPTZero, and they also fail the gut test.
I’m similar. Never been very over weight but I’ve put on 20ish lbs in the last few years. My wife and I started tracking calories and exercising a month ago and it has been so beneficial. I lost about ten lbs, but even better than losing the weight has been building a better intuition of what foods are calorie rich, higher in protein, etc.
This is fun. It would be interesting to build a single graph of concepts that all users contribute to. Then you wouldn't have to run LLM inference on every request, just the novel ones, plus you could publish the complete graph which would be something like an embedding space.
I've been long time listener of his show, so I don't necessarily disagree with your points. It just seems like he punches above his weight when it comes to guests.
This is perfect. We need more hobbyist apps. Minimalist design, simple and solid functionality, and no ads in sight. Shake off the shackles of the advertising dystopia!
For what it’s worth, my undergraduate was in Economics with an emphasis in econometrics and this article touched on probably 80% of the curriculum.
The only problem is by the time I graduated I was somewhat disillusioned with most causal inference methods. It takes a perfect storm natural experiment to get any good results. Plus every 5 years a paper comes out that refutes all previous papers that use whatever method was in vogue at the time.
This article makes me want to get back into this type of thinking though. It’s refreshing after years of reading hand-wavy deep learning papers where SOTA is king and most theoretical thinking seems to occur post hoc, the day of the submission deadline.
This is the move right here. And it doesn't have to be a GPT-4 I-know-absolutely-everything model. I think they could train a <1B param model that is capable of being cute and interactive.
So, the argument is: "I worked on old tech that didn't work without human's in the loop therefore no technology will ever work without human's in the loop"? I don't follow.