typon

White woman vs. Black guy treatment. We shouldn't gloss over that.

I kind of wish Casey would stop making these videos. I'm being completely serious when I say that this kind of knowledge and mindset gives you pretty much a superpower in the industry. Everywhere I go people think I am amazing, even though I just know a few things about performance and low level programming (like, enough to be amazing for "mere mortals", but someone at RAD would snicker at my knowledge). If the industry doesn't care, it's simply a competitive advantage for you as an individual.

Sundar Pichai becomes CEO on October 24th, 2015, GOOG stock price is ~$36.87

On Apr 25, 2023 GOOG stock price is ~$103.85.

I don't understand this narrative against Pichai. This is objectively a massive success in any reasonable metric. I can think of one glaring reason why there's this crusade against him.

That's actually the hidden secret. Millions of people are using it to solve their problems quietly - something Google and other companies whose lunch they're eating should be worried about rather than being worried about grand ideas like "AI Safety". Pretty much all of my non-tech family is using it in ways that I never expected: making meal plans, writing school essays, making assignments for students, writing emails to CRA (Canadian Tax Agency), fixing broken english and translation, etc.

No, that's the wrong interpretation. The truth is that there is no such thing as "electron flow". It's not a quantum mechanical concept, just a classical approximation.

Shortening sentences using longer sentences

that goes against the narrative

So if someone sells me a good are they democratizing it to me?

There's no a priori reason to assume that models would see a step function improvement in abilities as their size increases. Perhaps the null hypothesis would be that we'd observe a linear increase in capability. Empirical evidence shows that's not the case, and that's why studying this phenomena is interesting.

Is Duolingo basically spaced repetition for language learning?

I don't understand why that's so difficult for people? Just don't connect it to the internet. It's going to magically show you ads or track you without internet.

I use it for thing's that don't really matter if they're exactly correct. For example, coming up with a travel itinerary for a country I have never visited. Rewriting a work email with better English. Summarizing a news article. There are lot of things that don't require ultimate precision. I feel like people expect these models to do something they aren't really designed for - and the mismatch in expectations causes people to be let down. They are just tools - not "mildly conscious beings" like OpenAI founders wants you to believe.

So is doing drugs. Not the free part though :p

Yes, GPT3.5 gives pretty bad results. GPT-4 gives way better results

I actually clicked through to the links GPT-4 provided (Guardian or SCMP articles)

Google search has absolutely deteriorated to the point of being useless for me. The only valuable thing left from are walled gardens like Google Maps that contain curated information. The open web, or at least Google's understanding of it, is a wasteland.

Here's a concrete example of something I experienced just yesterday. Watched the excellent film "A touch of sin" and wanted to know about the background so I went on Google and started searching. I actually gave it a good try and gave up after 10 minutes. Asked ChatGPT and it gave me the perfect answer, zero shot:

My prompt:

The Chinese movie "A touch of sin" is based on four real-life events from chinese life, all random acts of violence. Can you provide the real-life events that were the basis for the movie? Only answer if you are confident that these were indeed the original events. Make sure this answer is based in news sources

ChatGPT-4:

"A Touch of Sin," directed by Jia Zhangke, is a 2013 Chinese film inspired by four real-life events that occurred in China. While the movie creatively adapts the stories, these are the actual events that served as its basis:

    * Zhou Kehua's crime spree (2010-2012): Zhou Kehua was a notorious serial killer and robber who targeted ATM users in several Chinese cities. He murdered at least 10 people and was eventually shot dead by the police in August 2012.
     

...

I'm leaving out the others but I double checked each one and they're all correct.

Each with a different temperature too - kind of like a multi-armed bandit but exploring different parts of the LLM manifold.

Build quality, Model 3 is not "much quicker", Ioniq has a higher range, customer service, interior design, better colors

Just test drove an Ioniq 6 the other day. Insanely fast, beautiful and the software is pretty good too. Model 3 doesn't even compete.

The last point is all that matters. It will be painful and take a while for Americans to come to grips with it, or we will all die in an apocalypse when they can't accept it.

Creating something is much harder than verifying it.

A simple setup for improving coding skills is the following:

1. GPT is given a coding task to implement as a high level prompt.

2. It generates unit tests to verify that the implementation is correct.

3. It generates code to implement the algorithm.

4. It runs the generated code against the generated unit tests. If there are errors generated by the interpreter/compiler, go back to Step 3, modify the code appropriately and try again.

5. If there are no errors found, take the generated code as a positive example and update the model weights with reinforcement learning.

Very good analysis. I disagree with a fundamental point though: If you don't consider compute cost and just want the best possible AGI, then there's nothing stopping you from supercharging the mega-models with the same capabilities as the smaller models - and if the current scaling shows anything, the mega models will just become even better.

Yes - for example adding memory to the attention mechanism (similar to RETRO or Memorizing Transformers paper)

It's not true we know nothing. We know a little bit by using the two models from their API. Given the time per inference and the limit on messages per day for GPT4, I'm willing to bet it's doing around 10x more compute than GPT3.5. If that's because it has 10x more weights, I don't know. But it wouldn't be a terrible guess.

Key point is vaguely good. Scale is still important and that manifests in the difference between gpt3.5 and gpt4 based chatgpts. It's qualitatively and quantitatively so much better in pretty much every benchmark. There is no way around the bitter lesson.

That's how Alpaca is made

All within three months. My head is spinning.

I found that "continue exactly where you left off" will continue in a way that you expect it to more often.

Exact same experience I had. I would also add that the CSS styling doesn't really work. And it's frustrating because I would prefer if it didn't work all time, but it seems to only not work some of the time and I can't figure out the algorithm of when it doesn't work.

I tried to use urwid too and gave up. Python needs a library as good as ftxui for C++

Unfortunately I felt the same way. I'm probably their prototypical target audience (neovim user, I stay in the terminal 99% of the time) and I couldnt justify switching over to nushell full time. I wish they didn't change the syntax too much - there's still tons of scripts and commands that work on zsh or bash and are used in normal work flows.