202 karma · joined May 25, 2022
More details on the link here: https://www.nasa.gov/feature/gemini-xii-crew-masters-the-cha...
A fresher take on scientific computing like Julia, if it is mainstream, might enable more contributions and in general understandability of the black box algorithms.
But this introspection gave me a reality check. In my own work, as much as I love Julia, I fallback to quickly using Python libs. In industry, most of the people's focus is on using someone's already written code. The byproduct is inheriting Python and C/C++ interface mess around dependency management. People needing to write their own algorithms are majority academics and researchers where Julia has flourished.
The incentives of corporate work are setup to build on top of Python's ecosystem which means (sadly) Julia will stay in its niche.
- What would you say is your biggest learning from this experiment? - What were the Aha! moments you experienced while learning like this? For example, something you vaguely understood before but is now super clear to you.
It takes guts to put out such bold bets in writing. We've seen many (senior!) tech people sneer at George's at times naive optimism. I actually find the "how hard could be" attitude refreshing against "no no it is complicated you can't do that" gatekeeping. Because otherwise we will end up using big-tech for lack of alternative. It is not the critic who counts and all that..
I often have arguments about utility of Python with my more technically solid engineer friends who keep telling me "Python doesn't scale" etc. And I often come back to the same point you highlighted: is the particular hell worth the result? For 85-90% of the cases the answer is resounding yes.
For nimble teams, startup projects, internal tools : most of the code is often thrown away eventually. Python's beauty is that the language quickly gets out of the way. Everyone then is forced to focus on complexity of the business domain, "does the code solve the business problem". The architecture/scale/speed problems will eventually arrive with Python. But most purist engineers overestimate how soon it will come. At most of the failed startup attempts I was part of, the PMF problem was more pressing to solve than software constraints.
Early days of YouTube, Dropbox, Instagram (and maybe OpenAI too) are big testament to this. I've made my peace these days by not fighting to prove Python is the best language etc. If someone tells me "Rust beats Python" kind of argument: I say I agree, wish them luck and focus on shipping things.
tl:dr; Python is still the best choice to "quickly deliver value and test your business hypothesis".
Not to belittle the library, but most of it is a very thin wrapper classes that reek of premature abstraction, couple with hit-n-miss docs. At this point, given the hype, it is primarily optimized for cooking up demoes quickly. But not sure if the valuations or production use is justified.
I taught my 4 year old to read English through the book. She's far ahead in reading amongst her peer group (English is not our native language). I know sounds cheesy, but please count me in for exchanging ideas or fixing education using his methods.
The catch is trying to keep up with Javascript complexity. Javascript/react world is entirely different beast. Spent some time understanding class based React. But then things moved to hooks. I sort of started grokking hooks. But now there is new clamour (not yet mainstream) about something new like server side components or signals or hydration or what not. Same is the case with Svelte. I tried the basic tutorials, understood it. But Svelte kit started talking about 100 different ideas about server pre-rendering components etc. I thought why bother? Jinja2 templating and bit of inline JS looked a better option.
Python devs want to stick to Python because it is still second best language to get something out there, quickly. Except for UI. Currently I'm looking into using HTMX as much as possible. If this thing solves for UI side of it then I'll definitely give it a try.
Also word of advise from my experience (I'm not an expert in DL either): Think of DL field as a game of lego blocks. The ideas in this book / Karpathy's videos are the basic lego blocks: parameterised linear functions, non-linearities, auto-grad, cross-entropy / KL divergence loss and gradient descent. Then there is entire body of more complex legos discovered simply by practice (alchemy!): transformer blocks, layer norm, max-pooling etc. It is impossible to understand how the second kind were obtained from first principles. The trick is to not beat yourself up about the advanced blocks too much but just play around, read up things in papers. Just focus on fundamental blocks.
My criticism with fast.ai, (I am part time educator), is that this approach is an information overload and poor sequencing. Their comparison with Teach whole game approach is flawed because a game of, say Football, is essentially simple. So you can say just start kicking around. But we don't teach chess this way. It is accepted that you have to spend some time upfront to learn the rules before you can play even simple game. Sure one need not learn castling or en-passant upfront. But you get the drift.
This book (looking at the preview chapters) is going to follow the lego blocks approach or bottoms up approach to build it. For me, this is correct way to teach supervised ML focussed on neural networks and deep learning. We have a problem of too many library plumbers in the ML field currently. People who can piece together library function calls without knowing why it is working. But this house of cards is not sustainable strategy to build AI based application over long term.
Long story short, the book will need patience but that patience will be worth it!
Read the preface. A very high praise coming from Guy Steele Jr and Peter Norvig.
At the same time, a warning! Scheme based introductions don't appeal to everyone. Some people feel way out of comfort zone in coding with it (which is sad because it is much simpler). Also, the utilitarian appeal is low: it won't right away see a step change in your Pytorch knowledge or whatever. The appeal of these books is to think deeply about fundamental ideas by implementing them in simplest language without too much help.
In short, YMMV. But if you have a long term view it might help you a lot than sort of currently fashionable trends. (Though I must admit that fast.ai is not just a flavor of the season resources but much better!)
I wonder if this will lead to a stratification of work in the society: lot of jobs can operate on the blur. "Just give me enough to get my job done". But fewer (critical and hopefully highly paid) people will be engaged in a profession where understanding the details is the job and there's no way around it.
In Asimov's foundation novel this is a recurring theme: they can't find people who can work on designing or maintaining nuclear power. This eventually leads to stagnation. AI tools can prevent this stagnation only if mankind uses the freeing of mental burden with the help of AI to work on higher set of problems. But if the tools are used merely as butlers then the pessimistic outcome is more likely.
The general tendency to lack of details can also give edge in some cases. Imagine if everyone is using similar AI tools to understand company annual reports which gives a nice, tiktok style summary. Then an investor doing the dirty work to go through the details may find things that are missed by the 'algo'.
I did 2 of his courses and RoI in terms of learning is huge. The crucial point is: he starts with an empty file and codes up the stuff with you at an understandable pace, line by line. A lot of concepts I just understood in hand-waving sense became much more clearer after going through this routine.
I'm planning to next take his Physics course but maybe code it up in Rust so that I'm not re-inventing some wheels (like dynamic arrays). But if anyone wants to brush up on most important pieces of maths: vectors, matrices, calculus I will heartily recommend his courses.
Google is afraid that ads driven business models might get dislodged for "give me a service and I will pay you money for that" model. That will be a nail in the coffin. Imagine someone brings a marginally better conversational search engine with a low enough subscription price that some people migrate to use it rather than ad infested, SEO driven web results.
Also, with benefit of hindsight, Microsoft seems to be planning to attack Google from the flanks.. competition on a different business model than a better ad driven engine.
Sundar isn't exceptionally promising is spot on. We have so many cases of small business people being locked out of their accounts etc. They seemed to have preferred being a cool engineering company, not a B2B product company that other businesses can trust.
It is probably incorrect question to ask, but do some Google employees anonymously feel that lot of culture wars waged within company (Damore episode, the fights of ethics AI team with leadership) have caused the company to loose focus? I'm not disregarding moralities of the positions, just thinking if it was the case that many senior executives have to spend time firefighting company image in media due to these episodes.
By and large mathematics education has missed the point of invention of computers. It is only occasionally used to make a point. We should be teaching mathematics with a programming first approach: get your function code to compile, ponder on its signature, write some test cases to really understand what is going on.
Sussman (who wrote the famous SICP book) wrote another book structure and interpretation of classical mechanics. Tough book to go through. But they start with the same premise: mathematical notation is confusing (and hand wavy at times). A better symbolic notation should reveal enough details to be able to code up the mathematics in a program. I found this approach to be bang on target, but could never get enough time to actually go through the book.
And I realised why the 'let us build it up from scratch' books work. They force you to think about the function signatures and shape of objects passed to each function. This approach reveals gaps in our understanding much better. For example, F=ma is looks like an algebraic statement, hiding the fact that `a` on the right is about time evolution of the system through the derivative.
Steven Strogatz made a funny quote in his infinite powers book. (I'm paraphrasing), if Newton was doing this in today's era he might create a flipbook animation to make this point and not symbols.
The deeper fact that Stephen Wolfram talks about in comparing ChatGPT vs Symbolic AI is true too. A layer deeper from ChatGPT is just a bunch of floats. So it is a lot harder to debug why the system arrived at the wrong result. For symbolic computations there might be a way to resolve how the expressions were transformed by the compiler and where it went off track, a debugging in true sense.
It is my gut feeling that we might see a next AI winter in a decade where we will finally start combining symbolic and connectionist approaches.
Two ways to out from the existential dread: one startup on a problem statement that feels exciting to you, so that you get to pick your tools, processes and what not. But this is not a realistic/affordable option for most people. Also, if your startup achieves even a moderate level of success, you'll be back to solving boring problems.
The other is to find a pockets of software development outside work, where you can still solve interesting problems, like developing small games. Joseph White, who created the famed programmable fantasy game console Pico-8 alludes to this (https://youtu.be/87jfTIWosBw?t=1080) ! It was an explicit goal to build a tool to solve cute, interesting problems because routine software development feels like glueing things together. The whole youtube talk linked above is worth watching. But the trick here is to forgo commercial motivations. People make fun, cute little games on Pico 8, distribute them as cartridges. Occasionally, there's a hit like Celeste. But otherwise its a small community of people solving problems and building things just for fun.
But as I grew older I realised I need to pick my battles wisely. I still use emacs for some simple edit tasks and tramp for remote server file edits. But for normal day to day development that pays the bills, I've switched to vscode.
GUI based apps are offering better experience. For example, I can still configure Emacs to work well with Golang or Python. But the out of the box setup you get by clicking couple of buttons in VSCode is far superior.
It is separate matter if terminal world is getting abandoned is right thing or not. Maybe it is relic of a different era: when computers were time shared machines and connecting to them with terminal was the only option. But I increasingly realised that isolating myself out of better GUI based options is not the right battle to pick.
Most user loved or impact making products like YouTube and Android have been acquisitions. I tried setting up Google Analytics for a business and touched it after many years. And the sheer complexity of the documentation and puzzling talk of UA to GA4 migration left me with a sour taste. I instantly switched to setting up plausible.io instead.
Notwithstanding the popularity of their tech ecosystem (because they are coming from Google), it is often painful to use. For all the flak Meta gets, the developer oriented technologies released by them: Thrift, PyTorch are fun to use as compared to Protobuf, Tensorflow.
But as someone who loves small companies taking on giants, it is heartening sign. Hopefully with ineffective leadership and employees focused on activism instead of actually building things, in next five years, un-bundling of Google will happen. Vertically focused players will bleed them by thousand cuts.