1,905 karma · joined June 17, 2016
Website: https://zhenghaotan.com Email: hi at zhenghaotan dot com Twitter: @zhenghaooo
YC Badge: 0x4405dd9f2de9c10f45ab4013fc56c461728a4795
https://pytorch.org/blog/pytorch-for-amd-rocm-platform-now-a...
Agree with this insight. One thing Nvidia got right was a focus on software. They introduced CUDA [1] back in 2007 when the full set of use cases for it didn't seem very obvious. Then their GPUs had Tensor cores, and more complementary software like TensorRT to take full advantage of them post deep learning boom.
Right as Nvidia reported insane earnings beat too [2]. Would love more players in this space for sure.
[1] - https://en.wikipedia.org/wiki/CUDA [2] - https://www.cnbc.com/2023/05/24/nvidia-nvda-earnings-report-...
It's incredibly insane but I manage it by adding a namespace postfix so it's easier to identify which when autocompleting login forms. For example, "Google (Personal), "Google (UMich)" etc.
[1] - https://www.modular.com/mojo [2] - https://ai.facebook.com/blog/meta-training-inference-acceler...
> Sure, the actual implementations can be tricky, but a single engineer (trained in it) can get that done for a specific hardware in a couple months.
I want to agree with you on this, but in practice, it's...
1. Hard to hire that engineer with deep expertise in handwritten kernels. CUDA engineers are still hard to come by and doesn't scale with productionized AI engineering demand.
2. "A few months" is a tough pill to swallow from an engineering roadmap POV, especially when models are deployed on a monthly basis. Most of the hand tuning efforts aren't scalable and will have to be done again on most iterations. This is especially true in reinforcement learning and robotics.
> But, in retrospect, the actual value add from a system that does all this for you is quite low. It's a realization I've been struggling to accept.
Yeah I remain neutral on this. On one hand, I can see that especially having to invest significant engineering effort (see point 2 above). On the other hand, you won't really know until you start benchmarking these models (and as you should).
The Fed is now stuck in a really hard place - raise too much too fast and you get many more SVBs going belly up, posing a larger systemic risk, assuming we even "make it out of the woods". Pause / raise slowly, and you risk inflation running red hot, and people who are technically employed, live paycheck to paycheck start to rely on food banks to cover the rapid raise in cost of living - a tax on everybody.
The Fed is also fighting other dynamics like supply consequences of the Ukraine war, or energy crises, some of which are transitory, some are not. Still lots of uncertainty.
All this on top of a tug of (policy) war between an administration that favors handing out more packages, and the Fed's duty in achieving pricing stability. Can't see this ending well.
Lots of innovation here. Time for a proper DL compiler.
TensorRT is proprietary to Nvidia and Nvidia hardware. You'd take a {PyTorch, Tensorflow, <insert some other ML framework>} model and "export / convert" it into essentially a binary. Assuming all goes well (and in practice rarely does, at least on first try - more on this later), you now automatically leverage other Nvidia card features such as Tensor cores and can serve a model that runs significantly faster.
The problem is TensorRT being exclusive to Nvidia. The APIs for doing more advanced ML techniques like deep learning optimization requires significant lock-in to their APIs, if they are even available in the first place. And all these assuming they work as documented.
OpenXLA (and other players in the ecosystem like TVM) aim to "democratize" this so there are more support both upstream (# of supported ML frameworks) and downstream (# of hardware accelerators other than Nvidia). It's yet another layer or two that ML compiler engineers will need to stitch together, but once implemented, they in theory can do a lot of optimization techniques largely independent of the hardware targets underneath.
Note that further down in the article they mention other compiler frameworks like MLIR. You can then hypothetically lower (compiler terminology) it to a TensorRT MLIR dialect that then in turn runs on the Nvidia GPU.
All these processes become items on a hiring checklist, but to your point, it doesn't guarantee hiring a great engineer. Just one of those tragedies embedded alongside other big co bureaucracy processes.
Reddit was also partly successful in that regard because it catered to niche communities who ask the most interesting questions and, trolls aside, responses from the community that's generally trustworthy. There are literally subreddits with discussions on solutions that are deemed better than what you get from customer support. Reddit also doesn't have "promoted" content / responses, which is largely the issue with Google ads outlined by author.
I went through the whole phase of adding a bunch of plugins, custom macros and have a huge .vimrc file that, once in a while, would break if I was using the wrong vim variant {vi, vim, nvim}. For the sake of this argument, emacs users face some form of the same too.
It got to a point where all my hardcore configurations made my editor experience diverge even further away from the promised "out of the box" experience, and not on par with autocomplete functionalities available in modern IDEs like VSCode.
So now I'm on VSCode. A M1 MacBook can handle it, so I told myself to quit being a masochist, and that itself saves me hours especially navigating a big and complex codebase. But I remember how to use vim, and that comes in handy if I have to SSH into some remote machine and make some tweaks to scripts.
[1] - https://github.com/pallets/click [2] - https://click.palletsprojects.com/en/8.1.x/why/#why-not-argp...
Exceptions should never be used if the control flow has means to recovery. For example, if-else statements with more different, downstream instructions.
Thinking more abstractly, there is benefit for code and data to live "next" to each other, if possible. Atomically committed to a codebase and the latter loaded / used by the former without connecting to yet another workflow.
In my experience, you need to first define a small set of north star metrics that lead to the promised land, which could be years, if not a decade away. Ideally, this should not be more than 3 so there's focus in the long term. If you define this well, everybody will understand the significance of it and be aligned to hit those objectives. Any increase, exponential or not, is progress the team should feel proud of. This is 80% of the "selling" for long time scale projects.
Once you've done that, all other initiatives should focus on improving only these metrics. Generally speaking, very aggressive milestones and keeping that momentum going, while not punishing teams for missing targets will move the needle faster than the other way round.