On the other hand, real world performance is really the only useful metric for matrix multiplication. I don’t really care about the theoretical performance, or the number of operations. Not disagreeing with your take that the claim is grandiose, just pointing out that finding a generalizable way to automate the improvement of what is almost certainly the most important mathematical operation a computer can do is worth some attention. It also suggests other mathematical operations could be improved this way as well - potentially algorithms that haven’t gotten as much theoretical or practical attention as matrix multiplication.
With all that said, they even point out that other people have worked on this before using other optimization strategies. I’d guess they got into Nature either by using the Deepmind name, because Nature really loves reinforcement learning because it’s a cool topic that draws views, or because they’re the first group to find their algorithm leads to real improvements. (Probably a mixture of all three, I’d guess)
The kernel can’t fail to complete its operations, because then the entire system crashes and no logs are created. Instead, you can finish the operation and check the result.
Somewhat tangential, but at my work we have found WebEx’s background noise removal to be absolutely amazing. So many times we’ve had someone in a meeting say “sorry about X/Y/Z, it’s so noisy”, and the rest of us won’t hear a thing. This sorta tech has gotten so good, and is a really nice quality of life improvement for remote work. (Or for meetings with people in noisy offices of course)
Considering basically all of modern technology has been invented in the last 150 years, and just how far we have come in that time, i think it’s relatively safe to assume that in 100x that time (say 15000 years in the future) we will have either destroyed ourselves, colonized other portions of the solar system, or significantly terraformed earth and/or built megastructures in space in such a way as to be able to better control the environment.
In each library, their operations need to be implemented for each backend separately (CPU, GPU, TPU, etc). All of these libraries support CUDA as their default GPU implementation because it’s by far the largest in terms of market share. But since Apple GPUs do not implement CUDA or a translation layer (they use Metal, Apples graphics and compute acceleration library), that means all those mathematical operations need to be rewritten targeting Metal before torch can even communicate usefully with an M1/M2 GPU. That doesn’t even touch on the fact that different backends need to have work scheduled on them slightly differently. Some expect a graph of operations to be submitted, some are just submitted as a series of asynchronous operations, etc.
Also just wanted to point out that torch does support Apple GPUs now, however.
Complete speculation on my part, but I think the biggest appeal is supply side, that people can be Internet personalities with an avatar without sharing their real identity. That helps a lot of people who would like to start sharing content but are otherwise too shy, or for various reasons think they can’t/shouldn’t use their real identity.
On the consumption side, I find the VTuber method to be more compelling than pure voiceovers, even if it’s silly. An avatar helps create a sense of engagement. It’s also interesting to see what sort of characters people come up with.
Strongly agree. Bandwidth and latency are the two big limiting factors always though, and unfortunately they tend to improve slower than compute. I think right now the fastest infiniband you can achieve is 3-400 GB/s, and compare that to the H100 HBM3 memory bandwidth at up to 3 TB/s. So I don’t see a time in the near future where we can naively treat our server fleets as a truly unified machine.
On the other hand, work being done in the big three DL packages for making distributed training easier has been quite nice. I know the least about TF, but their dtensor looks promising. JAX’s entire distributed paradigm using pmap/xmap etc make certain classes of models very easy to distribute. The one I’m following most closely though is Pytorch and their sharded tensor, and it looks like they’re planning on implementing native distributed ops powered by their RPC framework, which should make full tensor- and model-parallelism significantly easier.
I think for inference, analogue makes a ton of sense, and within a pretty short timeline (10 years maybe?) we’ll see it deployed for those workloads.
For training, I’m certainly interested but not at all convinced that it will dominate. I work on multiple projects right now where the reduced dynamic range of analogue signals would be a complete non-starter given the problem domain. I’m not sure how they get around that.
Beyond the comparison to the brain that the other replier gave, you also have to consider that some problems are just big. Really big. Not every AI problem is NLP or CV. For instance, in physical simulations, the representation of the system you’re simulating could be on the order of 10s of gigabytes (a very large mesh, a fine 3D grid, etc.) These are the kinds of problems I work on, and we massively benefit every time a new chip comes out with more memory or increased memory bandwidth, because a single training example, plus it’s activations, the model, and optimizer state, can take about a half a TB of memory or more depending on model size.
I’d also point out that as hardware devices get better, it’s good for everyone. It allows the large corporations to train even larger, more capable models, and it allows smaller players to train models they couldn’t have afforded before on smaller budgets. A rising tide lifts all boats, so to speak. Algorithmic and efficiency improvements are also important, but they’re additive, not a replacement.
And while combining models is not novel and is certainly an avenue that should and is being explored, the models that have the most generalizability for recombination are these extremely large models! They have the capacity to learn very generalized patterns that are then useful for far more transfer/combination tasks.
I’ve had the chance to test a few of the up-and-coming dedicated AI accelerators, and except for the TPU, none of them gave a speed up or perf/watt improvement that was worth the additional complexity of using something that didn’t natively support Cuda. (And the reason the TPU is different is because XLA is a first class citizen for TF/JAX, and has pretty good support in pytorch)
And I definitely see some future for FP16/BF16 or INT8 devices for inference, but I don’t know how widespread such devices will become for training. For many types of models, they simply won’t converge if they don’t at least use a mixed precision scheme. And there are certain problems where even for inference you get a significant benefit from using FP32 - for instance, I work on building DL surrogate models for physical simulations. We get better results with the increased dynamic range and precision.
Very cool to see the rundowns like this. As someone in the ML space, I’m a little concerned that they may be ceding the AI accelerator market to Nvidia. The fact it’s exposed as two separate GPUs, the focus on fp64, etc, all point to them aiming more at traditional HPC workloads than at AI. To be clear, I don’t think that’s a terrible move. HPC is a large market, and these cards look like they’re going to be great there. My concern is that there is no one really trying to challenge the Nvidia A100/H100+Cuda ecosystem. I’d love to see some better competition, both to bring down prices as well as to spur more innovation.
From what I’ve seen of Biren’s new chip it’s nice but won’t be competitive with any of Nvidia’s chips. Maybe in a few generations it’ll be more compelling, and they can build out their software stack. AMD seems to be aiming directly for HPC. Maybe Intel and their OneAPI strategy will get there? I want to root for Intel, but it’s hard to have a ton of confidence after watching their performance through the late 2010s.
Current architecture, no - this big+little architecture just started with the last generation, like a year ago I think? However, if you mean process node, then yes, I think this is on the same one they’ve been using for awhile. But they have 2-3 more nodes coming in the next 3-4 years. If they don’t screw those up, they should be ok. That’s a big if, though.
They’re also changing to a chiplet-based architecture with their next Xeon line which will be interesting.
This is… amazing misinformed. I am assuming you’ve never done scientific simulation work if you think that. Physical simulations in many fields get better due to increases in compute, memory, and bandwidth faster than they do from algorithmic improvements (there are only so many algorithmic improvements one can make to a PDE solver). And certain problems simply can’t be simulated until a certain amount of compute (and more importantly memory) are available.
And while some of the time the entire cluster will be given to a single large scale project, most of the time it will be acting as a massive GPU farm for all sorts of research. A win-win for everyone.
I’d be interested in what you mean by that - VC investment has slowed down, but investment by large firms (tech or otherwise) is only increasing.
And with large language models, generative models, diffusion models, etc becoming more and more common its becoming less novel, but certainly not less popular.
I don’t see how it’s any different for EU long term, even if they haven’t caught up to the US/other markets yet. Automatic ICE vehicles are more fuel efficient, since they can have significantly more gears. And electric cars (usually) only have 1 gear, completely getting rid of the need for any gear shifting at all. The combination of electrification and the continual march to higher efficiency and more performant powertrains naturally leads to a reduction in manuals, looking 10-15 years out.
I think there are two reasons. First, most people might be mildly annoyed by the ads at most, and certainly wouldn’t make a buying decision based on them. Second, it’s almost impossible for new companies to compete on both quality and price at the same time, unless they’re subsidizing the product somehow (ads, perhaps? :) or as an unsustainable VC funded business I guess). How often is it that we see a new company that can make a smartphone that has the fit and finish, camera quality, and specs of an equivalently priced iPhone/Samsung/Pixel phone?
Many of the top rated comments in here are kind of missing the point. “No touch screens” isn’t the optimal configuration. Instead, you want buttons on the dash or wheel for commonly used actions, and you want less common actions on a nice large touch screen that also functions as a good GPS screen. The touch screen, since it can have scrollable menus, gives significantly more customization opportunity than pure physical controls allow.
I’d also point out many of the things they asked the driver to do are things you wouldn’t normally be doing once you’re already driving, you’d do then before you started. A much more representative sample of things you’d do while driving would be something like “turn down volume, turn on windshield wipers”.
Honestly I’ve never considered an analog computer, and I’m a physicist but not a lattice expert. While an analog computer would let you use real numbers, you’d still need a way to store state at every point in space, which would lead to the infinite memory problem (and you’d need infinite compute to operate on the infinite memory). Perhaps there’s a clever way to get around that, but my suspicion is that if it were possible someone would have done it already.
I apologize I am actually running out the door at this point, so I can’t reply in a ton of detail (I’ll try to remember to do so later)
But the point to remember is that these aren’t normal algebraic equations, they’re based on the quantum operators, right? And so we can always do symbolic math on them, but to get numeric results, they have to be instantiated at some point. The operators exist at every point in space, and some of the operators can be approximated on a lattice discretization, but some of them cease to be well defined as soon as there is any distance between the operators (so they require true real numbers, not floating point numbers - ergo, infinite memory).
One point that i think is missing is that there’s a bit of a difference between numerical solutions to QFT equations (like the calculation of g-2 referenced in the paper) and lattice calculations in that in general those numerical calculations are giving averaged quantities. We couldn’t, for instance, take that average quantity and use it instead of the dynamically fluctuating quantity in a lattice simulation. We could run a lattice simulation and estimate the value of g-2 from the lattice to see how well our discretization -> continuum extrapolation worked. But we couldn’t go backwards from the numerical solution to the lattice, so to speak.
The point is that it can’t be simulated on any lattice of any density. It doesn’t matter how fine the lattice is compared to the sensitivity of the measurement. For your case you showed, it would be like if x^2 simply didn’t exist on a lattice, and couldn’t be calculated by a computer no matter how much memory you threw at the problem.
Certain parts of QFT work fine in lattice based calculations (lattice-QCD, for instance, where it’s simulated on a certain lattice scale, and then extrapolated to the continuous limit), and some seemingly don’t work at all.
“Fudge factor” massively undersells it. For instance, measurements from the bullet cluster are quite compelling that there is significant mass in a galaxy that we don’t directly measure. That’s the definition of dark matter.
You can think of it, at a very high level, being the difference between a force mediator that has no self interaction (photon) and a force mediator that does have self-interaction (gluon). Gluons interact with each other, whereas photons don’t. Even this by itself isn’t naively enough to get all the strong force’s interesting behavior. But the math works out in such a way that when you start to pull two strongly bound particles apart, the color field forms a flux tube of gluons between the two particles, and the force required to pull them apart further remains approximately constant (in other words the energy stored per unit length in the flux tube is ~constant). This only happens because of the self interaction.
It doesn’t violate energy conservation at all! For the trivial reason that field itself is not the entity using energy to displace two bound particles ;)
What? If you think a government can’t enforce crypto bans, then you haven’t thought about it very much. MITM certs to snoop all internet traffic, tying all devices to real identities as a prerequisite for internet access, making it a crime for merchants to accept any payment but the national currency, etc, etc.
In the long term, complete displacement of hydrocarbons from the economy is probably the end goal. However, we are nowhere near being able to do that right now. Obviously for the billions of vehicles on the road and the current grid power generation we need not just solar/wind generation, but also massive amounts of grid storage (or massive increases in nuclear) and vehicle batteries. But in industry there are tons of chemical processes that also require hydrocarbons that we haven’t found suitable replacements for yet.
Because it’s not clear how fast we can get absolutely massive amounts of grid storage or nuclear deployed, finding ways to make effectively carbon-neutral natural gas/petroleum is a good step in the right direction, where we can continue to use some hydrocarbons without a net increase in greenhouse gasses in the atmosphere and oceans.
It’s also not clear if we will be able to replace all of our current petroleum reliant industrial processes with non-carbonized alternatives, and in those cases, this will be the absolute best case for us.
Yes, your explanation is essentially correct. There is work done in the areas you’re talking about - essentially forcing models to more explicitly learn “concepts” - and in very large language models that seems to be emerging naturally. But current vision models would almost certainly break when trying to identify a vehicle from the bottom shot if it had never seen a vehicles undercarriage during training. Current vision models are capable of identifying vehicles from arbitrary angles (when viewed from the side/head on) and in arbitrary shades/colors/models/etc, and that’s about the amount of extrapolation we’d be talking about.
Learning in this sense means it’s able to extrapolate to unseen data (I.e., it’s learned correlations that are inherent to the problem, not just inherent in the training data). Memorizing, on the other hand, implies that it will do very well on any examples in the domain of the training data, and will break (sometimes pretty hilariously) on unseen data.
The fact that refitting once a day improves real world performance actually makes me think that the problem/data they work on is highly non-stationary, not that the model is memorizing. If it was purely memorizing then the model would perform poorly on all non-training data, and would not work for even one day.
To your point about large datasets - the large datasets are what allow learning to take place. With the most common forms of models we have now, they will memorize when they only have a few examples, and only “learn” when the training data is large enough. There is work to improve learning from a handful of examples, but in many of these cases they require a model that was already trained on the domain in question, and then are specialized to a specific use-case.