Nvidia announces financial results for second quarter fiscal 2024
nvidianews.nvidia.com
nvidianews.nvidia.com
This is what having a monopoly looks like !
This is also why companies that manufacture their cards didn't report any uptick in profits. I'm wondering how this play out in some months ? Do they have any pricing power with respect to NVidia ? Or NVidia could just switch to another manufacturer ?
Hopefully Intel continues to improve it's GPU offerings
I don't think GPUs are commoditized. You can't swap a Nvida GPU with a AMD GPU, and get the same performance/results.
The HIP interface even has almost exact interoperability with CUDA, so you don't have to rewrite your code.
Not sure AMD would be considered a big player, what would be the percentage threshold for that?
According to the Steam Hardware (& Software) Survey (https://store.steampowered.com/hwsurvey/Steam-Hardware-Softw...), ~75% of computers with Steam running has a NVIDIA GPU, while ~15% has a AMD GPU.
AMD is the closest to a competitor NVIDIA has, but they are also very far away from even being close to their market-share.
I'm sure in AI/ML spaces, NVIDIA holds a even higher market-share due to CUDA and the rest of the ecosystem, at least in gaming things are pretty much "plug and play" when it comes to switching between AMD/NVIDIA hardware, but no such luck in most cases with AI/ML.
and thats the consumer market, which lets say is 30% of the B2B enterprise market, which is probably even higher % nvidia
I'd wager the enterprise market is 90% intel integrated graphics.
It's the cheapest option, the best for laptop battery life, and modern integrated graphics can run excel and a web browser at 4k just fine.
As someone who has been in the AI/ML space for over a decade, and even had an AMD/Radeon card for more than half of that, I can't help but feel that this is partially AMD's own fault.
For many, many years it seemed to me that AMD just didn't take AI/ML seriously whereas, for all it's faults, NVIDIA seemed to catch on very early that ML presented a tremendous potential market.
To this day getting things like Stable Diffusion to run on an AMD card requires extra work. At least from my perspective it seems like dedicating a few engineers to getting ROCm working on all major OSes with all major scientific computing/deep learning libraries would have been a pretty good investment.
Is there some context I'm missing for why AMD never caught up in this space?
All systems hit snags. In most, you skid the tires a bit, maybe lose balance. With nvidia you're flying over the handlebars on to the asphalt.
I got snagged by this just about 2 weeks ago. It gets nasty. Not as bad as CUPS, but probably #2.
Do you want solid dependability and settings that are right first time, because you've only got one PC and if the graphics get broken you've got no browser to google for a fix?
Do you want the absolute most up-to-date drivers, to support the very newest GPUs, while running an LTS version of your OS?
Do you want to always run the latest driver version and upgrade without testing or worrying, like we do for web browsers?
Do you want to run CUDA and ML stuff, but also want to run Steam which for some reason wants 32-bit support available?
Do you want to run on a laptop with hybrid graphics, and have suspend/resume work reliably every time?
Do you have a small /boot/ partition, because you expected initrd.img to be 50MB or less?
Do you want to support Secure Boot?
If you want to achieve all these things at once, it'll take you a few tries to get it right :)
[0] https://www.overclock3d.net/news/cpu_mainboard/amd_predicted...
And I don't mean to disparage the switch, just point out that the way it's designed is very different which makes the comparison questionable
Nvidia's dominance today is the product of at least over a decade of work and investments to make better products. Today they are finally reaping their rewards.
>>> struct.pack('f',math.nan)
b'\x00\x00\xc0\x7f'
It was a nice way to debug tensors: render them to the screen, the green sticks out.
https://encycolorpedia.com/76b900 says Nvidia green #76b900.
[Edit] Could not find it under the name, but it shows how color blind users perceive it. And it loads much faster.
https://www.colorhexa.com/76b900
[Edit] Encycolorpedia has a color blindness simulator too. Have to check on desktop.
The thing that caused me to ragequit the AMD ecosystem was when I took an OpenCL program I had been fighting for two days straight and ran it on my buddy's Nvidia system in hopes of getting an error message that might point me in the right direction. Instead, the program just ran, and it ran much faster, even though the nvidia card was theoretically slower.
In terms of quality, I expect the competition to catch up in a generation or two, but then there is still the decade+ of legacy code to consider. Hopefully with how fast AI/ML churns that isn't actually an insurmountable obstacle.
Where both the sign (s) and the x bits can be anything and it will still be treated as a NaN.
There are lots of ways to encode colour, but there would be too much red with RGBA, and ARGB could be almost any opaque colour, but the red channel has to be at least 0x80, which is still too much red.
So NaNs are too red to encode nvidia green.
Wonder what caused the difference in the latter bits being all 1 or 0 together.
practical work with PhysX 13 years ago: https://www.realworldtech.com/physx87/
"For Nvidia, decreasing the baseline CPU performance by using x87 instructions and a single thread makes GPUs look better."
Nvidia magically released PhysX compiled with multithreading enabled and without flags disabling SSE a week after this publication. But couple of days before release they made those funny statements:
"It's fair to say we've got more room to improve on the CPU. But it's not fair to say, in the words of that article, that we're intentionally hobbling the CPU," Skolones told Ars.
"nobody ever asked for it, and it wouldn't help real games anyway because the bottlenecks are elsewhere"
>Nvidia's dominance today is the product of at least over a decade of work
Nvidias decade of work:
Ubisoft comments on Assassin’s Creed DX10.1 controversy https://techreport.com/news/14707/ubisoft-comments-on-assass...
AMD says Nvidia’s GameWorks “completely sabotaged” Witcher 3 performance https://arstechnica.com/gaming/2015/05/amd-says-nvidias-game...
AMD Dubs Nvidia’s GameWorks Tragic And Damaging, Fight Over The Developer Program Continues https://wccftech.com/fight-nvidias-gameworks-continues-amd-c...
"Number one: Nvidia Gameworks typically damages the performance on Nvidia hardware as well, which is a bit tragic really. It certainly feels like it’s about reducing the performance, even on high-end graphics cards, so that people have to buy something new.
"That’s the consequence of it, whether it’s intended or not - and I guess I can’t read anyone’s minds so I can’t tell you what their intention is. But the consequence of it is it brings PCs to their knees when it’s unnecessary. And if you look at Crysis 2 in particular, you see that they’re tessellating water that’s not visible to millions of triangles every frame, and they’re tessellating blocks of concrete – essentially large rectangular objects – and generating millions of triangles per frame which are useless."
"The world's greatest virtual concrete slab" https://web.archive.org/web/20121002034311/http://techreport... (images "somehow" vanished from original article at techreport where Nvidia runs marketing campaigns)
"Unnecessary geometric detail slows down all GPUs, of course, but it just so happens to have a much larger effect on DX11-capable AMD Radeons than it does on DX11-capable Nvidia GeForces. The Fermi architecture underlying all DX11-class GeForce GPUs dedicates more attention (and transistors) to achieving high geometry processing throughput than the competing Radeon GPU architectures."
From the arstechnica article about Witcher 3.
How dare Nvidia optimize their game enhancing effects for Nvidia hardware and forget to do it for their competitors hardware as well! And as for a lot of these complaints, could it be that a lot of companies only optimize for hardware that has the largest market share?
According to the steam hardware survey in July of 2023, Nvidia accounts for 75% of the GPUs[0]. Nvidia and Amd have a lot of incompatibilities, and it can be hard to make the same code performant on both. It makes sense, as a game company, to prioritize optimizations for the largest market. No collusion and evil corporate mega lord scheming needed for this.
Edit: Also, Nvidia does put out a lot of research efforts for free. Path rendering on the GPU for example (PhysX being another). You can find research papers and videos published by Nvidia for these things. I would consider that practical work. You can hate on Nvidia for lots of things, but this is one thing I find weird to be combative over.
Second Edit: Also, why do you find the statements Nvidia said about the PhysX improvements funny? They’re right. Most games 13 years ago left a lot of idle time on the GPU while the CPU worked overtime to do logic, physics, sound, culling, etc. Lots of that stuff has also been moved to the GPU to minimize the amount of idle time on either the CPU or GPU. Nothing funny about what they said there.
Yes, but also Nvidia partners directly with companies too (money can/does change hands).
Now the flip side is: so does AMD.
And their antics shutting out Nvidia (eg: FSR only, no DLSS) aren't being received well, not the least because their offerings are objectively inferior to Nvidia's.
https://old.reddit.com/r/pcgaming/comments/3vppv1/crysis_2_t...
Instead, they've been going after the CPU market (and winning), HPC/scientific computing (high FP64 performance, in contrast to Nvidia's focus on low-precision ML compute), and integrating Xilinx.
However, I agree that it's an unfortunate situation, and I hope AMD becomes competitive in this space soon.
However, a lot of this has to do with the fact that AMD was on the brink of bankruptcy before the launch of Zen in 2016 (when their share price was ~$10). They simply did not have the capital to the kind of things Nvidia was doing (since '08 ?).
The bet on OpenCL and the 'open-source' community failed. However, ROCM/HIP etc. really seems to be catching up (I even see them packaged on Arch linux).
There are now distro-provided packages on Arch, Gentoo, Debian, Ubuntu, and Fedora.
I believe it's a highly undervalued stock right now.
They just don't have those capabilities in their consumer GPUs.
AMD is also nearly 50/50 with nVidia for supercomputers in the Top500 (and dominates at the top)
It took a few years after completeing the massive purchase of Xilinx to get going, but they are picking up speed rapidly.
It's not like Intel and AMD don't exist, but if everyone is pushing each other at the door for Nvidia chips..
MAMAA
And by saying FAAMG instead of FAANG I make the statement that Cramer was high as a kite when he put Netflix in there instead of Microsoft. Today you might make a case for an N, but not at Microsoft's expense.
the other companies have to up their game to compete with a company that has been executing well for 20+ years.
They prices were already double, it was just scummy resellers capturing that value rather than Nvidia.
Yep and we are suffering as a result. Want the best in computing and CUDA? Give M$ and Nvidia money.
Linux and Nvidia don't play well. Apple doesn't even attempt to try.
The absolute state of computing right here.
That level of programming gave such rewarding moments in between difficult debugging sessions. When I wanted to test a new kernel driver build I needed to walk into some massive room with all of these interconnected machines that emulated the non-yet-fabricated GPU hardware. One of the full time people on my team was going insane trying to track down a memory corruption issue between GPU memory and main memory when things paged out the entire time I was there.
Back then the stock was around $7/share and the CEO announced a 10% paycut across the board (even including my intern salary) and had an all hands with everyone in the cafeteria. It's pretty cool they went from that vulnerable state, with Intel threatening to build in GPU capabilities, to the powerhouse they are today.
How does a 1070 fare versus a 4090 in Hogwarts Legacy or another modern game at 1440 or 4K?
I expect that if you have a 4090, you have an Intel or AMD CPU that exposes a core clock multiplier. You could run this benchmark with whatever value it's at, then reduce the modifier by say, half. That should halve your CPU's clock rate, and I'm guessing you'll see the frame rate decline similarly. You can conclude that the game is "CPU bound", then.
Even if your GPU was infinitely fast, you have to remember that game developers are not optimizing for the game event loop to run in sub-millisecond times. If the core loop in the game takes under 16ms to run on a common CPU, such as one in a console: that's better than 60hz and the overwhelming majority of video game players will never see a benefit.
Some game developers, I see someone in another thread mentioned Doom Eternal, pride themselves on that optimization. With a fast enough CPU and GPU, you could probably reach 1000 FPS on Doom Eternal. A quick search suggests this might have been accomplished with a liquid cooled, this was done with a liquid nitrogen cooled PC with a 6.6GHz CPU: https://www.pcgamer.com/heres-doom-eternal-running-at-1000-f...
My guess is that we'll see improvements at closer to the current rate rather than at an increasing rate.
A much better benchmark would be to take a game designed for 120 or fewer FPS on a 4090 and try it on a 1070.
I've never had hardware to run above 1080p at 60fps (monitors, GPUs), so my desires are a bit orthogonal - I wish more games let you customize the graphics settings so that you could maintain a consistent framerate (be it 30 on a low spec laptop, 60 on a regular PC or more fps) at a resolution of your choice.
Things like switching between baked or dynamic shadows, their type/filtering, things like ambient occlusion and tessellation, various post processing effects and filters, model LOD limits and texture resoltion, resolution scaling and so on.
More so, it would actually be nice if games let you download a version with lower fidelity assets (like War Thunder sort of does, for example), so you don't need 100 GB of models and textures if you're realistically only ever going to see 20 GB of those on your hardware.
Thankfully, most modern game engines scale both up and down decently, for example, Unity's URP (though historically a bit half baked and fragmented the community/assets). It's just up to the developers to get over the hubris of wanting "low" settings to still be pretty, that choice should be up to the user.
No. Please don't spread this nonsense rumor. It hasn't ever been true and still isn't.
There are always diminishing returns, but human vision is perfectly capable of noticing the difference between 60 and say, 120. You can literally try this yourself just moving your mouse around on a 120Hz monitor and then capping the refresh rate. In general human vision is really complicated and not tied to any specific "speed".
Also, displays are more complicated. There isn't just a refresh rate, there's differing pixel responses at differing brightness and a variety of different processing delays all the way from the moment you put in some hardware input (like the mouse), to the final result. It's a huge chain.
The end result is that two 120hz displays can behave completely differently in terms of motion clarity.
Do you have sources that show otherwise and back up your claim that this idea is “nonsense”? I will dig up some scientific sources. Are you perhaps reading into my comment and not responding to what I said literally?
You aren’t really addressing what was main point: that fps throughput isn’t the reason for high frame rates in games. The primary reason for this happening is to decrease latency.
We wouldn’t need 500fps for games if we lowered the latency. Or at the very least, the benefits would be much lower. Reducing latency is a great reason to want high fps, but there are other ways to reduce latency.
You replied to the wrong comment, btw. I almost didn’t catch your reply.
Edit: links that discuss the measured speed of human perception:
http://web.cs.wpi.edu/~claypool/papers/fr/fulltext.pdf
(Study is limited to 60 fps, but clearly shows the trend of diminishing returns.)
https://www.healthline.com/health/human-eye-fps
(Argues it’s higher than 60 for some tasks… up to 90 or 100fps.)
https://www.rtings.com/monitor/learn/60hz-vs-144hz-vs-240hz
“there are diminishing returns when it comes to the refresh rate. Most people can perceive improvements to smoothness and responsiveness up to around 240Hz; however, the difference between a 240Hz and 360Hz panel is so small that even competitive gamers might have a hard time telling them apart. If you have a choice between a 1440p 240Hz and a 1080p 360Hz monitor, you're probably better off getting the 1440p option, as the increase in resolution has a much larger impact on the overall user experience.”
https://www.neurotrackerx.com/post/5-answers-to-the-speed-li...
(Mentions seeing a single frame of color at 500fps. This is true! Humans can see a flash of light that’s much shorter than 2ms. For perceiving imagery and tracking motion, the available evidence shows little benefit to going higher than 120fps.)
>my statement that 500 fps is not 5x better than 100 fps to a human
I never said that. I also stated that I know about diminishing returns. You claimed "There’s a good reason to never go above 70fp". There absolutely is a large difference in motion clarity between 70 and something like 120/240/etc. Is the difference from 70->120 as large as 30->60? No, absolutely not. But it is significant and can be seen easily with a cheap monitor.
>You aren’t really addressing what was main point: that fps throughput isn’t the reason for high frame rates in games. The primary reason for this happening is to decrease latency.
>We wouldn’t need 500fps for games if we lowered the latency. Or at the very least, the benefits would be much lower. Reducing latency is a great reason to want high fps, but there are other ways to reduce latency.
While there is a latency improvement and some people care about that, the motion clarity is also significantly improved. (again, just drag some windows around on a 120hz monitor) That's true of movies just as well as games. Movies have pulled a lot of tricks to mask this issue over the years, but 60 is quickly becoming the standard over 30. (And once bandwidth and processing improves, it's likely some day decades from now it will jump even higher)
>You replied to the wrong comment, btw. I almost didn’t catch your reply.
Yeah, not sure how that happened.
When you said “No. Please don't spread this nonsense rumor. It hasn't ever been true and still isn't.”, combined with the downvote, I assumed you were referring and objecting to everything I said including diminishing returns (even though I see you acknowledging it next paragraph.)
We are mostly agreeing violently, I acknowledge that there’s no known hard fps threshold above which nobody can see something. I acknowledge that there are benefits above 60fps, even if they grow smaller.
But it’s still true that the primary reason games are going to 500fps is for the latency benefits, not for the smoothness or high flicker rate. A frame rate that high isn’t generally perceptible, while the latency of today’s games - a latency of multiple frames - is actually well inside the known measurable threshold of response times. The problem isn’t generally the need for more frames per second, the big problem is the time between input and the visible change on screen.
The other topic that would be nicer to discuss is the quality trade offs. High frame rate takes away from other options.
TAA fundamentally decouples the sample generation from the output generation. TAA samples (even native res) have no direct correspondence to the output grid, they are subpixel sampled and jittered. The output grid takes the samples nearby and uses them to generate an output for each pixel.
TAAU/DLSS2 take this further and decouple the input grid from the output grid resolutions entirely. So now you have a 720p grid feeding a 1080p grid or whatever. Thinking of it as "input/output frames" is clunky especially when (again) the input frame doesn't even correspond directly to the input samples - they're still jittered etc. Think of it as a 720p grid overlaid over a 1080p grid, and DLSS is the transform function between these two (real/continuous) spaces. Samples are randomly (or purposefully) thrown onto that grid.
OLEDs are functionally capable of individual pixel addressing if we wanted to. Current ones probably can scan out lines in non-linear order already, so you could scan out the center more than the top/bottom, for example. And OLEDs are already capable of effectively >1000hz real-world response time from their pixels. They just are absolutely critically bottlenecked by the ability to shove pixels through the link and monitor controller quickly enough (give me 540p 2000hz mode, you cowards)
This all leads to a question of why you are still sampling and transmitting the image uniformly. If there's parts of the image that are moving faster, render those areas faster, and with more samples! Maybe you render the sword moving at 200fps but the clouds only render at 8fps. And Optical Flow Accelerator can also allow you to identify movement within the raster output and correlate this with input - if you think about oculus framewarp, what happens if we framewarped just one object? Translate it around against the background, stretch it to simulate some motion aspect, etc. And if you further refine that to a 1x1 region, then you have individual pixel framewarp.
You can also have a neural net which suggests which regions are best to render next, for "optimal" total-image quality after the upscaling/framewarp stages, based on object motion across the frame and knowledge of the temporal buffer history depth in a particular pixel/region (and this is the sort of abstract, difficult optimization problem ML is great at). And those pixels actually might be rendered at multiple resolutions in the same frame - there might be "canary pixels" rendered at low resolution that check whether an area has changed at all, before you bother slapping a bunch of samples into it, or you might render superfine samples around a high-motion/high-temporal-frequency area (fences!). So now you have "profile guided rendering" based on DLSS metrics/OFA analysis of the actual scene being rendered right now.
Another random benefit is that "missed frames" lose all meaning. The input side just continues generating input for as long as possible, as many samples at whatever places will be most efficient. If it's not fully done generating input samples when it comes time to start generating output... oh well, some cloud has less temporal history/fewer samples. But DLSS does fine with that! As long as you still have some history for that region you will get some output, and I'm sure if it was important then it'll be first thing scheduled in the next frame.
You can use this idiom with traditional scanout/raster-line monitors, but ideally to take advantage of OLED's ability to draw specific pixels, you probably end up with something that looks a lot like a realtime video codec - you're sending macroblocks/coding tree units that target a specific region of the image with updates. Or you use something like Delta Compression encoding, like with lossless texture compression.
https://en.wikipedia.org/wiki/Coding_tree_unit
This already is sort of what Display Stream Compression is doing, but that's just runlength encoding, and if the monitor can draw arbitrary pixels at will (rather than being limited to lines) then you can do better.
And again, you can't transmit the whole image at 1000fps, but you can target the "minimum error approximation" so that on the whole the image is as correct as possible, considering both high-motion and low-motion areas. This is sort of an example of the concept - notice the scanline errors in high-motion areas. Great demo but linked to the direct example:
https://youtu.be/MWdG413nNkI?t=176
https://trixter.oldskool.org/2014/06/20/8088-domination-post...
This also raises the extremely cursed idea of "compiled sprites" inside the monitor controller - instead of just running a codec, you could put a processor on the other side (like the G-Sync FPGA module) and what you send is actually the program that draws the video you want. Executable Stream Compression, if you will. ;)
(but there's no reason you couldn't put THUMB or RISC-V style compressed instructions in this - and you certainly could change up the processor architecture however you want, as long as you don't mind doing a Gsync-like compatibility story. It makes upgrading capabilities a lot easier if you control both ends of the pipe, that's why NVIDIA did g-sync in the first place! And there is probably nothing more flexible or powerful than allowing arbitrary instruction streams to operate on the framebuffer (or a history buffer, whether frame history or macroblock/instruction history) or to draw directly to the frame itself. With the bottlenecks that DP2.0 and HDMI2.1 present, this is probably the way forward if you want lots more bandwidth out of a given link speed.)
I said a long time ago that this is basically a "JIT/runtime" sort of approach and people kinda laughed or said they didn't get it. But it's funny that trixter actually used the same analogy there for his demo. But basically DLSS is an engine unto itself already, DLSS is what does the rasterizing, and the engine just feeds samples (that are entirely disconnected from the output, they go into the black box and that's the end of it). And with fractional rendering you can basically view that as a big JIT/runtime. DLSS provides quality-of-service to pixels by choosing which ones to schedule for "execution" and with what time quanta, to produce optimal image quality ("total QOS") over the whole image. And then the resulting macroblocks are individually sent over when ready.
Effectively this works like the biological eye - there is no "frame", changes happen dynamically across the whole frame constantly. Or another analogy would be "what if we could 'chase the beam' but across arbitrary pixels in the frame"? If the motion is predictable then you can schedule the rendering so that the output and blit happens at the exact proper time, down to 0.1ms accuracy. It's Reflex for Pixels.
DLSS knows when a person is running and about to peek out from behind a wall. DLSS knows when someone was sitting there and can render the lowest-latency-possible update when they suddenly pop out of cover and AWP you (can't read minds/make network latency disappear, but it can update it as quick as it can). That already shakes out of the motion vector data and just needs to be generalized to a world where you can render out specific regions/lines at 1000hz.
(I guess technically "frame" still exists as a notional concept inside the game loop, you are generating game state in 60fps intervals or whatever, but rendering is totally decoupled from that and you run the GPU on whatever parts of the image would benefit the most from touchups at a particular moment.)
But again you can see how this whole idea fundamentally inverts control of the game engine - Reflex already tells the game when to sleep and when to start processing the next game-loop, now DLSS will tell the engine what pixels it wants sampled, and handles rasterizing the samples and compressing/blitting them to the monitor. The engine is just providing some "ground truth" visual input samples and calling hooks in DLSS.
(sorry, long post, but I've been musing about this elsewhere and I looooove to chitchat lol)
> you are generating game state in 60fps intervals or whatever
This is also an excellent point that might question my suggestion that high frame rate is being used to reduce latency. Games state updates are already decoupled from rendering in lots of games. Having an extremely high render refresh might not mean that the latency between controls and visuals is reduced proportionally. Or maybe it helps but has a limit to how much.
DLSS is an interesting topic here. Do you see it eventually working for fractional updates? We’d possibly need a new style of NN or of inference? DLSS currently operates on a full frame, and the new version even hallucinates interpolated frames to boost fps artificially. This doesn’t help with control latency at all, in fact it makes it worse.
Yeah I've been playing fast and loose with terminology here, there's several overlapping but synergistic ideas that aren't the same things. To try and clean this up:
DLSS1 required the full image, it actually was an image-to-image transformer that "hallucinated" a full-res image from an input image. This sucked and NVIDIA gave it up (except for the 500mb of DLSS models that live eternally in the driver for the 2 games that opted for driver-level model distribution). Nobody cares about this anymore at all.
DLSS2 does not need a full image, because it is a TAAU algorithm that weights samples using a ML model. If there aren't enough samples in an area oh well, you just get crappy output (like immediately after a scene change). It manifests as either obvious resolution pop/detail pop, or visual artifacts on moving/high-res things. IIRC this can be assisted by drawing invisible (1% alpha) objects in motion/fences/etc to "warm up" DLSS sample history on the (invisible) edges before just popping them into existence iirc lol.
You need at least some samples near that area, it can't render from nothing, but DLSS2 is not dependent on rendering out the full image to work - if some unrelated part of the image doesn't have samples, oh well. (and this may allow mGPU scaling with reasonable correctness for partitioning an image!). Personally I consider this "loosy goosy correctness" attribute of ML models to be extremely desirable for GPGPU programming - if some edge case messes up 0.1% of samples, the ability of ML models to just ride over it and spit out a reasonable output is super desirable. This includes things like camera noise, dead pixels, etc. Extremely tolerant of data ingest etc. Like if 10 threads aren't quite finished with their sample output because you don't want to wait for kernelfence sync when it's time to start rendering the output buffer... just start going. It'll be fine.
Fractional rendering is a separate and unrelated idea, but I think the time is right with OLEDs here, and with everyone searching for a way to extend perf/tr with costs spiraling it makes sense to see if you can render "better" imo.
Variable rate sampling is another concept that builds on fractional rendering. Render some areas at a higher rate than others. And again this is something that DLSS2 plays nicely with.
--
DLSS3 is actually the successor to DLSS2 and is supported by all RTX cards (yup). Framegen is one of the features in this, and that is only supported on Ada. Supporting framegen requires the inclusion of Reflex, which does benefit everyone hugely.
Reflex basically flips the "render+wait" model to be a "wait+render", by adding a wait at the start of the game loop that delays until the last possible second to start processing the frame, so it's as fresh (input latency) as possible. And this does legitimately cut latency significantly (by ~half) in highly gpu-bound scenarios. And that gives NVIDIA some headroom to play with in framegen tbh. Igor's Lab and Battlenonsense both found NVIDIA to have much lower click-to-photon latency than AMD Antilag, by like 20-30ms in overwatch f.ex.
Framegen as currently implemented is interpolation and yeah that does increase latency. But NVIDIA do have some headroom to play with there, in CPU-bound situations (which is different!). And tbh most people who actually have used it generally seem to find it not too bad, it's the "eew I tried it at the store and it was awful!"/"i've never tried it" who are most vocal about the latency. It's at least an option in the toolbox (see again: starfield).
I think it is possible to move to extrapolation and I hope the current framegen is only an intermediate step. And I think Optical Flow Accelerator is a really cool building block for that. The performance and precision has improved a bunch over the gens, and now it can support 1x1 object tracking (which I mentioned above as seeming like a significant threshold/milestone) so it's flowing pixels really. I see that as being a Tensor Core-like moment that people scoff at but has big implications in hindsight. Being able to incorporate realtime image data back into the upscaling/TAA pipeline seems big even beyond just framegen itself, I don't doubt DLSS3.5 will make further progress too.
You don't need extrapolation (or interpolation) at all. but if you can extrapolate per-pixel, the ability to do a low-cost "spacewarp" that accomplishes most of the squeeze of a full re-render (in terms of moving edges/texture blocks) at much lower cost could would be very interesting. And the OFA could end up being a key building block in that sort of thing.
--
Again kind of a topic shift but there's also this issue of display connection (there is never enough bandwidth) and whether it's lines or macroblocks etc. That's a capability that's offered by OLEDs in theory, and could be explored with a similar FPGA approach/etc. If you can do that, it pairs with variable rate sampling concepts (and ML input tolerance for bad data) very nicely - render out the regions you're updating whenever they're ready, or whenever is optimal for that element to be drawn (to get minimum error).
And in fact quite a few of these ideas synergize nicely together. If you put it all together.
--
These are all kinda separate in general but quite a few of them synergize if you put them together. And I think the zeitgeist is ripe on some, brian heemskirk was talking about some similar ideas on MLID's show a few months ago (not the most recent appearance).
https://www.igorslab.de/wp-content/uploads/2023/04/Overwatch...
https://youtu.be/7DPqtPFX4xo?t=727
(highly pronounced in this game but)
Yes I agree that driver overhead+latency reduction in the pipeline matters a lot and that's the tool Intel just dropped (and has probably optimized their own driver for of course). Classic Tom Peterson, lol, just like FCAT.
Of course new games are running at 70 fps. Most games are absolutely not designed for 500hz, and have zero reason to do that. Cs:go wan’t designed for 500fps, I bet the designers of cs:go never intended people to play at 500fps, or even imagined that would ever happen; it wasn’t possible when the game came out.
There’s a good reason to never go above 70fps: human perception experts tend to agree that our visual system gives us quickly diminishing returns above 60fps, we can’t really see things any faster than that, so I’m quite skeptical that 500fps is necessary.
The whole reason 500fps is useful for competitive esports gaming is to reduce latency, not really to keep increasing frame rates forever. Because games are triple-buffering and sometimes monitors are too, and there’s another frame of latency for controller inputs to be recognized, you might still have 10ms or more of latency between controller input and changes on-screen, even if your game is rendering at 500hz. This means you could get away with 100fps if you had zero latency. When your fps is 60hz and there’s 5 frames of latency, you don’t see responses to your controller until almost 100ms later(!).
It’s been about a decade since I was a game dev, but it’s wild to me that anyone would want 500 fps, or that any games would aim for that. This gives you a grand total of 2 milliseconds do to everything, gameplay + animation + physics + audio + rendering. It’s not a lot of time, and you have to compromise your visuals and rendering (by 10x!) in order to achieve that frame rate. Looks like a lot of new gaming monitors are 144hz, and these new 240/360/480hz monitors are pretty extreme and still a bit rare.
> I understand that some games purposefully limit FPS for physics calculations, but there are many that do not.
FWIW, this isn’t the way I’d frame it, it’s kinda misleading. Very few games are purposefully limiting FPS for the sake of slowing it down. All games, however, have a budget. There are always limited compute & render resources, and both game devs and players want the highest quality available for their budget. Until very recently most games aimed for 30fps, and only really fast twitchy games went for a relatively very smooth 60fps. The games that went for 60 had to cut their polygon counts and physics and gameplay in half in order to achieve high frame rate, so you are totally trading away a richer experience in favor of high frame rates. Only certain kinds of games should even try to do that.
The whole "science says the human eye cannot see beyond 30fps :)" is actually a very old meme at this point. You are correct that high FPS/Hz is about decreasing latency, but you underestimate the importance of it. 144hz is a very clear improvement over 60hz and the PC ecosystem is going to move over to it as a new standard relatively soon. You don't need to be playing a hyper-competitive FPS to notice it either, any game with significant movement will do.
> Because games are triple-buffering and sometimes monitors are too, and there’s another frame of latency for controller inputs to be recognized, you might still have 10ms or more of latency between controller input and changes on-screen, even if your game is rendering at 500hz. This means you could get away with 100fps if you had zero latency. When your fps is 60hz and there’s 5 frames of latency, you don’t see responses to your controller until almost 100ms later(!).
You're assuming VSync or something. There's no guarantee that the input will line up with the next frame, so an input latency of 10ms while running at 100 fps equals a worst-case latency of roughly 20ms, not 10ms. That's why the higher fps & hz numbers really matter.
> an input latency of 10ms while running at 100fps equals a worst-case latency of roughly 20ms, not 10ms. That’s why the higher fps & hz numbers really matter.
Yes exactly, I agree and this was the point I was trying and I guess failing to make. Latency is typically multiple frames, so at 100hz, latency can easily be longer than known science-meme perception times. Even with vsync there’s up to one frame to recognize inputs, then another frame to produce new game state & submit the render, then 1 or 2 more for double or triple buffering, then maybe supersampling and/or denoising, then whatever the monitor does, and I might be missing some steps that add latency. So I suspect 500hz has nothing to do with seeing smoother motion compared to 144hz, and everything to do with getting overall control latency down to below, say, 10ms.
Does that really work though? Does CS:GO and do other modern games poll the controller at the display refresh rate? I know it’s pretty common for a lot of games to decouple rendering from game state. That can mean a lot of different things, but one of the implications might be that controller latency is limited to one or two frames of, say, 60hz game state updates, followed by 3-5 frames of 500hz display refresh. Is latency in today’s PC games limited by the game state update, regardless of what the display refresh rate is?
I’m curious where the perceptual limits really are, and what the max framerate we actually need is. Suppose an imaginary world where worst case input-to-screen latency is 2 frames. Then in that case, how high should the FPS be? What if there was zero latency - like if the next rendered frame magically reflected mid-frame controller inputs and magically displayed with no latency - then what should the ideal FPS be? Would there be any benefit to going higher than 144, or would we be wasting electrons?
VSync introduces latency, so it always gets disabled, same with double/triple buffering. I don't think gaming monitors do anything to process the image, they're usually advertised with ~1ms response times.
> Does CS:GO and do other modern games poll the controller at the display refresh rate?
Modern gaming mice and possibly keyboards use 1000hz polling.
> I know it’s pretty common for a lot of games to decouple rendering from game state.
I think all modern games do. Some old games have issues running at fps other than 60 due to this coupling.
> I’m curious where the perceptual limits really are, and what the max framerate we actually need is.
I don't know exactly; there are definitely diminishing returns, like you've mentioned:
- 30fps: 33ms / frame
- 60fps: 17ms / frame (2x cost for -16 ms)
- 144fps: 7ms / frame (2.4x cost for -10 ms)
- 240fps: 4ms / frame (1.6x cost for -3 ms)
- 500fps: 2ms / frame (2x cost for -2 ms)
- 1000fps: 1ms / frame (2x cost for -1 ms)
Just from looking at those numbers, 144fps is still a clear win and 240fps probably makes sense too, if you've optimized your setup for latency. Everything beyond is probably pointless, unless it's your job to play competitive FPS games ;)
Graphical fidelity in games improves until it maxes out current hardware. This is natural. You can always turn the settings down to get higher FPS, but you can't turn down the high FPS you get in older games into more modern graphics.
> Avg FPS are not going up fast enough in my opinion.
FPS isn't the one and only metric. Audiences generally care more about graphical fidelity. FPS just has to be good enough and stable. Typically that sweet spot is 60fps, though we're slowly moving to 144fps being the standard.
Even if I had hit received some great stock award in all likelihood I would have sold it for index funds at my first opportunity. I imagine some of my old coworkers made out quite well though.
Midway through, our intern friend group found out one of the smaller buildings had a buffet lunch and started taking the shuttle there often.
Saw this tweet just now and that lanyard holder really brought back memories, hasn’t changed at all: https://x.com/jimcramer/status/1694465908234699243?s=46&t=NA...
Which is still better than a 10% layoff, anyway!
EDIT: It was just named Salad. https://salad.com/ https://salad.com/download
https://aws.amazon.com/blogs/aws/new-amazon-ec2-p5-instances...
3.2 terabits.
Or their dominance leads to competition throwing in the towel and investing resources in a market with less stiff competition.
I wouldn't be surprised to see AMD start to pair back ivnestment on high-end GPUs if things continue down this path. I would say Intel likely keeps pushing, but I'm less convinced they can actually make much headway in the near future.
It seems that Intel is making great headway on their fabs and may somehow pull off 5 nodes in 4 years. Intel 3 is entering high volume production soon and according to Gelsinger 20A is 6 months ahead of schedule and planned for H2 2024.
If they do pull this off and regain leadership that would change outlook.
Every year just sounds like “Nvidia’s new consumer GPUs are adding new features, breaking previous performance ceilings, running games at huge resolutions and framerates. Their datacenter cards are completely sold out because they can spin straw into gold, and Nvidia continues to develop new AI and graphics techniques built on their proprietary CUDA framework (that no one else can implement). Meanwhile AMD has finally sorted out raytracing, and their consumer GPUs are… well not as good as Nvidia’s but they’re a better value if you’re looking for a competitor to one of Nvidia’s 60 or 70 line GPUs!”
I'm unsure why you're criticizing the Efficient Markets Hypothesis or even using it here, but you need to also analyze this with some time horizon because the market and marketplaces are not static.
https://www.investopedia.com/terms/e/efficientmarkethypothes...
> The efficient market hypothesis (EMH), alternatively known as the efficient market theory, is a hypothesis that states that share prices reflect all information and consistent alpha generation is impossible.
Best I've got is "central planning". One firm being able to handily out perform others despite them also being both motivated and well capitalised lends itself pretty heavily towards markets being good, but I hardly think they were referring to "central planning" when they wrote "efficient market hypothesis".
If it's so obvious to you that you're dropping ellipsis, care to clue us in?
If there was any, the prices would go down as we have seen a billion times.
This is a textbook situation that would be perfect for a competitor to come in and undercut. However not only is that not happening, nobody is even trying.
Making the “theory” pretty worthless if it’s not even applicable in cases that would naturally produce this market entrant.
The reality is that private equity does not actually want to compete with large global brands.
Is it? A competitor can enter the market and undercut by producing a cheaper and otherwise undifferentiated commodity-type product. Nvidia's focus is adding moats that prevent competing on pure specs such as CUDA, design, and so on.
Irrespectively though I don't see how a competitor can replicate the context and tacit knowledge associated building something like CUDA for close to 20 years without putting in a similar amount of time.
Nvidia also made sure that CUDA runs on gaming GPUs and supports Windows. This is why its tools are so good. You don't need to buy a datacenter GPU, no need to mess with Linux. Just buy any gaming GPU, install CUDA SDK and you're good to go.
AMD wasn't like that. Their alternative - ROCM didn't even work on gaming GPUs. Their datacenter GPUs didn't even support Windows. Basically the opposite approach of NVIDIA. Now AMD is rushing to add Windows and consumer GPU support to ROCM, but it's a bit too little too late.
They already are undercutting.
Some business that are really into hyper-scaling are already pouring man-hours into making those "undercut" alternative products work. Specifically because sometimes you can't affort Nvidia at that scale. Or they have a large enough scale that they can make it work with suboptimal tech.
It's just that for most business the man-hours required to make "undercut" products work still aren't cheap enough to win in cost-benefit.
A lot of companies are trying. It's just that nvidia is really good and significantly ahead.
Do you have any idea how much R&D it takes to make a GPU, let alone something that could possibly compete with NVIDIA?
This is not something a dozen people in a large garage are going to do.
> However not only is that not happening, nobody is even trying.
AMD has been trying to compete for years. If you're a gamer that doesn't care about getting the top performance and just wants to optimize price/performance, their offerings aren't bad.
> The reality is that private equity does not actually want to compete with large global brands.
Because the barrier of entry into the GPU market is probably at least a couple billion dollars.
There are $BN Private Equity deals done everyday, there's plenty of money to fund a competitor.
No, I think people are just scared and don't care about the existence of monopolies.
You mean a theory that no economic school actually believes? Not even Austrians would sign that.
I think the principle is called perfect market or perfect competition, but it's only a theoretical concept. However, it's certainly possible that the market is less perfect than it could be due to interventions and regulations.
People talk about AMD being competition - but from most stats I've seen, they're ~10% of dGPU sales, with Nvidia being the other 90% (with new Intel offerings being pretty much noise now). That means that if they invest the same proportion into development, NVidia nearly have 10x the resources.
It may be that tech companies like this would "naturally" form a monopoly without outside (IE government) interference, as the only reason that multiplier of development resources doesn't completely crush new entrants is rather extreme mismanagement, or a new segment is created where the design resource don't really cross over that much.
I don't see anything like that happening in the short term, if anything there seems to be more opportunity for cross pollination of development within these corporations, as there's a fair bit of design similarities between various silicon (GPUs, CPUs, accelerators for the current ML techniques etc.) that may encourage more consolidation in the whole semi market to take advantage of that, not less. But again the only thing stepping in the way of that seems to be governments trying to keep national interests, like the blocking of NVidia buying ARM to pull in one of the big CPU players. Plus all their other IP that they may benefit from, like low power GPU designs or other accelerators ARM have designed.
This means that nvidia capital is spend on testing/development infrastructure and creative labor.
So, you don’t need monopoly breaking to handle nvidia if it keeps growing, but instead rethink IP laws.
Testing infrastructure is less capital investment than production manufacture. (See ASML and TSMC beeing booked), and humans can be persuaded to work elsewhere.
This means Nvidia cannot fall a sleep, even if they keep snowballing. In a few years a rival can always arrive, or current ones snag key people or have dev/test infra breakthrough, or IP law could change as its critics rise every year.
Sure, if Nvidia keeps its good game it will keep on growing and get even bigger share, but if it doesn’t, it will happen what happened with intel, intel got greedy on its position, and the company got fat.
As long as nvidia keeps its game, it’ll be good for customers even if they swallow more market share, as prices are always limited by the value business customers get from AI, and the investment needed for a competitor is not even close to infrastructure or resource extraction stuff like high end chip fabs, oil extraction, energy grids, telecom grids.
Again, Nvidia does not produce any physical goods, only designs.
With Nvidia and other GPU competitors being IP-focused, effectively outsourcing all this "manufacturing stuff" (to the same 3rd party much of the time), that's one less thing for them to keep up with, and one less think that'll hurt if they do start "falling asleep". I can't see this happening to Nvidia in quite the same way right now. My point was that not having manufacturing makes advantages of consolidation larger, not smaller.
I wonder what would have happened if Intel realized it's manufacturing wasn't hitting targets and "quickly" added TSMC as an option, would AMD even have had a chance with ryzen? There was clearly a time when AMD had superior manufacturing processes through them, if Intel's designs of the time were on the same process would they have managed to grab the headlines?
And no, I Strongly disagree that NVidia running unchecked over the entire market being "Good for consumers", and not sure if the capital expenditure of getting over this moat is really much smaller than things like resource acquisition or infrastructure, they have $billions in current software ecosystems and hardware designs. Those $billions probably could buy you a fair bit of infrastructure investment on the scale you mentioned. Look how much Intel is burning right now just to get a toe into the market and not laughed out the door - and they're still clearly behind their competitors right now. Their chips aren't anywhere near competitive from a performance-per-area point of view, and their software is rather poor for the vast majority of use cases.
AMD compute is nowhere compared to NVIDIA. NVIDIA wanted to buy ARM, has got its finger in RISC-V, but apart from that, they don't really care. To be fair AMD has done decent with GPUs, but never enough to dethrone NVIDIA, whose playbook for the past few gens is "just make everything bigger than last gen and increase the frequency." Surely AMD could have chosen the same lazy approach to surpass the 4090 only just, but instead they didn't, so it's still NV undefeated in its space because AMD forgot to squeeze the last 1% out of their card.
The market is powerless if the competitors aren't really competing. Intel is the only chance, unless they manage to get their own Taiwanese CEO somehow related to Huang and Su.
There are in fact startups, also doing what they can (and probably not trying to go head on against the most productive competitor they can find.) And it has been reported countless times that some of the biggest customers of Nvidia are actually trying to design their own.
If you want to point out a market with broken competition, this isn't it.
The situation is created by artificial restrictions on free market (namely state enforced monopolies on "IPR", or as some call it, imaginary property).
Yes, there has been repeated efforts to chip at Nvidia's market share, but there's also a graveyard full of AI accelerator companies that fail to find product market fit due to lack of software toolchain support - and that applies even for older Nvidia GPUs and their compatible toolchains, let alone other players like AMD. This isn't a hit on Nvidia, I'm just saying things move so quickly in the space that even the only-game-in-town is trying to catch up.
Nvidia is also leading by being one or two hardware cycles ahead of their competition. I'm pretty confident AI workloads in enterprise is their next major focus [1]. I think this more than anything else will accelerate AI adoption in enterprise if well executed.
To your point, I think the industry needs to focus more on the toolchains that sit right between the deep learning frameworks (PyTorch, Tensorflow etc.) and hardware vendors (Nvidia, AMD, Intel, ARM, Google TPU etc.) Deep learning compilers will dictate if we allow all AI workloads run on just Nvidia or several other chips.
[1] - https://www.nvidia.com/en-us/data-center/solutions/confident...
"The Render Network® Provides Near Unlimited Decentralized GPU Computing Power For Next Generation 3D Content Creation."
"Render Network's system can be broken down into 2 main roles: Creators and Node Operators. Here's a handy guide to figure out where you might fit in on the Render Network:
Maybe you're a hardware enthusiast with GPUs to spare, or maybe you're a cryptocurrency guru with a passing interest in VFX. If you've got GPUs that are sitting idle at any time, you're a potential Node Operator who can use that GPU downtime to earn RNDR."
The main reason why you need massive ammounts of fast VRAM in the first place is that the main limitation of AI is memory bandwidth. Can't simply distribute an algorithm that is already throughput limited by memory bandwidth and distribute it with awful latency and bandwidth and hope for any improvement.
Good luck with that.
In the current state of things, Nvidia is like a car manufacturer that exclusively owns the concept of tires.
So nice to see a “hard” engineering (from silicon to software) SV-founded company getting all this recognition. Especially after what has felt like a decade of SV hype software companies dominating the mainstream financial markets pre-pandemic with a spate of overpriced IPOs or large ad-revenue generating mega corporations.
AMD GPUs get comparable results as of late on Stable Diffusion.
Software and hardware from competitors will catch up, crunching 4/8/16 bit width numbers is no rocket science.
I made the mistake of buying an A770 from Intel, based on the spec sheet. Hardware is comparable to what Nvidia is selling, for 70% of the price.
It's basically a useless paperweight. The AI software crashes constantly, and when it's not crashing, it performs at half the level of Nvidia's cards.
Turns out that drivers and software compatibility are a big deal, and Intel is way way behind in that arena.
I hope but do really foresee that AI/ML systems will have similar competitive stack like Nvidia.
I used to think like that, until I got a job there and... Oh, boy! I left five years later still amazed at all the ever more mind bending ways you can multiply two damn matrices. It was the most tedious yet also most intellectually challenging work I've ever done. My coworkers there were also the brightest group of engineers I've ever met.
What does this mean? Tedious is pretty much the opposite of “intellectually challenging” when I think of careers.
The recent Stable Diffusion results are great news, but also don't include comparisons to an Nvidia card using the same optimizations. Nvidia claims that Microsoft Olive doubles performance on their cards too, so it might be a bit of a wash: https://blogs.nvidia.com/blog/2023/05/23/microsoft-build-nvi...
Plus, none of those optimizations were any more open than CUDA (since it used DirectML).
> crunching 4/8/16 bit width numbers is no rocket science.
Of course not. That's why everyone did it: https://onnxruntime.ai/docs/execution-providers
The problem with that "15 competing standards" XKCD is that normally one big proprietary standard wins. Nvidia has the history, the stability, the multi-OS and multi-arch support. The industry can definitely overturn it, but they have to work together to obsolete it.
I bought a RX 6800XT to do some AI work because of the 16GB VRAM, and while the VRAM allows me to do stuff that my 6GB RTX 2060 wasn't able to, on performance side it's actually a downgrade in many aspects.
But the main issue is software support. To get acceptable performance you need to use ROCm, which is Linux only. There was some Windows release of ROCm few weeks ago, but I am not sure how usable it is and none of the libraries have picked up on it yet.
Even with a Linux installed, most frameworks still assume CUDA and it's an effort to get them to use ROCm. For some tools all it takes is uninstalling PyTorch or Tensorflow and installing a special ROCm enabled version of those libraries. Sometimes it will be enough, sometimes it wasn't. Sometimes the project uses some auxiliary library like bitsandbytes which doesn't have an official ROCm fork, so you have to use unofficial ones (that you have to compile manually and Makefiles quickly get out of date). Which once again, may work or may not.
I have things set up for stable diffusion and text generation (oobabooga), and things mostly work, but sometimes they still don't. For example I can train stable diffusion embeddings and dreambooth checkpoints, but for some reason it crashes when I attempt to train a LORA. And I don't have enough expertise to debug it myself.
For things like video encoding most tools also assume CUDA will be present so you're stuck with CPU encoding which takes forever. If you're lucky, some tools may have a DirectML backend, which kinda works under Windows for AMD, but it's performance is usually far behind a ROCm implementation.
However AMD is still as terrible at H264 as ever, and their AV1 encoder also has a hardware defect they’ve patched by forcing it to round up to 16 line multiples, so it is incapable of encoding a 1080p video and instead outputs 1082p that won’t be ingested properly when streaming.
http://freedesktop.org/mesa/mesa/-/issues/9185#note_1954937
Also the AV1 quality is not as good as intel+nvidia even with resolutions that aren’t glitched. AMD seemingly went big on HEVC (supposedly because of stadia?) but everything else is a mess.
And most places won’t touch HEVC because of the licensing costs. Microsoft makes it a windows store plugin you have to buy separately etc. somewhat odd that google picked HEVC for stadia but I guess those customers are actually directly paying you vs YouTube being a minus on their balance sheet (at least until recently possibly)
I’m really speaking about Nvidia’s ability to perform well in both hardware and software, at chip-scale and datacenter-scale. Also speaking of their product/business direction that revolutionizes multiple industries (leaders in graphics with ray tracing and AI frame/resolution sacking; leaders in AI infra and datacenter systems, etc.) all resulting in big impacts to their respective industries.
You’re right that many of those software-only companies do very real engineering with distributed systems and such. I should’ve been more precise and was really complaining about the SV hype of the 2010s focusing on regulating-breaking companies like Airbnb, Uber, wework, etc. and on companies like Meta and Google who focus on pushing ads for their revenue.
These companies exist today. Which small or ignored companies do you think have a bright future?
Record revenues, and a dividend of $0.04 on a $450 stock? That's not even worth the paperwork. For example, if you bought 100 shares, that's $45K. From that, around September $4 will show up in your account, which you have to pay taxes on. So $3 or so net on a $45,000 investment. Sure, there were stock buybacks, but why keep the token dividend around?
For large shareholders, the dividend would still be worthwhile. From what I could find, Jensen has 1.3 million shares, so he'd receive over $200k in dividends this year. You might think that's chump change, but another source lists his salary at just under $1m; another 20% bump in liquid income is nothing to sneeze at.
especially when it is awarded for merely having a stack of papers.
Why?
> From what I could find, Jensen has 1.3 million shares, so he'd receive over $200k in dividends this year. You might think that's chump change, but another source lists his salary at just under $1m; another 20% bump in liquid income is nothing to sneeze at.
Jensen Huang is worth $42 billion and has been a billionaire for probably a decade or so now? Any CEO with that net worth would use stock-secured loans/LOCs for liquidity. 200k is very much chump change.
I'll get right on that...after I go look up what that means. :-) I'm but a simple options trader who sells calls to unload stock I didn't want anymore anyway, and the premium is the icing on that cake. Left some money on the table this time, but I otherwise would have just sold the shares outright, and I did make some bank regardless.
Gonna be missing that sweet, sweet $0.04 dividend, though.
- a GME ape :)
To remind us all, they're selling capitalized assets, not contracts or services.
Is the marginal demand for GPU chips over the next 3 years enough to sustain (or grow?) current revenues and keep this valuation afloat? To me, it feels like a comparatively fragile situation to find themselves in, to convince the world of 2025 that they need even more chips, unless "everybody needs to train their own LLM" is a secular trend.
I'm not sure if investors fully appreciate the nuances of this boom, or if I'm not fully appreciating how many GPUs "need" to be held by different companies (and sovereign entities, if you've read the headlines in the last couple weeks) to train LLMs in the coming decade.
I admit I'm biased in favor of Tesla however.
AI is not showing any sign of slowing down so far.
Once the tech settles down a bit, Google and Amazon and others can absolutely snipe the revenue at a lower cost, just like they did with the previous TPUs/gravitons. But then some new innovation comes out that the ASICs (or, ARM+accelerator) don't do, and everyone's back to using NVIDIA because it just works.
AMD potentially has a swiss-army knife too, but, they also have a crap software stack that segfaults just running demos in the supported OS/ROCm configurations, and a runtime with a lot of paper features and feature gaps. And NVIDIA's just works and has a massive ecosystem of libraries and tools available. And moreover they just have a mindshare advantage. Innovation happens on NVIDIA's platform (because NVIDIA spent billions of dollars building the ecosystem to make sure it happens on their platform). And it actually does just work and has a massive codebase etc. Sure it's a cage but it's got golden bars and room service.
https://github.com/RadeonOpenCompute/ROCm/issues/2198
So I guess I'd say it's sticky until the technology settles. Steady-state, I think competitors will capture a lot of that revenue. But during the periods of innovation everyone flocks back to NVIDIA. AMD could maybe break that trend but they'll have to actually do the work first, they have tried the "do nothing and let the community write it" strategy for the last 15 years and it hasn't worked. You gotta get the community to the starting line, at least. Writing a software ecosystem is one thing, writing runtime/drivers is another.
P/E still above 50 even after the AI craze 9x'd eps this quarter. Still hard for me to see that valuation ever makes sense but what do i know.
No.
Lol... Intel is famously stingy when it comes to salaries.
Probably Intel's best bet now would be to try to be the fab for NVDA.
Principals usually broke into the three digit range from what I hear.
For different people, the timespan between now and when they may want or need to sell it is different, and thus different people will arrive at different conclusions.
And note that “worth more” above simply means growth in market capitalization, so as long as someone is willing to buy the shares at a price supporting that increased market cap, then it does not matter if Tesla is still selling fewer cars than the others combined.
And as somebody with a significant short position, I would add - "The market can stay irrational longer than you can stay solvent" !
Their numbers for this and next Q are absolutely amazing. It's also quite "refreshing" - a company with great product, almost without competition (so far - it will come real quick). And fun part being their main advantage is probably CUDA and not even the chips itself (which by the way they don't manufacture - they "only" do the design).
But still - even with those numbers, and even with this pace of growth (both being absolutely not sustainable, and will probably reverse hard next year) - the valuation doesn't make any sense, especially given the current interest rates.
As time goes on I don’t see how you break the CUDA moat even if you had all of nvidia Al’s engineers.
CUDA means you need everyone in AI to target your new (hopefully open) platform and that platform is faster than CUDA is. Given how most frameworks of the last 10 years have been optimized for CUDA you would need to turn around a global sized cruise ship.
If Intel’s GPUs are only 3% faster, will that be enough to rewrite my entire software stack for something not CUDA? If intel opts for a translation layer, could they ever match nvidia’s performance?
If Intel is 10% slower but 50% cheaper and the open source stack you use has been heavily updated to work well with Intel drivers would that not be an enticing product?
Yes, and in fact both AMD and Intel have libraries. You can run Stable Diffusion and suchlike on AMD GPUs today, apparently. And you can export models from most ML frameworks to run in the browser, on phones and suchlike.
> If Intel is 10% slower but 50% cheaper [...] would that not be an enticing product?
Sometimes, yes. Some of the largest models apparently cost $600,000 in compute time to train [1], so halving that would be pretty appealing.
However, part of the reason for nvidia's dominance is that if you're hiring an ML engineer for $160,000/year spending $1,600 to give them an RTX 4090 is chump change.
[1] https://twitter.com/emostaque/status/1563870674111832066
I worked at Sears corporate when Amazon was getting big, about 25 years ago.
Always made me chuckle when armchair quarterbacks on TV would wonder why Sears couldn't do what Amazon did.
Bezos took tremendous risks in the late 90s and early 00s, while Sears was trying to figure out how to wring a few more pennies out of their stores. Sears Corporate was 110% focused on taking the existing business and maximizing profits, not on innovation of any kind whatsoever.
Raptor Lake is at 7nm and incredibly competitive there (~2700 single core on geekbench, taken with a pinch of salt). They’re still planning on being on 1.8nm/18A within 2 years, while at the same time ramping up their GPU efforts (albeit using TSMC for 4nm). Nvidia is very much in the lead, but this is just the beginning.
tldr; I ain’t hear no bell.
1. They don't pay - Nvidia/Google/Apple easily pays 1.5-2x Intel before appreciattion.
2. They're cheap/beaurcratic. The office sucks, your laptop sucks.
3. They suck at software. https://pharr.org/matt/blog/2018/04/18/ispc-origins
4. They can't develop/retain talent. Half the ML-HW/FW teams at AMD/Google/Nvidia/Apple are ex-Intel.
(Retention notwithstanding :P)
So, if intel 7 is a 10nm class node then so is tsmc N7. But the whole industry cooked their node names a decade ago and intel is allowed to do the same terminology. To do anything otherwise would be a needless marketing disadvantage. If someone’s cheating on marketing, everyone has to - and the reality was everyone but intel was already cheating.
i.e. This would mean Intel will be getting a four-node-scale jump in performance (10nm->7nm->5nm->3nm->1.8nm) from where they are with Raptor Lake by 2025 if they can stay on track.
I don't have any insight into this but I would hope Pat Geslinger is righting the salary, perks and incentive structure as a matter of priority to stop the shedding of talent.
Absolutely not and this is one of the strongest headwinds intel faces. They just did a big round of layoffs and then everyone who didn’t get cut got a surprise 20% pay cut after, and retention bonuses for seniors was typically low-3-digits to high-2-digits. Principle engineers were solidly 3 digits though. And then they did stock buybacks that exceeded the size of the layoffs and pay cuts.
Intel is fucking broke and they’re gonna have to cut every cost they can to focus on their key markets, and do it with personnel that were already underpaid and have better opportunities elsewhere. Not that labor is the straw that’s breaking intel’s back here but the situation is really AMD-in-2012 level dire, they have an incredibly long and expensive road to return to profitability and they will have to do it with whatever they can scrape up out of a staff that was already abused even in the “good times”.
Oh and they want you to move from desirable west-coast locations (well, except Arizona) to Ohio. And they’re massively behind on their tech stack (fighting uphill is not what everyone wants) in a failing org with a ton of middle-management rot and neglect. And they’re still failing to execute a huge amount of the time, causing massive delays, etc.
It’s really really dire and just because intel has staunched the worst of the financial bleeding doesn’t mean there’s much light at the end of the tunnel. Like Boeing I don’t think they will be allowed to go under, they’re strategically too important, but they’re gonna have a bad time for a while and probably legitimately do need a bunch of subsidy and “pity contracts” for national labs HPC computers to stay on their feet - just like AMD did.
(oh and to be clear by "2 to 3 digits" I don't mean 10-999% bonus... I mean they gave senior engineers a $75-125 retention bonus and principals might get 2 or 3 hundred. Those are the people who will build Intel's next foundational architectures and the nodes and the packaging technologies that lead themselves and their foundry customers to success. Have fun with that.)
If they can hold the market and not implode for internal reasons, not sure this has any upper limit?
Anyone who is a lot more versed in company valuation methodology see this as being near peak value, or does Nvidia have a lot more room to run?
Crypto and blockchain never had an actual proved out use case. There was an interesting idea but no one ever could figure out a way it was useful. The costs associated were much higher than the risks of not using it.
People who think this is a meme aren’t paying attention, and they’re certainly not in the rooms of power where AI planning is happening at megacorps. I’ve been in them, and it’s serious and material and we are just now beginning to scratch the surface.
How we prove it's one is probably another matter.
I already saw a few posts here on HN from companies that threw down insane amounts of $$ on H100s and are now looking to rent out their excess capacity. I'm guessing we'll be seeing a lot more posts like that soon.
But that's still a high valuation - that is even if that new market grows to the sky, it's not clear that it can justify that new valuation.
Is Nvidia failing anytime soon? No. Is it the best investment you can find? That's harder to tell which is why the complaint of "very high valuation already". It's not in doubt that it's a great business. It's less easy to decide whether it's a great investment to get in right now.
But everything is relative a PE around 40-60 is NOT historically crazy high for the verge of a giant new market. And yet it is very high for a trillion dollar market cap. This is exciting: a trillion dollar market cap at the verge of a giant new market!
8% is close to nothing in stocks. Biotech stocks go up and down more than that without earnings announcements.
> Anyone who is a lot more versed in company valuation methodology see this as being near peak value, or does Nvidia have a lot more room to run?
As long as fine-tuning, training or even using these models are inefficient and no other efficient alternatives to that without these GPUs, then Nvidia will remain unchallenged unless that changes.
EDIT: It is true like it or not AI bros. There are too many to list. For example, just yesterday:
Fulcrum Therapeutics, Inc. (FULC) 38% up.
China SXT Pharmaceuticals (CM:SXTC) down 25%.
Regencell Bioscience Holdings (RGC) 28% up.
NanoViricides (NNVC) up 20%.
Armata Pharmaceuticals (ARMP) down 23%.
Nvidia is just one of many lottery tickets and 8% in one day is hardly volatile in stocks.
Also I would add, volatility measures continued volatility over periods of time, not single days. Most pharma stocks are essentially static for months or years until they announce one good result and the stock jumps 30% in a day. This doesnt mean they are volatile stocks, compare Gileads IV to Tesla for example.
Exactly, using the IV for supporting their highly risky 0DTE option trading strategies and gambling on earnings rather than investing long term?
> Most pharma stocks are essentially static for months or years until they announce one good result and the stock jumps 30% in a day.
That is the point on long term 'investing'.
A much lower risk and patient strategy to invest and buy such low priced stocks at the bottom for the long term and take profit after the price jump rather than gambling on ridiculous short term option trades or the retail favourite of 0DTE trading strategies which that is just plain gambling.
> Not if you understand what stocks are and how betting on biotech stocks is not a wise investment.
So you're giving investment advice for putting money in NVDA stock at the top or all time highs, right now on earnings as a 'wise investment' to make 8% (after hours) when others are clearly taking their money out of the market.
Unless you already invested in NVDA stock last year, that move is gone and you're just telling retail late comers to throw money at NVDA at the top for others to take their profits.
After warning not to buy stocks like NVIDIA at the top when other investors with larger capital are taking their money off the table?
An 8% move in reaction to earnings is close to nothing to what I've seen as 'volatile' compared to the stocks shown in my original comment, which were up / down > 20% in ONE day without earnings. 8% is close to nothing for retail investors like almost everyone here and likely yourself.
Also I'm not the one giving advice over looking at the IV or telling retail investors to jump into Nvidia stock at the top as a 'wise investment' or to use 0DTE option gambling strategies, a popular favourite with lots of retail traders.
Now it is 10% down after the smart money started taking their profits off the top as I warned days ago whilst many retail investors who bought in at the top are now left stuck in the market.
Please.
Stocks are lottery tickets and Biotech stocks are stocks.
> You're just gambling on binary results.
The risks are no better than most of the AI bros buying Nvidia and overpriced stocks at the very top or all time highs or extremely risky 0DTE strategy trades on earnings announcements.
Do AI bros who jumped in late really have to be married to their stocks that are already overpriced to make 8% on earnings when the very early folks start selling to take their profits?
If you want the responsible advice, it's overpriced. If you want my personal advice, well I bought more yesterday afternoon.
Its not overpriced anymore. In fact if there's anything left in the tank or this is at all sustained it's cheap.
Now it's just crazy.
Nvidia has had a watershed moment of revenue growth, because they're the only significant player in the space of top-of-the-line GPUs for training LLMs.
Their current valuation bakes in the assumption that not only is this unprecedented level of pricing power durable, but that top line revenue will also grow significantly over the next few years.
Reminder that Nvidia is selling chips, mostly to datacenters, whose purchasing habits are primarily driven by "customer" demand (where customers are, in this case, tech companies wanting to train neural nets).
So a bet that they go up from here is a bet that datacenters will want at least as many chips in the next N continuous quarters, and will be willing to pay the current premium that Nvidia is charging, today. The corollary is you're betting that "customer demand" for ever-improving GPUs is climbing an incredibly upwards, secular (read: permanent, non-cyclical) trajectory, such that it outpaces the assumptions of these already-lofty expectations.
I think betray my own opinion, but the price is what it is. :)
To make a flawed analogy, but if people want oil (AI), NVDA sells drills (means of AI production). Demand for each is obviously correlated, but the profile of demand for the two products is really different.
You d think that, at the level of capitalization of tech companies, competition would be cutthroat
What really counts is the profit. It is pretty huge now, but not 'that' huge (at least yet).
Your point stands but in both directions: The market cap is not "the value of the company" as in, you can neither buy it or sell it at that price.
There's also the problem of industry hostility, anyways. Even if Nvidia was dethroned in the hardware-space, it's unlikely their successor would improve the lock-in situation. It will take an intersectional effort to change things.
People are having a really hard time grasping that sometimes established companies 10x.
I no longer buy nvidia hardware but I do enjoy stock price getting higher. I just wish I had the sense to buy more, a lot more, stock when it was much cheaper. How does a chicken shit like me make big money :(
That’s a product designed and manufactured by AMD, eight?
In what way has Nvidia “forgotten” gamers with the rise of their datacenter business?
It used to be that $200 would get you a low-range GPU, $400 a midrange, and $600 a pretty darned good one. The 1070 launched at $379 (or $480 in 2023 dollars), the 2070 at $499, the 3070 at $499, then all of a sudden the 4070 now is $599. That's a big difference, even accounting for inflation.
The 1080 Ti was $699, an already-unfathomable price back then that cost more than a whole console setup. Today? The 4080 is $1200. That's often more than the rest of the system put together.
On the other hand now that it’s stopped, so has raw performance scaling. There was a few generations of cleanup, but you can’t squeeze blood from a stone forever. Maxwell arguably cut too far and pascal had to start adding functionality back. Gpus are like e-cores, large powerful cores mean you get fewer of them and that often works out to a lower PPA. There aren’t many opportunities for cool tricks and the model doesn’t favor using lots of area. The coding is written for extreme parallelism already, so, that’s not a problem, it’s inherent to the platform.
Transistor per $ growth hasn’t completely stopped but it’s certainly nowhere near what it was 10 years ago, even if you factor in things like packaging/stacking the total wafer area still runs up a big bill that offset most of the density gains. And that’s what we’ve been seeing over the last 10 years in gpus too. 4070 is about the same area and cutdown as GTX 1070 but wafers cost like 8x as much.
What you have to do in this operating regime is find ways to get more performance per transistor - and that’s exactly what Jensen made a big bet on 5 years ago with dlss. 7% more transistors that with DLSS 2.5 give 30% speedup at native quality and 50-70% speedup at iso-quality with FSR2 Quality mode.
That’s what the future looks like - rewriting your applications to take advantage of new accelerators that provide large speedups. And the rewriting is very minimal for any application that uses TAAU already. It sucks but if cost/transistor is not going to come down you have to get more out of the transistors you have. Work smarter not harder.
Yeah the upscaling is nice, but I dunno about $1200 nice...
It was cool when they instead focused on things like creating mobile smaller versions of the cards with laptop level power draws, or better cooling systems that weren't as noisy etc. While staying within reason price points. Just the mundane stuff, not necessarily real time raytracing (gimmicky and not very noticeable IMO, even as someone who uses 4080s on Geforce Now). Something like the Steam Deck package with its integrated console-like APU is much more consumer friendly, but also much much weaker than a 4090 I guess. Different priorities that Nvidia might've reconsidered if not for the AI supersampling stuff also bleeding into crypto and ML, justifying their bet.
Edit: Yeah their AI profits are skyrocketing, while gaming is a has-been. Good bet for the company, bad omens for gamers :/
https://www.pcgamer.com/nvidias-record-breaking-profits-are-...
Because moore's law wasn't just about transistor count but about the economic impact of exponential growth in transistors-per-$. In a world without moore's law, using more transistors will result in a higher-cost product. If you want to hold product cost fixed, or even contain the cost spiral, you need to do more with the same amount of transistors - performance-per-transistor is the metric that matters now.
AMD and NVIDIA have already stripped down their pure-raster implementation as far as they can go, with RDNA1 and Maxwell respectively. Maxwell actually cut too far (software scheduling, "minimal" DX12 support, etc) honestly. So where do you keep making perf/tr gains after that?
The gaming world has already pretty well settled on TAA (although some people will never accept it) and upscaling is already common in the console world. So, do TAA upscaling better such that you get the performance gains but not the reduction in visual quality that usually comes with it.
Tensor makes up a relatively small amount of die area (5.9% of total Turing die area, based on comparisons between Turing Major/RTX and Turing Minor/GTX SM engine die shots). And that gets you to about 30% faster than FSR2 for a given level of visual output quality. So the perf-per-transistor metric increases. Also, unlike a fixed-function accelerator, it can be used for all kinds of other stuff too. It's basically a whole programmable sub-processor, an accelerator for your accelerator.
https://www.reddit.com/r/hardware/comments/baajes/rtx_adds_1...
Now, why ML as opposed to just running it on shaders? Same logic as adding an AVX unit, math density is a lot higher and it can do a lot of work for applications that are specifically tailored to it. DLSS2 uses a relatively standard TAAU (similar to FSR2) but determines the weighting of the samples using a neural net. This produces a lot higher quality than a procedural algorithm currently can - especially under "bad conditions" like higher degrees of upscaling, low framerate/limited sample count, or temporally unstable/high-temporal-frequency areas of the image.
http://behindthepixels.io/assets/files/DLSS2.0.pdf
https://raw.githubusercontent.com/NVIDIA/DLSS/main/doc/DLSS_...
FSR2 does ok at 4K quality mode, but at 1440p and (especially) 1080p output resolutions and in performance modes it does much worse. FSR2 quality 1080p is more like DLSS2 performance mode or maybe balanced mode, so NVIDIA gets more speedup at a given level of visual quality. And DLAA can produce a better-than-native image when running with a native input quality.
The neural weighting just is a lot more efficient at using its samples, it understands what is going on in the scene (moving edges/occlusion etc) and can extract a higher signal-to-noise ratio from the samples and the textures. It's like an op-amp, the ratio of input:output pixels is the "gain factor", and FSR2 and other traditional TAAU algorithms are simply noisier at any given level of gain, whether that's unity or extreme gain, and have other edge-cases like turn-on threshold (bad performance with low samples). ML is the "schottky diode" of graphics amplification (dangerously mixed metaphor, lol), it's simply a lot more agile at shaping the signal than what came before.
(and while on paper plenty of people have argued that procedural programs should be able to do anything ML can, it's not like AMD and others haven't tried to improve TAAU with FSR2, and many others before them. DLSS2 is better, just like LLMs and Stable Diffusion are a lot better than procedural algorithms in their own niches.)
--
All of this exists completely orthogonally to actual wafer costs or packaging or other things. Packaging may boost that transistors/$ metric a little bit but that just gives you a little more to play with. It does allow you to make chips with twice the transistors at twice the cost and have them yield at high rates, but fundamentally 2x400mm2 is still 800mm2 of silicon even if you yield at 100% - you're using more wafer, which drives up costs. Wafer costs have been increasing nearly as fast as density (and predicted to match/pass at 3nm) but there has been a small gain in tr/$, certainly nowhere near the rate of moore's law days. But if wafers cost 8x what they did for 28nm, and are continuing to increase at ~50% per generation, and you keep using more wafer area to compensate for slowing shrinks, then costs will go up (even more than they have).
There is no direct link between "running deep learning" and costs going up. That is just happening independently and affects AMD too, even when they didn't go in on deep learning. TSMC prices keep going up (as do their margins, even now) and even if they went to zero, the design+validation costs are still going up too. A lot of these costs are driven by hard physics problems and not just TSMC profit margin (although it doesn't help).
--
> It was cool when they instead focused on things like creating mobile smaller versions of the cards with laptop level power draws, or better cooling systems that weren't as noisy etc.
I think a 30% performance boost at native visual quality, without power increase is pretty cool. Don't laptops benefit from having 30% higher perf/w just from turning on a setting? And Ada itself is a ~60% perf/w increase over previous generations too.
Like Ada is one of the most efficiency-focused generations ever, much moreso than Ampere or Turing with their trailing nodes. DLSS just stacks on top of this - and unlike FSR2, NVIDIA doesn't fall apart at 1080p resolutions that laptops tend to be using.
Cost is higher than people want, but on the other hand (a) that's going to be the reality unless there is a breakthrough in transistors-per-$, you can't make a fixed number of transistors infinitely fast, there is some asymptotic limit. And (b) people are cherrypicking favored examples or comparing against trailing-node products that had larger dies on slower, less energy efficient nodes to keep costs down.
GTX 970 at $329 was an outlier and the lowest x70 product of all time, on a trailing node (28nm again after 20nm fell through). GTX 670 launched at $399 for a similarly sized die over 10 years ago. GTX 1070 launched at $449 7 years ago with another similarly-sized (~300mm2) die. Turing, Ampere, and Maxwell were all abnormally cheap due and large due to the trailing-node but you paid for this with worse efficiency. There has never been a x70 product launched at $299 and you are welcome to check this!
First x70 product: https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_proces...
And yes I think the consensus is that nodes like 8nm probably are "good enough" especially if they are much cheaper, especially at the low end where PHY size is becoming a problem. PHYs don't shrink, so you can't scale a product arbitrarily small - the logic may shrink by 70% but those PHYs are just as big as ever. So there is a de-facto "minimum die size" that is ever worth producing, because there is a fixed PHY area that you simply cannot eliminate. And in a world where wafer costs are going up, that area costs more and more every generation.
A 3060 Ti 16GB wouldn't even need clamshell (it has 8 PHYs, 8x2GB per module=16GB) and could probably have hit $299 or $329 launch cost, if NVIDIA had gone down that road. And it's Good Enough for 1080p, and avoids some weird compromises that shake out of the need to trim PHY area.
AMD already did exactly this with the 7600 - which is held back on 6nm (N7 family) rather than using N5P (N5 family) like the rest of the RDNA2 lineup. Why? Cost.
GPUs will be bought like TVs. It took 4060, about 6 years to double the Passmark score over 1060.
BUT that comes at a price - Nvidia consumer chips are also notoriously expensive, but if you want best-of-breed for gaming, it does come at a price.
I am hoping that AMD and Intel will be able to compete with Nvidia someday but I'm not holding my breath.
The hoarding isn't going to stop unless there are either efficient alternatives that are competitive on performance and price.
Perhaps that is why I keep seeing gamers crying over GPU prices and unable to find cheap Nvidia cards due to the AI bros hoarding them for their 'deep learning' pet projects.
So they settle with AMD instead.
Supply issues should be gone too; I got a 4070 Ti shortly after launch, no problem.
Thus only gamers that care about RT and only in the games that make a good use of it (virtually none) have any serious benefit.
So raw performance has stagnated and gains have been concentrated in things like DLSS 2.5 that let you get native quality at a 30% speedup or better than native DLAA at 0% speedup, or FSR2 Quality level quality at 50-70% speedup. Cause that gets you more performance out of a small increase of transistors/cost.
My main gripe is that at 4k resolution, top of the line GPUs shouldn't be using AI frame scaling to get decent fps unless you are taking the raytracing penalty for funsies.
Seems like you’re really dismissing the massive speed ups these past few years. Agreed that ray tracing in games is only at the beginning. A lot of that is gated by the consoles/AMD but that’s generally how it goes. Would love to see Nvidia in one of the powerful consoles to accelerate adoption of these technologies.
My fear is that Nvidia, seeing record-high profits for their datacenter cards, will gradually phase out budget/midrange gaming cards in favor of the higher end stuff. Maybe AMD and Intel will step up their game to tackle that segment, who knows. Or maybe Apple Silicon gaming will gradually take off. But if neither of those happen, there might just be a huge void left in the "affordable PC gaming" segment (which used to be most of it, not so long ago).
On the other hand... I do have to give them credit and say that GeForce Now is AMAZING -- limited library aside. If they can keep growing that segment, hell, maybe the future of PC gaming is just in the cloud, like everything else, and nobody would want to buy expensive GPUs for home use that just become obsolete in a couple years anyway.
But aren't most hardware producers these days fab-limited, with Apple, Samsung, Nvidia, etc. all competing for the same few foundries, especially at the smaller processes?
If they only have X amount of production available a year, it seems like they'd want to focus those on the super-high-margin high-end datacenter cards rather than the leftover low-end stuff. Or maybe the high/low stuff are manufactured on different pipelines altogether, that don't suffer from the same production limits? I'm not sure how that works...
The gamers paved the way!
All you have to do is take 5 seconds in a typical code base to determine that the way we write software today isn't exactly... ideal. Given another 6-12 months, I cannot comprehend another ~OOM not being extracted somewhere simply by making the software better.
That's their price differentiator. It's the reason for their profits. nVidia is now <20% gaming in terms of revenue and that gaming revenue has much tighter margins than the 80% datacenter/professional market revenue which is almost all profit sine 80gb ram doesn't cost $40k more.
So yes it'd be nice if nVidia made it so there was no reason for their $40k AI cards to exist. But they aren't going to do that.
No one's catching up to the topline perf or even power efficiency (Intel actually suck at perf/power right now) but they might be able to compete with raw GPU costs at least. Give us a $2k card with 256gb ram and we'll probably take it over the $40k 80gb cards despite the power efficiency and topline perf that the $40k card has.
It will be this way until they get a competitor.
They're a major moat/hurdle (depending on your perspective) for operating systems, new hardware platforms, graphics cards, custom chips, and more.
It's interesting to think that we're not _that_ far from being able to generate decent drivers for things on the fly with the latest code gen advancements. Relevant to this, that could reduce the monopolies here, but perhaps as interesting is we can have more new complete OSes with more resources allocated to the user experience vs hardware compatibility.
It seems like it would be "easier" now that the path has been cleared and all they have to do is reach parity.
We also need more Nvidias. More competition in the chipset industry.
Is it just promise to sell chips in advance, so that's how far it's booked, do they own a Time Machine...?
It's not black magic, they have contracts in place and know both how many GPUs will be produced and sold give or take few %s.
Once enterprise adoption of AI picks up, demand for chips will increase 2-3 times further.
I'm told Nvidia's building their own fab in Southeast Asia over the next few years. This will massively boost their output.
In any case Nvidia is expecting to ship ~550k H100s in 2023, hardly enough to satisfy every user.
Tesla decided to in-house. TPUv4 and Gaudi2 exceeded A100 performance, they just never hit scale or the market and then Hopper added optimization for transformers rendering these chips relatively obsolete.
Nvidia's lead is not unassailable and it seems incredibly unlikely that they would not face serious competition within the next 2-3 years given the $ being thrown around.
It's not unassailable. But it's going to take a lot to make any difference to Nvidia's volume or pricing, let alone a meaningful difference. They already face serious competitors in google and aws with TPU and inferentia, but those competitors are at a pretty big disadvantage for now (and others too). The cuda ecosystem is a big advantage. Nvidia has a lot of leverage with semi manufacturers because of volume. They spend way more on chip R&D than their competitors in the space. They have brand recognition. You can buy and own Nvidia chips v tpu and inferentia. It's... a tough road ahead for competitors.
For what function have you experienced LLMs being used at significant scale in production right now? It seems unlikely most enterprises have built up sufficient technical know-how to run these workloads in-house already.
> They already face serious competitors in google and aws with TPU and inferentia
TPUv4s aren't widely available and are not competitive with Hopper. Inferentia isn't for training and is also not competitive on throughput.
> The cuda ecosystem is a big advantage.
Is it really though? A year ago it wasn't the case (see TPUs). Nvidia hasn't maintained a stranglehold on the software stack.
> Nvidia has a lot of leverage with semi manufacturers because of volume.
With TSMC*. If we believe Pat Gelsinger, Intel is starting to catch-up and their roadmap would place them ahead of TSMC in H2 2024.
> They have brand recognition. You can buy and own Nvidia chips v tpu and inferentia.
Intel, AMD and Google don't have brand recognition? There isn't a competitive alternative today for sale or rent, that doesn't mean this can't/won't change. It also doesn't mean they won't become available for enterprise purchase when we're talking about 60B/year in revenue.
> It's... a tough road ahead for competitors.
While hard I disagree that their moat is as wide as being touted. At the end of the day, whoever gets the most FLOPS, can deliver product faster and/or is cheaper will win as long as it's trivial to migrate workloads.
I really don't believe anyone actually cares who the manufacturer is.
Yep.
I believe so.
I don't believe Pat.
With the relevant audience, Nvidia is the brand which is the strongest. Google could just drop support for TPU, AMD isn't viewed as currently credible for people doing the work, nor Intel.
You may be right, only time will tell. I've been involved with GPU use for general purpose workloads since Cuda was launched. At every step of the way there was apparently credible competition at different layers of the stack just around the corner. That's over 15 years. OpenCL, ASICs, FPGA, Intel this that and the other, AMD this that and the other. TPU. Others I've forgotten.
Are you saying that you've seen enterprises developing, training and running their own in-house LLMs from scratch directly on large (i.e. 100s to 1000s) GPU clusters, whether on-prem or cloud, for this to be relevant to Nvidia?
Pardon my skepticism but it seems odd that a generic Fortune 500 co has the in-house talent and will-power to manage large distributed training runs when much easier and cheaper alternatives like OpenAI/open-source models or one of the Google/MS/AWS MLaaS options are available.
> With the relevant audience, Nvidia is the brand which is the strongest. Google could just drop support for TPU, AMD isn't viewed as currently credible for people doing the work, nor Intel.
I think we're confusing some things here. Right now, there is no good alternative for anyone requiring H100s for loyalty to even matter, this could very easily change with the next generation of accelerator chips.
Intel had the strongest CPU "brand" for a while and enterprises/datacenters readily switched to AMD when it became the better option.
> You may be right, only time will tell. I've been involved with GPU use for general purpose workloads since Cuda was launched. At every step of the way there was apparently credible competition at different layers of the stack just around the corner. That's over 15 years. OpenCL, ASICs, FPGA, Intel this that and the other, AMD this that and the other. TPU. Others I've forgotten.
The TAM, and profit margin, for enterprise-grade GPUs (or accelerators) is several orders of magnitude larger than it has ever been including the crypto craze.
Right now it doesn't matter, but you are suggesting there will be real competition soon. I'm saying, it will matter.
Intel did have a great brand. And they proceeded to screw up and it caught up with them, but it took a long time. A company with a moat can destroy it with time and effort.
It is huge market. And it's a huge lift to build a product that can compete with Nvidia at this point. I'd handicap it at 2-1 against someone being a real competitor within a decade.
I'm not moving the goal posts but maybe I wasn't clear. When we talk about corporations adopting fine-tuning and inference that's not especially relevant to H100 sales which is the main cash-cow (~10B of revenue at 80-90% margins) and Nvidia's massive market cap growth. What is relevant is corporations like Inflection AI building 22k H100 clusters.
I work in academia so PyTorch is more common but are people in industry fine-tuning LLMs actually working directly with CUDA that much for this to be a big moat?
No, no one is using CUDA directly. But if you are a vendor with a new equivalent, it's no small feat integrating it with a framework like PyTorch. There is a lot of work required by various parties.
PyTorch has gotten a lot better on TPUs this year, I don’t believe there’s much of a performance hit now. Jax and TF (I don’t use the latter anymore) of course work. I never used Gaudi2 but it apparently works.
All of this to say, is it possible that Intel gets their fabs working and strategically partners with their long-time partner MS and OpenAI to extend Triton to Gaudi3 or 4 and be a threat to Nvidia within 2-3 years? Absolutely.
Is it similarly possible that Google increases development on Jax and TPUv5? Sure.
Neither of these possibilities, regardless if you think they’re probable or improbable, would need a decade to catch up to Nvidia.
They’ve got a monopoly. And with AI’s coming explosion, I’d wager 50/50 odds Jensen becomes the world’s first trillionaire.
Possibly their greatest asset, as an investment, is their crazy high margins. Nvidia in 2023 is where Intel was in 2007, where they could basically charge almost any price because they were so dominant in the market. I remember when E5s were selling for $2000 a pop and data centers were using thousands of them.
That and possibly way more than that is already priced in. Nvidia's stock is extremely expensive not because of they are making now (which is not a lot relative to valuation, they just barely surpassed Intel this quarter in revenue) but because investors expect pretty much exponential growth over the next few years..