Apple M1 support for TensorFlow 2.5 pluggable device API
developer.apple.com
developer.apple.com
The raw compute power of M1's GPU seems to be 2.6 TFLOPS (single precision) vs 3.2 TFLOPS for Vega 20. This can give you an estimate of how fast it would be for training.
Just for reference Nvidia's flagship desktop GPU(3090)'s FP32 performance is 35.5 TFLOPS.
I was talking about the mobile GPU on MacBook Pros which is based on a 14nm chip. The full name is Radeon Pro Vega 20:
https://www.amd.com/en/graphics/radeon-pro-vega-20-pro-vega-...
https://www.techpowerup.com/gpu-specs/radeon-pro-vega-20.c32...
Vega 20 seems to also refer to a discrete GPU. This has been later rebranded to Radeon VII (maybe because of this confusion). The number you are quoting is for the discrete GPU.
In context of ML, nvidia’s flagship is the A100, which has 312 TFLOPS. You can also compare with a TPU device which has 180 TFLOPS (v2) or 420 TFLOPS (v3). You can use at least the TPU v2 reliably on colab for free.
FP16 performance is also relevant as a lot of people now train in FP16. The default for pytorch/TF is still FP32.
Would like to see some benchmarks with that sort of hardware or with a 6900XT, driver support which came to MacOS only recently. Now I just have an M1 Mac Mini & a PC with Nvidia.
Doesn't look too bad for Nvidia, especially when you consider 3090 is still on Samsung 8nm, which is equivalent to TSMC 10nm, compared to 5nm on Apple M1.
It would be great if they did it the eGPU route with TB4, in a slick package
We’ve only see the Apple equivalent of the i3 with integrated graphics. It’s going to get interesting over the next several months as Apple unveils their middle and upper performance solutions.
To get the M1 warm, I need to be charging it while also playing an Intel based game like the latest Subnautica, with the machine on my lap. Even then, it's not as hot as my Macbook Pro gets while unplugged and browsing the internet.
That is exactly what the rumours suggest Apple is working on.
Regardless, Apple's threat to vendors like that is their complete vertical integration. Ran some of the new object capture code (photogrammetry) on my M1 Mac yesterday and in no time at all the 11 trillion op neural engine blasted through and generated a remarkable model. We've seen Apple respond to discovered performance needs by plugging in a matrix engine, a neural engine, and scaling appropriately, dedicating cores and silicon to the greatest needs. They are in a very unique position relative to someone like nvidia who effectively throws something over a fence.
Also, Apple desperately needs to support a real graphics API. Metal is a joke, and even the translation tools like MoltenVK, while impressive, still end up beholden to Apple's arbitrary limitations. If they don't end up supporting Vulkan on the M1, it's a moot point for me. You could have the most powerful GPU in the world, but I won't use it if it's bottlenecked by the shittiest modern graphics API.
The GPU in the AS M1 is the fastest integrated graphics available in the mainstream computing market [1]. That is the competition, not a standalone, 120W GPU. Apple is purportedly now working on separating their GPU designs into a much larger heat and power profile (which contrary to some of the comments on here clearly isn't going to be for laptops, beyond an external TB4 enclosure) and it might just change things a bit.
Scaling a GPU is easier than scaling a CPU, by design. Apple's GPU has nothing to do with ARM.
And to your original point, yes, Apple does largely choose which workloads run on Apple Silicon and how. By controlling the APIs along with the silicon, Apple abstracts it to a degree that gives them enormous flexibility. The Accelerate and CoreML APIs are abstract vehicles that might use one or a thousand matrix engines, neural nets, or an array of GPUs. Apple has built a world where they have more hardware flexibility than anyone. And while close to no one is doing model training on Apple hardware right now, Apple has laid the foundation so a competitive piece of hardware could change that overnight.
[1] The SoC graphics of the chips in the PS5 and Xbox Series X have more powerful graphics, but the GPU chiplet alone on those systems uses a magnitude more power and more die than the entire M1 SoC. In another comment you mentioned that Zen 2 integrated graphics come close. They aren't within a ballfield, literally with 1/4 or worse the performance. In discussions like this unfortunately the boring "n years old / n process" trope is used to excess, yet again there are zero competitive integrated graphics on the market. None. Apple isn't a GPU company, yet here we are.
Bzzzzt
We aren't comparing performance-per-watt, we're comparing raw compute for the price.
All while Apple keeps executing superbly. Eh.
It's extremely doubtful any new Apple Silicon device will come out with AMD GPUs.
Following the instructions: -----
python -m pip install tensorflow-macos
...
ERROR: Failed building wheel for numpy
Failed to build numpy
ERROR: Could not build wheels for numpy which use PEP 517 and
cannot be installed directly(base) dave@daves-air ~ % uname -a
Darwin daves-air.lan 20.5.0 Darwin Kernel Version 20.5.0: Sat May 8 05:10:31 PDT 2021; root:xnu-7195.121.3~9/RELEASE_ARM64_T8101 arm64So basically the software that you require to run and do your work is still not available for Apple Silicon? In this case, Apple Silicon support for Python libraries.
The whole point of Apple Silicon is to supersede its Intel Mac counterparts and to run software natively with a noticeable performance increase, especially with software with high performance requirements.
In the case of python users and the ecosystem around it, it is still not ready.
Glad I sat that one out.
Nothing hypothetical about being an early adopter and then complaining and wasting months since the November 2020 release day chaos that several software is still unavailable or unstable, the very basic software doesn't even run on the system and the excessive disk writes on the M1 quickly wears the SSD on the machine. All completely real and happened to many people since launch day.
So, what's the point of buying a laptop that doesn't even run your software in the first place? Might as well stay on your existing laptop. You would rather wait N amount of months for the software to mature and use the laptop reliably for your work than to skip all of that, use your existing laptop and get a better one? (M2 Mac).
Why would I want to wait for months for the developers to port their software and its library ecosystem to Apple Silicon or waste time with broken workarounds when I can use my existing laptop that already 'just works' with everything.
Mind you, I actually bought the M1 MBA recently to try it out and returned it at full price due to the software ecosystem not being ready. Not only I saved my money, I >5x'd that money it in a recent investment anyway and now I'm glad I did that.
$ pip install -U tensorflow-macos --no-deps
That seemed to work for me at least.
The first time I did 'import tensorflow', the import line ran for a long time and then complained about flatbuffers being missing.
Then I ran several commands to install flatbuffers - all of which I could have sworn claimed it was already there, but then I tried the import again and it worked.
'import numpy' works too so the env feels complete enough.
My python binaries are coming from ~/miniforge3/bin
So all is well now, thx.
python -m pip install --upgrade pip
pip install --upgrade pip
It's ultimately just a matter of experience I guess
I've also had to use nvm to install an old version of NPM for a specific project because otherwise one of the NPM dependencies couldn't compile a certain C++ executable that I apparently needed? There was also an incompatibility with some binary that another dependency downloaded that required me to mess with soft links to libraries in specific places.
I don't think either NPM or PIP are inherently hard to use as long as you keep them updated (which is exactly what the parent comment is suggesting to do) and as long as you don't need binary dependencies. When you end up in binary territory, which this type of software eventually will, you'll run head-first into stuff that requires arcane commands to get stuff to run.
Python’s ecosystem is the worst for get-up-and-go usage.
Nvm or nodenv make managing node environments trivial compared to virtualenv, pip, and co. Same with rustup or rbenv… in fact python is the only language i have the problem with.
i can't be arsed to deal with all the various version dependencies and incompatabilities and system install vs local install nonsense that comes with installing it on my actual computer.
First, I ran:
pip3 install virtualenv
cd ~
python3 -m virtualenv tfenv -p python3 --system-site-packages
source ~/tfenv/bin/activate
alias tf2='source ~/tfenv/bin/activate'
So, the goal is to install tensorflow-macos and tensorflow-metal, but the problem is that their pip3 command is failing with some obscure numpy error.
Here's the command that works:
pip3 install --no-dependencies tensorflow-macos tensorflow-metal absl-py wrapt opt-einsum gast astunparse termcolor flatbuffers
That method worked for tracking down every library except absl (unknown library name). But googling pip3 install absl showed that it was named absl-py, not absl.
Now I can run python3, then this code:
import tensorflow as tf2; tf = tf2.compat.v1; sess = tf.InteractiveSession(); sess.list_devices()
Note that it spits out a warning like this:
WARNING:root:Limited tf.compat.v2.summary API due to missing TensorBoard installation.
ERROR: Could not find a version that satisfies the requirement tensorflow-macos (from versions: none)
ERROR: No matching distribution found for tensorflow-macos
How is Node/React development, for example. Last I heard, Postgres was good to go on native M1. I know Node 14 is good natively. Are there commonly encountered problems? I hate my 2017 MBP and am desperate to upgrade.
Can you back that statement up with anything, or at least clarify it? You seem to suggesting a non-mac i7 with a separate GPU. Also, just an FYI, "i7" says pretty much nothing. The i7s have existed since 2009.
I don't know. The statement is just so vague and ridiculous. The M1 is probably the worst hardware you could have picked in 2020-2021 if computational power was your main concern. For highly parallelizable work tasks, the top end GPU alone has 10x the computation power than the M1, and a top end CPU has around 4x the computation power than the M1. Not to mention a rather limiting 16GB of memory. That the M1 is computationally powerful is a myth started out by exceedingly misleading marketing and reinforced with hard-to-compare benchmarks.
To possibly save someone the trouble, the responses to a comment such as this, from experience will be:
1. "power consumption is much much better on M1 than anything else." True, but, then again, your use case must then prioritize power consumption, and not computation power. So which is it? The use case for a compute cluster on a train is rather contrived.
2. "When apple scales up the M1 to more cores, they will magically be able to retain all the benefits possible with a low core count, and scale it up without any problems or compromises, just you wait.". Ok, I'll wait.
3. "It's not fair to compare a laptop with a desktop". Of course it is. The constraints for comparison are already stated: computation power being the main priority. If someone buys hardware to do heavy computations, you can pick and chose depending on your needs. If you need it to be a laptop, or you need it to draw little power, then I'm sure you can factor this in accordingly.
Apple themselves changed their tune about the difference almost overnight.
It feels like you've constructed quite the straw-man to tear down.
Praise for the M1 is in the context of the form factors it exists in and the efficiency it works at.
Of course you can find more powerful hardware in larger form factors drawing 10x the power from the mains. The M1 runs in an iPad for crying out loud.
> The constraints for comparison are already stated: computation power being the main priority.
Then don't buy a bloody M1. The M1 has always been Apple's entry-level efficiency-first processor. We haven't even seen the "Pro" stuff yet.
> Then don't buy a bloody M1. The M1 has always been Apple's entry-level efficiency-first processor.
No, I gotta disagree. Apple's marketing around the M1 was intentionally deceptive: they were forced to revoke their claim of having the "fastest CPU cores" after it was vehemently disproven. Their "faster than 97% of Windows laptops" conveniently didn't compare itself to AMD laptops or laptops with dedicated graphics. I'm just not really impressed. I seriously worry for Apple if this is all they were able to get out of the 5nm node on ARM. Considering how poorly ARM scales with higher TDPs, I don't think I want to see their "Pro stuff".
My point is that this isn't even an interesting conversation to have. Your desktop can outperform a chip that runs an iPad? Cool story bro.
> they were forced to revoke their claim of having the "fastest CPU cores" after it was vehemently disproven.
Their announcement explicitly said "when it comes to low power silicon." So many of you insisted on missing this part it's practically a meme now.
Watch the accouncement video here: https://www.youtube.com/watch?v=5AwdkGKmZ0I and scrub to 8:45 and actually listen to what is said.
> laptops with dedicated graphics
Why should they be forced to compare their integrated graphics to laptops with dedicated graphics? This is yet another uninteresting comparison. Of course a machine with some dedicated mobile 3080 is going to win the day in a head to head. But it's an absurd comparison because said machine will draw vastly more power and be heavier to boot. It's a different category of machine at that point.
> I seriously worry for Apple if this is all they were able to get out of the 5nm node on ARM
I think you're the only one who's worried.
Apple very clearly laid out their design goals with the M1: supremacy at performance per watt. And the fanboys protested with "I can build a faster desktop" or "my 4.5 lb laptop with dedicated graphics can get more FPS." It's just baffling. You've missed the whole point.
The desktop I'm comparing it to is 7 years old and cost $600 new. You can't even buy an M1 iPad for that price today.
> Their announcement explicitly said "when it comes to low power silicon."
Which is an arbitrary goalpost that means nothing. The M1 uses 7w at full tilt, does that mean we can compare it to an AMD 5800u running at the same wattage? It's a nothingburger, and that's why Apple doesn't use that zinger anywhere else in their marketing material.
> Why should they be forced to compare their integrated graphics to laptops with dedicated graphics?
Because that's what you can buy for $1000. That's the performance standard. If Apple wanted the M1 to be compared to machines with integrated graphics, they should have released a computer at that price point.
> Apple very clearly laid out their design goals with the M1: supremacy at performance per watt.
Sure, they have it. But I frankly don't care, and I have a hard time believing that other people do too. Performance-per-watt is Apple's neat way of giving performance a denominator, because they simply can't compete with the rest of the industry wholesale. It's something they've done over and over, insisting on pointless metrics like thinness and beauty to measure a product of objective capability. The datacenter market is looking at the M1 and laughing. Unless your business was already Mac-based, it's not like enterprise customers are going to be interested in beta-testing Apple's new hardware either. Honestly, I'm more impressed with Apple's social engineering than hardware engineering here.
Which desktop is this with what components? What's the metric you're comparing? Show me the benchmarks. This still feels like the world's dumbest comparison but hey I'm at least curious now.
> Which is an arbitrary goalpost that means nothing.
But it's the goalpost they very clearly set--you're the one who either missed it or willfully chooses to ignore it, which means you're the one being misleading, not them.
> Because that's what you can buy for $1000. That's the performance standard. If Apple wanted the M1 to be compared to machines with integrated graphics, they should have released a computer at that price point
A pure price comparison on its own is missing so many relevant factors and you know it. You can also buy a desktop that outperforms a smartphone that costs the same. Is that a fair comparison? How much energy do these machines consume, what's their battery life under load, how much do they weigh, etc etc. I don't just walk into Best Buy and say "here's a $1,000 bring me the thing with the best Geek Bench score, no other criteria."
>The datacenter market is looking at the M1 and laughing
What are you even talking about now. Since when was the datacenter in the design goals for the M1.
>But I frankly don't care, and I have a hard time believing that other people do too
No one cares about battery life or the weight of their laptop? This is news to me. I have a feeling it's news to all the people who buy ultrabooks, too.
>Performance-per-watt is Apple's neat way of giving performance a denominator, because they simply can't compete with the rest of the industry wholesale.
Because they're not trying to outcompete the entire industry in every form factor with the M1, since the goal is to provide stellar performance while still achieving efficiency that enables small form factor designs. You are exhausting with this "willfully missing the whole point" thing.
Again, they haven't even released the parts that are actually meant to compete at the higher wattages you're trying to compare against the M1.
I'll make an analogy--maybe it'll help. Car Company A just came out with a brand new sedan with industry best Miles Per Gallon. And beyond just the MPG rating, it can actually tow some pretty heavy loads, too! Heavier than other sedans in its weight class. Impressive stuff.
You're the guy with the F-350 dually pumping his chest saying "mine can still tow more!"
Basically all of them, as long as you aren't training your models on your CPU like it's 2011.
Who compares ultrabook to desktop for the multithreaded computation power? Assume you only want that without any regards to power or space or cost, by your logic you could buy 1000 mac mini and the computational power will be more than any desktop computer.
This statement is so banal that I am not sure how to comment on it. I never understood the logic of people who point out that an entry-level, low power chip targeted at ultraportable laptops and kitchen computers cannot compete with high-end desktops. It's just as ridiculous as to complain that AMD EPYC is too big to fit into a laptop.
M1 is obviously a terrible choice if you are looking for a deskbound HPC workstation. It's a terrific choice if you are looking for an ultraportable laptop with excellent battery life that you would still like to prototype your ML code on before running the full workload on a mainframe.
> That the M1 is computationally powerful is a myth started out by exceedingly misleading marketing and reinforced with hard-to- > compare benchmarks.
More like a myth perpetuated by people who like to take facts out of the context. In terms of the underlaying IP, M1 is terrific technology. It can deliver the same performance as state of the art designs at a fraction of the power consumption. In terms of absolute performance, it's obviously an entry-level chip, and it performs exceedingly well compared to other offerings in this segment. And even outside its segment it is no slouch either. It runs my database scripts and builds code faster than my Intel i9 laptop, despite using 70% less power and half as many performance-oriented CPU cores.
> M1 is obviously a terrible choice if you are looking for a deskbound HPC workstation ...
So, in one breath you say my statement is banal, in the next you agree with it. The thing I take issue with is having to deal with the misconception that it is a replacement for a HPC workstation, because many (not you obviously) think it actually is a powerhouse.
You go on to repeat the stuff about power performance, which I've already listed, and you could have spared yourself the trouble.
If you haven't already, I'd suggest looking at https://news.ycombinator.com/newsguidelines.html. And I mean that too in a polite way, as it is easy to get carried away and assume the worst in anonymous conversations with strangers.
So, to summarize. Apple originally made highly misleading statements on the computational power of the M1, and since then, this misconception is repeated quite often. After a while, it gets annoying to see new posts on HN, week after week. Now, alluding that an M1 is 3x faster at training models than discrete GPUs can only be true if it is a decade old hardware. And as that stands, your original comment was asked to be clarified. After all, it's not unheard of for M1 benchmarks to disable the discrete GPU of the PC it is compared to, for "fairness". Limiting the training to a single CPU core for "even more fairness" would be par for the course.
So, I did not suggest you were an "appleboy", as you put it. I asked what kind of hardware you actually experienced a 3x speedup on, as the i7 goes back to 2009, and "GPU", well, further back.
In any case, and this might come as a surprise, feel free to not answer. But, then, I can only wish that you refrain from making the effort at a rude response.
I also noticed that in my project I got a decent speedup immediately when executing my model, but I have not run any benchmarks.
But, where do you go to file bugs? Ask questions? etc. I am not a big Mac developer, so is there something I don't know?
Does this give a GPU:0 device? You can check via:
import tensorflow as tf2
from pprint import pprint as pp
tf = tf2.compat.v1
sess = tf.InteractiveSession()
pp(sess.list_devices())
Init Plugin
Init Graph Optimizer
Init Kernel
>>> tensorflow.config.list_physical_devices
<function list_physical_devices at 0x128011f70>
>>> tensorflow.config.list_physical_devices()
[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
God, it’ll be nice having a gpu for tensorflow. I’ve dreamed about this for like… a long time.
Ahh, I've never had a local GPU before. Thank you Apple Santa Claus, whoever your team is.
Wow, that list of CVEs is 110 lines.
Are people actually using tensorflow to run untrusted models?
In short, you will probably get much better performance using Google Colab to train a deep learning model, than the M1.
I yearn for the day when someone makes a nice, simple, "install this and python ML works fine with your GPU" package.
It is then supposed that with enough stacking, apple could get close to a dedicated GPU performance.
However this is speculative.
It's a monolithic SoC, and while it is packaged with the RAM that's just closer integration of chips that would normally be on-motherboard or a DIMM?
Eg., just look at the M1 variations present atm: variable gpu cores.
The "chiplet" speculation is that given the M1's variable high-perf, low-perf and cpu cores, this can scale up.
Leaks at least are all pointing in a 32-core, Xeon-competitor direction. It is theoretically possible they could do the same with GPU count, and try to compete perhaps in the mid-pro gfx range.
The "you can scale up" thing is actually just how e.g. Intel makes bigger monolithic chips (Xeon/HEDT) with the same or very similar cores as the desktop ones. Meanwhile AMD, actually using chiplets, can cheaply do something more like "scale out" in the sense that they put more of the exact same die on a package.
Might also be related: Tensorflow Lite Core ML delegate enables running TFLite model on CoreML enabled iOS devices.
https://www.tensorflow.org/lite/performance/coreml_delegate
Would nice to see performance comparisons on M1 Mac/iPad for which way is more performance and efficient. (Admittedly TF vs TF-lite is a 100% apple to apple comparison, pun-intended).