Hacker News Reader

Comment by Twirrim | Hacker News Reader

One thing I think IBM has failed to grasp about POWER architecture is how to get people to use it.

Lots of people in tech talk about being interested in trying it, it has a number of interesting characteristics, but the barrier for entry is fairly high.

It's possible to get a POWER8 based server on Softlayer, IBM's cloud product, but not as a VM. You can get one as a bare metal server, but you can't get one for an hourly fee. You have to pay for a full month, which starts out at around $1000.

There are very few individuals that would be willing to make such a commitment, but so many that would be willing to spend a few tens of dollars on spinning up a VM for a few hours to see if it provides value.

If you want people to get excited about it, or interested in using it, you really need to make it easy for people to test it on a small scale.

View on HN

This. A million times this.

There are no entry-level POWER machines, SPARC machines. There is no entry-level IBM i or Z. There isn't even a software emulator for i (and even for Z, where there is a very nice emulator, you can't legally run anything remotely current on non-IBM hardware).

You can get a cheap POWER or SPARC on Ebay, with a sharp price drop when the equipment is EOL'ed, but that's no way to entice people to build new stuff on a platform.

I love my PA-RISC, my RS/6000's and my SPARCstations, but, neat as they are, they aren't anything that would make me consider building for their modern descendants. And I won't pay more than a decent, proven, certainly useful Xeon workstation worth of money for a cool, but exotic-that-may-not-run-my-software-well, POWER9 performance equivalent.

If IBM wants me to pay Xeon Platinum prices for their production gear, they'd better allow me to pay Core i prices for development gear, or else I'll just keep deploying on Xeon, which is good enough.

Typed on a very comfortable and reasonably priced Core i7 laptop.

Seconding what you just said, and another thing is that the cost barrier to entry to developing x86-64 software on a very current Linux platform is almost zero. If you are a curious 15 year old who wants to learn Linux stuff you can get an older Core 2 Duo desktop PC for free, or almost free ($20) from a number of recycling places and install the latest debian-testing AMD64 on it. Put it on your home LAN, ssh into it, and start doing whatever you want.

I know of companies that regularly throw away second generation core i5 dual and quad core systems with 8GB of RAM.

The same software you develop on your $20 headless desktop PC can be easily moved over onto a $15/month KVM virtual machine somewhere in hosting. Or onto a used $250 1U server that you can pay to colocate somewhere for $65-100/month with bandwidth. All x86-64 platform.

The massive, massive economies of scale for x86-64 platform stuff are going to be very hard to get people to move away from unless there is an amazingly compelling reason. The ability to cannibalize random recycled computers to make one working computer out of "free" parts is a big thing for the developing world.

A very important point is that you mostly don't need to buy a new computer. Chances are a lot of teens already own perfectly good x86 computers that can do a lot of things they want and be the machines they learn to develop software on.

IBM has to compete with free hardware.

IBM might be having trouble letting go of their old model. If you want their amazing hardware they’ll do everything for you. Which means on day one you have none of the expertise needed to reverse that decision.

If you come from a world where lock-in is normal then you may well build something that locks everybody out.

You aren't their target market share. They are focusing solely on the HPC Market at this point. Their lowest in shape has 16 cores. Contrast that with a xeon at 6 cores. I think you were hoping that they have a desktop processor that you could play around with the actual architecture on. That would be nice, but they likely don't want to put all their eggs in too many baskets right now.

Except for special US DoD things (like supercomputers that simulate nuclear weapons) for Sandia Labs and the like, the math also doesn't work out for HPC...

Let's say that a 44RU rack of 1U, narrow (half of 17.5" width), dual motherboard, single socket systems costs $N if it's built with Xeon or EPYC CPUs. As an example here the Supermicro barebones which are two long, narrow motherboards in 1RU with single sockets and front to rear wind tunnel airflow. Same rack of systems costs $N multiplied by 2.25 if built with power9, but only performs 1.25 times faster.

I don't follow the logic. Where are you getting your pricing information from? Why do you think power is more expensive?

the cheapest power8 server I have ever seen was probably 4x more expensive then the equivalent dual socket xeon. That info is a bit stale. But there is also a complete lack of good information online about actual dollar figures for power8 systems (a lot of "contact us for more info!"). The platform's pricing is aimed at the sort of CTO people who don't care about the cost of Oracle licensing. Not the sort of people who care about $/MIPS for HPC applications or cloud scale hypervisor platforms.

I would very much like to be proven wrong so if you have access to some specific "this hardware spec costs this much money and you can buy it now" info, that i can compare to barebones supermicros, please do share it.

Email supermicro and ask about pricing, they are trying to gauge interest currently.

You're right in that the sticker price is much, much higher. I'm surprised you were even available to find one. IBM tends to like to keep things secret until you reach out to them, for better or for worse. I would like to see some public power 9 benchmarks to compare to xeons instead of just marketing speak as well.

But I will say that your comment about $/MIPS isn't accurate. There are many people, including Oakridge national labs, who are using it because of the higher performance and v100 integration. Intel is obviously not too keen on letting Nvidia succeed.

somehow, compared to the vast amounts of money they spend on other things, I think pricing is way down the list of things that are important to organizations like LLNL, Argonne, Sandia, Idaho National Labs, PNNL, etc. Basically everything that uses HPC for DoD top secret projects.

At the same time, you need mindshare at the lower end to drive demand for the high end. I previously developed for POWER by getting an Apple Mac Mini. Very affordable, and while not a full POWER instruction set was sufficient to test and validate my code. Today, there is no equivalent, and I'm about to dispose of the Mini. I couldn't justify use of POWER today even though we have a datacentre stuffed full of machines for HPC, and could likely make use of it. We have zero experience with it, and no easy way to get any experience. Being able to buy a cheap low-end system for experimentation is important. If I wanted to do this for ARM, I could buy half a dozen cheap boards online this morning and have them delivered by Monday. But POWER is out of reach.

So...

I sit on the POWER Customer Advisory Board, what you have raised has certainly been something I've told IBM multiple times over the last few meetings I've had with them.

I plan to continue pushing on this front.

From IBM's history, I'd assume this is chalked up to IBM institutional inertia (enterprise customers make big deployments, nothing else matters) than willfully doing it for some other profit-oriented purpose?

There are NDA items I can't get into but certainly it's not the case where I've just been shouting into the wind.

All of IBM DNA is structured to sell big contracts and deal with a relative small number of customers with very deep pockets.

Even when they try some new trend ( aka cloud computing for the masses ) the economics aren't that good because the way they are structured, it's a culture thing.

It's like Intel developing ARM CPUs, they have the means, but they can't do it successfully.

Thank you. I would love to meet around with POWER, but I don't have any way to get my hands on it for a price I can justify.

One of the things that Nvidia quite successfully did (and Intel to some extent) is a ridiculously good academic outreach program. Making it easy (and cheap) for students and researchers to use you hardware leads to long-term buy-in.

We (Oregon State University Open Source Lab) have been working with IBM for the last few years to help with this problem. We host a set of P8 (and soon P9) machines for them so FOSS projects can work on porting issues. You're welcome to sign up if you're interested [1].

[1] http://osuosl.org/services/powerdev/

On the same note, we are providing the same for OSS projects wanting to leverage mainframe ( https://www.openmainframeproject.org/blog/2018/01/08/make-ma... ) and service that lets try a mainframe running Linux in the cloud ( https://www.openmainframeproject.org/trymainframe )

Not any worse than the past decade of arm server "promise"

Bootstrapping an alt arch is hard, it really needs a loss leader OR a significant performance delta (which they now have with accelerator workloads). Google deploying it in prod is a major milestone and will help with volume and confidence.

Supermicro has a P8 and P9 line. It is price competitive with x86. If you are interested, send their sales a note.

Come on, you can buy a load of ARM 64 bit development boards starting from just a few tens of dollars. There is no problem at all getting access to ARM. And from there you can scale up to Cavium 2 or Amberwing which have Xeon-like performance (better than Xeon in some ways). If there's a problem with ARM I'd say it's still the lack of standardization, and also very few mid-range servers.

POWER is nothing like this. There are no development boards at all, for anything less than $thousands, and the real servers have great performance but sky-high prices.

ARM dev boards are generally based on toy-sized SoCs, though.

They are still useful though. If you want to build an 7-node cluster of 2GB nodes on a single Mini ITX motherboard, it's even cheap. And requires no messy cables.

https://www.pine64.org/?product=clusterboard-with-7-module-s...

and seven https://www.pine64.org/?product=sopine-a64

That's neat. I had heard of the PINE64 board but never the SOPINE. Curious, have you implemented any projects on this? Seems like it might be neat for testing a small distributed system.

I'm still building a cluster of Pi Zeros connected via a USB hub. Since I'm doing it on my spare time, it's not going fast.

The point of clusters like this is not the speed, but the fact it's a cluster, with all the bottlenecks a cluster has.

On USB it seems I'll have to route all traffic between the nodes via the cluster controller, which will also serve the shared NFS volume.

Those SBCs are still stuck with using very outdated SoCs usually with a quadcore A53. Even a 7 node cluster of SOPINEs is going to be slower than an iPhone X.

That's not the point. The point is that it's a cluster of network-connected nodes that can run something like Kubernetes.

You can't actually buy TX2. I don't know about market availability of Amberwing either (if you know please let me know). There is an abundance of arm stuff in the embedded space where it is king, and it's true you can pick them up for next to nothing but somewhat orthogonal to servers (flattened device trees vs ACPI/UEFI etc)

You can web buy an S821LC straight from IBM for $5k for the past couple years. The AC922 is GA. Supermicro will sell you a P8 right now and P9 in May for nominal prices and the CPUs are cheaper than Skylake by a very wide margin.

Not directly related to your comment but I get the feeling a lot of people complaining about price are navel gazing and have no idea how much a production server costs and how the costs break down. Right now storage is generally 50+% of the cost. DRAM is a very high fixed cost at the moment as well. Intel flatted the quad socket SKUs into the "Scalable Series" so Skylake represents a big price increase for a lot of builders. To help re-calibrate people, IBM doesn't have $6-12k CPUs in the dual socket config but Intel does at bins people would want for common workloads.

> Not directly related to your comment but I get the feeling a lot of people complaining about price are navel gazing and have no idea how much a production server costs and how the costs break down.

They're not asking for production servers.

$5k for a workstation, for a minority architecture, is basically a fancy way of saying "no" to the army of tinkerers you need to widen the base.

Why can't you buy a TX2? Arrow appears to have 462 in stock right now.

I have my own Cavium rep. Show me a real CPU, board and worthwhile sheet metal (i.e 10+ nvme) I can buy in volume because you might know something he doesn't and I'd be happy to eval it.

see i.e. https://www.servethehome.com/gigabyte-thunderxstation-using-... for public info

"At STH we are working with Gigabyte and Cavium and will share more about the ThunderX2 architecture as we are given the go-ahead. We have heard the next production run is in the Q2 2018 timeframe."

This has been my experience with cavium, bait and switch slideware and when you do finally get a sample it has so far been undesirable (octeon, TX1)

What is the maximum dollar amount that you would be willing to pay?

Remember, it's never going to be the same price as a RPi.

But it is worse. Over 12 million Ras Pis have been sold, I see them just lying around on developer's desks all the time. I don't think I've ever even seen a bootable Power system.

The local UUG I was a part of had one available through the IBM innovation center. You were able to schedule time on it to experiment. Even with that close access it seemed like a PITA.

Other than the one in my house, you mean?

Have I been to your house?!

Does that mean it doesn't boot?

I will concede one point: it was not easy to purchase. I prefer to host my personal site on Power hardware; I started with AIX in the 3.2.5 days and I ran Floodgap on an Apple Network Server 500 for the better part of 14 years. I wanted to get a POWER7 to replace it in 2010, I budgeted $15k for it, and IBM wouldn't take my money. I couldn't find _any_ IBM VAR who would do an end-user sale because I wasn't going to buy the service contract.

Eventually I found a reseller who was more than happy to take $10K of my budget for a decent 2-year-old POWER6. It had a backplane burp a couple years ago but otherwise has been pretty damn spiffy.

I'll concede IBM has to do a lot more to get these systems into people's hands to achieve a critical mass and it certainly wouldn't hurt to make them cheaper up to a point, but the systems are out there, and you can get them (and find them).

As a postscript, in my current job (a large local government agency) I was in the CIO's office one day and the regional IBM salesdroid dropped by. Just to needle him I told him this story and he gave me his card and told him to call him with any parts requests, any time. I still buy from the reseller, though. They've earned my personal business.

He said he's never seen one, not that they don't exist.

Even in your example, the barrier or entry is still so high — I can buy a very nice new x86 machine for $10k, instead of a 2 year old POWER system. I am interested in the POWER architecture, but not buy $10k worth of old hardware interested.

I have a couple RS/6000's in storage...

But yes... Vintage computing doesn't count.

That pretty much tells the story. A single, last man standing vendor like IBM with a business model of making money on Mainframe/“Midrange”/Old-School Unix boxes will never, ever be Intel. POWER is cool, but who cares.

It’s more productive to think about how to harness lots of cheap ARM cores if intel isn’t doing it for you.

My employer has a few. It’s fast and looks impressive. But when you cut away the bullshit, it mostly exists because the sales guy presented a story where the cost of a new POWER box is a better deal than maintenance on the old one.

If you really dig into it, there’s no scenario other than a license play where a transition to Intel isn’t more cost effective. Even in those scenarios, you can usually engineer a solution (Oracle, etc) where you deliver a better ROI on commodity hardware or cloud hardware.

IBM used power as a sort of step-down from Mainframe when big enterprises moved from cobol to java. That came with all of the bullshit like leasing CPU/hours and getting more software revenue using their "value unit" model.

The other thing with power is that it has been a scale up vs. scale out product. That makes sense when you want to optimize your Oracle/SAP licensing or something similar. It doesn't make sense for modern use cases where you're using open-source or other solutions where growing infrastructure into your use case makes more economic sense.

I don’t know how easy or hard it is to get on board or what strings are attached to it, but https://www-356.ibm.com/partnerworld/wps/servlet/ContentHand...:

”The Power Cloud that enables developers offers no-charge remote access to IBM hardware, including IBM POWER8, IBM POWER7+ and IBM POWER7 processor-based servers on the Linux, IBM AIX and IBM i operating systems.”

No real strings although the machines get recycled weekly and access is via vpn only.

An interesting discussion to be sure. As far as cloud based access go, off the top of my head both Oregon State and Nimbix give access to P8/P9 instances for developers, free or cheap. One of our members, Raptor do a nice high performance dev workstation, pretty sure they do everything from a mobo+CPU combo to complete systems. I think lower end than that might well come, we certainly see it as something we'd like to facilitate through the OpenPOWER Foundation.

https://www.nimbix.net/ibm/

I absolutely agree. It baffles me that after TWELVE years from the launch of S3 and EC2, IBM still hasn't understood what made them popular. (hint: the pay-as-you-go model).

> One thing I think IBM has failed to grasp about POWER architecture is how to get people to use it.

I don't think IBM cares about people using it, especially people who can't afford $1000 dollars. They are only interested in high margin corp/gov pork barrel deals and need something that they can plausibly claim is superior in order to charge 10X for it.

Sales-commission driven product development.

They definitely aren't charging 10X for it, it's very much in line and competitive at the Xeon level in terms of pricing, and they shine for certain workloads, especially with the new interconnects (CAPI, NVLink). POWER8 was harder to find, but the performance characteristics were good (in my experiments). And now the software is better (more solid ports) and cost is down. I'm eagerly awaiting the chips to hit this year and trying to find access to some.

Plus IBM licenses the design these days (through OpenPOWER), so several of these players are building their own chips, boards, etc. Someone could inevitably enter the low-end market but lower-end devices have thinner margins and a lot more competitors.

Even right now you can get real chips to go on a board on pre-order, just under $400 (TALOS II preorders) -- the mobo is the pricier part, but part of that is likely due to the BOM choices on that piece from Raptor Engineering. A smaller form factor motherboard (maybe with 1 socket) could land in the sub $2000 range for a whole mobo+cpu -- which is certainly competitive with similar HEDT/workstation prices[1]...

[1] I just dropped $1,500 on a 1950X threadripper and associated mobo earlier this year, so this price range is certainly alive and kicking, IMHO.

The mobo+single quad core CPU is $2,475, that's way more than the threadripper setup and the threadripper has 4x the core count.

Threadripper also only has 8MB of L3 cache per CCX, and you can't hold more than 8MB of data. (true, there are 4x CCX per chip, but you can't "combine" the datasets. You're effectively limited to 8MB per 4-cores. To communicate cross-CCX requires an expensive ping to main-memory) Its a great architecture, but cross-CCX communication is relatively slow and definitely is a concern for some work problems.

Power9 has 120MB of L3 cache that's shared between all cores. Which means you can ACTUALLY have a full 120MB-sized problem set and share all that information between cores.

In short: Threadripper is great for sure, but its not necessarily a fair comparison. In an apples-to-apples comparison (ie: Monero Mining), it seems like a Power9 server is 3x better than Threadripper (Power9 gets ~3000 hash/sec, while Threadripper is roughly 1000 hash/sec).

https://www.phoronix.com/scan.php?page=news_item&px=POWER9-C...

> Using the xmr-stak-power PPC64LE-focused Monero miner, they are seeing great performance with it running on dual pre-production 16-core POWER9 processors. There's a hash rate of 2945H/s while this POWER9 system is pulling 350 Watts DC power.

And mind you: Monero / Cryptonight only uses 2MB of L3 per core. So that's practically Threadripper's ideal problem. Imagine if you actually had a dataset that was larger than the 8MB per CCX that Threadripper is limited to.

Not to hate on Threadripper at all. Its cheap and high performance. I'm seriously considering a Threadripper system myself. But these Power9 specs are incredible, and I'd definitely like to test one if I could afford one.

I really hope AMD releases a 7601 variant that is clocked similarly to the threadripper chips.

That's a dual socket system with 16 cores per socket. Dual socket Epyc 7351 (also 16 core per socket) is faster and more power efficient than that. 3200H/s at 332W.

https://www.youtube.com/watch?time_continue=226&v=CS7M392Ia_...

I'm sure there are workloads that really do benefit from the huge unified L3, but it seems like a pretty small niche.

The cores are not directly comparable. The Power chip has significantly more hyper threads (and the resources to back them).

For those not in the know:

Each Power9 is consisting of either 2-super slices or 4-super slices, depending on which Power9 you get. (Corresponding to 4x SMT or 8x SMT respectively).

Threadripper has 4x integer pipelines, 4x floating point pipelines, and 2x load/store units (called AGUs by AMD) and supports 2-threads (aka: 2x SMT).

So Threadripper definitely is "broad", but Power9 is "broader". The 8x SMT Power9 can perform 8x loads / stores per cycle per core, while AMD's Threadripper can only perform 2x loads/stores per cycle per core.

The last presentation on Power at Hotchips left me with the impression that "Core" in IBM terms is 9/10ths marketing speak and whatever Power might have that their engineers might think of as "cores" probably has little to do with what their marketing people are telling their customers and software vendors who do their licensing "per core".

Could you expand a bit on this? Are you saying they are inflating their core count or the reverse? Everything I've looked at indicates significantly more powerful cores.

They "inflated" it from 12 to 24 but recently it looks like they deflated it back to 12 cores. However you count it, a full Power9 chip is competitive with a 24-core Xeon or 32-core Epyc.

A full POWER9 chip should be compared with a 48-core Xeon.

In regards to cores, the distinction is between the PowerVM and Linux (OpenPOWER/PowerNV) ecosystem variants. Both are made to process 96 threads, but the difference is in whether the thread processing units are grouped eight to a core (SMT8) or four to a core (SMT4).

The PowerVM version, some say for licensing reasons, gets the SMT8 cores. Either way, you get 96 threads:

12 cores * SMT8 = 96 threads 24 cores * SMT4 = 96 threads

Sorry but I just noticed this reply. I don't want to claim that IBM is "inflating" their core count, nor the inverse. Rather that what passes for what we might think of as a "core" based on the ways software vendors describe things isn't directly transferable to what IBM has actually built.

IBM has extremely flexible hardware which, using IBM's hypervisor, can be configured on the fly to present itself in a number of different ways. For example: few powerful compute units, many medium compute units, or very many less capable compute units.

So right up front the idea (which a lot of people might assume without really thinking about) that a given chip has a fixed number of "cores" doesn't really hold true. More to the point the next question that presents itself "what makes up for a core and what determines how many a given processor has" is best answered with "it depends" (at least as far as IBM Power goes).

One of the main reasons this matters is that all sorts of businesses use the idea of "cores" as a fixed entity as part of their pricing structures... and now it's all muddied.

Reply on news.ycombinator.com