Hacker News Reader

Comment by froh | Hacker News Reader

yup. error detection in the CPU (Alu, Decoder, etc) traps into the hypervisor which fires up a spare core, transfers the registers over to it and calls the technician.

technician calls you to tell you "your machine needs maintenance"

and at application level there was a short "mouse hang" latency for this hardware failure.

and of course that works cross-site, too, hot moving running software across data centres at the hypervisor level, because you want one data centre downtown, one at the airport.

but it's not just for finance. some mainframe customers have virtualized thousands of "your personal domain and web server running php/perl/nodejs" on a single physical machine which shares a r/o mounted /usr, in ram, transparently.

customer X needs more ram? click click life assign more ram. needs more cpu? activate another core. the Linux virtualization Features to support hit cpu and ram swap were driven by the mainframe guys from IBM, years ago.

View on HN

What would prevent, in your opinion, to make some changes to current commodity hardware architectures (say, x64, arm...) to allow for mainframe-like scalability and reliability ?

As per the article, how is the jitted/interpreted world of languages so bad ?

ECC bring the norm and better error handling all around. OSes that better handle memory and uptime. Possibly something like a user session being accessible from allowed terminals and applications that call for that. Think hotel front desk staff or anyone public facing who move from desk to desk.

That ECC isn’t entirely standard is shameful.

There's far more to reliability than ECC. What if a processor/CPU goes bad in an x86 server while running? You have to take the server down, replace it, and bring it back up. IBM mainframes allow for processor replacements while it's running. I think it's the same with memory. IIRC, in the mainframe world IBM has processor and RAM packaged together as a 'book' and it may be the book that can be hot swapped. Hopefully someone more knowledgable can clarify.

A mainframe reminds me more of a cluster of specialised machines networked over ridiculously fast interconnects and purpose-specific networks that host those specialised machines.

Some functionality of those specialised machines is exposed to user programs that can delegate some processing to these smart peripherals, offloading things like allocating blocks in the filesystem and letting the boxes that run your used code focus on running it without ever being disturbed with servicing network or disk interrupts.

patents, tongue in cheek...

on a more serious note yagni: for most workloads you simply don't need the dependability and Xeon has caught up sufficiently for most workloads.

however if your business depends on less than 5 minutes of unplanned downtime per year you need stacks designed for faul operational subsystems, i.e. subsystems which don't just detect faults and halt, but which gracefully handle many faults with a short degradation but no failure.

and that's not only hard to master it's also plastered with IBM patents.

wrt the jit/interpreted world my personal take is that silicon vendors are very happy about it. very few organisations feel the pain to an extent they innovate things like golang... and once they are in the business of selling compute power (as cloud service provider), there goes their motivation to create a sexy cobol.

I'd say cost (of research, implementation and resulting product ).

The additional hardware both in silicon and on PCB is massive to enable these kinds of reliability numbers. I played with some higher end HP servers back in the day, which allowed hot-swapping RAM carriers to maximize uptime. The construction on mechanical, electrical and software was much more sophisticated to enable these features. Which made the systems bigger, somewhat noisier and harder to maintain in a sense.

When everybody is moving towards more cattle-like systems, and the behemoths like HPC doesn't care about reliability in the worker nodes (we just re-schedule the job, no biggie), that kind of reliability is not wanted by the people, in most sectors, and this low demand doesn't make the capabilities worthy to pursue.

Also, there are system dynamics which affect operations profoundly. Your normal server boots in 5 minutes tops from power to "all services are ready to serve". Your run off the mill SMP server needs 15 minutes. A mainframe is never expected to be shut down, and if it's ever turned off, it's at least an hour required to spin that thing back up.

We're talking about machines which have small "guns" inside to physically sever power lines in an emergency (sorry I can't find the story I've read about moving a datacenter hence turning of the IBM mainframes, however a related reference can be found at [0]).

The JIT/Interpreted languages of the day were really bad. Java was heavy, other languages were slow. Today's JIT/Interpreted languages are fast enough, because the VMs running these have been optimized heavily, and the processors allows these VMs to run at much faster speeds because of available instructions, available memory and system interconnect speed at every level and interface. So the evolution of JIT and interpreted languages is a matter of chicken and egg. Hardware enabled both faster speeds and technology to write better VMs and optimize them further.

[0]: http://www.catb.org/jargon/html/B/Big-Red-Switch.html

It sounds very impressive until the entire mainframe wont boot up after a simple routine upgrade as happened recently at my place of work.

That’s more of a training / staffing issue than a tech stack issue. I’ve seen similar outages at my clients in distributed systems. A lot of mainframe staff has been outsourced / offshored and those providers suffer constant turnover because the money and job prospects in distributed systems is generally better.

Mainframe jobs pay well at the very apex of skills, but if you’re coming up through the ranks in an outsource provider you’re a cost not profit center. The compensation package and work life balance are good if you are with a company that keeps everyone in house, trains in house, have a deep bench of expertise who maintain the systems in an orderly fashion, and have a continuous succession plan for staff. Super low drama, and they keep the financial lights on in the house by processing the bulk of profit-generators’ needs. These tend to be in places like stodgy COBOL shops the Cool Kids would rather gnaw their feet off to avoid, though.

The part many people miss when comparing mainframes and distributed is you get all that mainframe magic with the application programmers accommodating the architecture little if at all. But your system programmers (system admins in Linux) have to be top tier.

You can accomplish most if not all of that same magic in distributed architectures with parallelizable workloads. But your application and devops staff have to be top tier.

> That’s more of a training / staffing issue than a tech stack issue.

I mean sure, but the same could be said for distributed systems. What breeds reliability is qualified personnel working on a system that they understand. Slapping an IBM logo on the front doesn't make something more reliable, and stacking proprietary technology that "transparently" moves subsystems between datacenters only increases the training burden.

Mainframes became a victim of their own moat. While proprietary .magic. can hold people in, those very same technologies also keep new people out.

It sounds impressive until a whole region or whole cloud goes down because of bug in the control plane.

> some mainframe customers have virtualized thousands of "your personal domain and web server running php/perl/nodejs" on a single physical machine which shares a r/o mounted /usr, in ram, transparently.

For general public use? Can you name one?

yes for general public use

page 15 in this white paper from 2012 is the closest I can go in terms of naming. there were more , in continental Europe.

https://www.slideshare.net/hcl/effective-web-hostingonzlinux...

IBM was paying them to do it. The problem is you’re running in a separate set of processors… it’s almost like a blade chassis at this point. The cost structure didn’t work as if you’re doing anything significant you’re going to end up licensing more CPU time (yes, you need to buy access to the stuff you bought), and it cheaper to buy x86 and run Linux OR just rent VMs from Amazon.

I had to do a POC of this for an IBM app in 2008-9, as it was seen as a way to save on IBM software licensing. I killed the notion after spending 6 months getting stuff to work (ie the IBM boys and inwere the first idiots to try) when i borrowed from surplus PCs awaiting shredding and replicated the entire environment on 6 circa 2005 pentium whatever desktops (which I did during the week between Christmas and New Years). My little lab outperformed the mainframe by a factor of 5 for the task.

If these solutions were sold commercially, there must have been some German compliance requirement that led people to do it. Nobody would ever do this without getting directly compensated for “choosing” to do so.

what works financially is providing a fleet of simple similar independent machines.

think "your personal wordpress instance w/ own email and ftp server", centrally maintained, identical baseline but individual content.

to this day Linux KVM does not enable multi tenancy. with z/VM you can provide that, and you still have the shared /usr in physically shared ram in z/vm.

the physical CPUs are also virtualized and shared, each "full own wordpress server" only needs slivers of that cpu pool.

it was done and it qorkwd. and it was profitable for both the hosters and IBM.

Yes - that was very interesting one of the drivers for what I was working on was that the z/VM segmentation would allow for partitioning of workloads to avoid over-licensing and address some security needs.

The tough part was if you didn’t have that perfect scaling opportunity, things fell down fast, and the marginal cost for more resources was way too much. IBM tried to oversell it.

> some mainframe customers have virtualized thousands of "your personal domain and web server running php/perl/nodejs”

Are there hosting providers offering this type of service?

IBM Cloud offers Z systems: https://www.ibm.com/community/z-and-cloud/ I imagine they're absurdly expensive like the real hardware.

Only available for existing Z/OS sites via your organization rep.

that was around 2010/2012, eg page 15 here

https://www.slideshare.net/hcl/effective-web-hostingonzlinux...

however virtualization on arm and Intel has meanwhile matured enough to replace that business model.

> but it's not just for finance. some mainframe customers have virtualized thousands of "your personal domain and web server running php/perl/nodejs

To be fair, shared hosting providers did this on x86 server machines on a scale of 100s of customers in the 90s, and I'm sure a modern revival on 2022 could do 1000s also

We have a 10 machine Proxmox cluster at the office, and no. It doesn’t lend itself to such capacities.

Maybe we over-provision systems, and can fit that number of VMs, but what will be the performance?

You can use a proper SMP machine with 16 sockets, but you are already at half mainframe price and journey-wise, and you’ll be wasting the machine’s capabilities.

Sure, but they way we did that back in the 90’s was very different, and carried a much lower overhead than LPARs in the IBM mainframe world or any hypervisor or container used today — shared hosting “virtualization” in the 90’s was little more than chroot jails, and frequently not even that.

I think you're confusing LPARS with guests under z/VM. A mainframe LPAR is a coarse-grained (relatively speaking) partitioning of the system. LPARS are created and managed by a Hardware Management Console (HMC), and run on PR/SM [0]. No one creates mainframe LPARS to run a VPS. Those would simply be a guest under z/VM. Realistically speaking, z/VM would be running in an LPAR (less than the whole of the mainframe resources) and then within z/VM's LPAR, z/VM hypervisor allows all of its guests to run concurrently.

[0] https://wiki.alquds.edu/?query=PR/SM

KVM is also available now in addition to z/VM. The IBM cloud “trusted environments” run on it.

>> ... or any hypervisor or container used today ...

> I think you're confusing LPARS with guests under z/VM.

I am not, but thanks.

Reply on news.ycombinator.com