577 karma · joined April 17, 2013
This would be an abysmal decision all around. Firefox doesn't have enough market share to command any sort of price for this, and selling ad-blocker access would immediately destroy all trust in Firefox as a product.
Engineers are incentivized to increase profits for the company because impact is how they get promoted. They will often pursue this to the detriment of other people (see: prioritizing anger in algorithmic feeds).
Doing Bad Things with AI is an unbounded liability problem for a company, and it's not the sort of problem that Karen from HR can reason about. It is in the best interest of the company to have people who can 1) reason about the effects of AI and 2) are empowered to make changes that limit the company's liability.
The groups of people who criticize real tangible languages and people who have the skills to critique the design of abstract language design don't necessarily overlap.
There's so much about the language of Shakespeare that needs to be vocalized and heard to understand. It took me way too long to realize that not all of his iambic pentameter was "correct' rhythmically, and that those variances _meant_ something. Watching and hearing these actors work through their scenes (and sometimes having their work adjusted live!) was truly eye-opening as to how the language of Shakespeare, arcane though it is, was deliberate, purposeful and useful.
- vendor lock-in: anyone who has worked at a shop running Sun SPARC machines when they got purchased by Oracle can speak to the pain involved with negotiating software licenses or hardware support contracts with the Only Game In Town.
- the price/scarcity of mainframe talent: you're going to have to pry IBM z-series experts away from banks who are paying 50-100% over market rate, oftentimes in straight cash, to maintain systems that are propping up the United States economy in its entirety. Not to mention - my first job out of college >10 years ago had a mainframe, and I was incredulous that _anyone_ still had or needed one in the 21st century. Now I can appreciate the specific tradeoffs being made that caused the business to choose a mainframe, but attracting top-tier cost-effective junior dev talent out of college becomes several orders of magnitude more difficult once the word "mainframe" leaves your recruiters' lips.
- scalability: in the event that you ever decide to add features or functionality (or, say, increase your tweet character limit by an order of magnitude), you have now committed yourself to scaling your systems in units of mainframes costing millions per unit, as opposed to servers costing five figures per unit (not to mention, you probably need a dev environment that's airgapped from your prod environment, which means yet _another_ mainframe...)
- build vs. buy: using the same commodity x86_64/ARM hardware and Linux kernel that everyone else is using allows you to take advantage of all of the open-source datacenter software being built for that happy-path profile. The minute you stray from that path, the engineering-hour cost of everything you do has the potential to skyrocket, because you can't use anything off-the-shelf and need to recompile everything for z/Architecture. In fact, based on some cursory web searches, it doesn't appear that you can compile the Rust toolchain to even _run_ on z/OS as of today, so at minimum, OP would be committing to implementing that.
But at the end of the day, the constraining resource in every software organization I've encountered has been engineering hours, and by choosing a mainframe you're drastically limiting the potential number of engineering hours available to you in the employee market.
There was a HPC cluster at Princeton when I worked there (which, looking at their website, has since been retired) that was assembled by SGI and outfitted with a customized Linux unikernel that presented itself as a single OS image, despite being comprised several disparate racks of individual 2-4u servers. You might be able to metaphorically duct-tape enough machines together with a similar technique to be able to run the author's pared-down scope within a single OS image.
With respect to the IBM z-series specifically - if the goal of the exercise is to save money on hardware costs, I'm imagining purchasing an IBM mainframe is in direct opposition to that goal. :) I'm not familiar enough with its capabilities to say one way or the other.
> There’s a bunch of other basic features of Twitter like user timelines, DMs, likes and replies to a tweet, which I’m not investigating because I’m guessing they won’t be the bottlenecks.
Each of these can, in fact, become their own bottlenecks. Likes in particular are tricky because they change the nature of the tweet struct (at least in the manner OP has implemented it) from WORM to write-many, read-many, and once you do that, locking (even with futexes or fast atomics) becomes the constraining performance factor. Even with atomic increment instructions and a multi-threaded process model, many concurrent requests for the same piece of mutable data will begin to resemble serial accesses - and while your threads are waiting for their turn to increment the like counter by 1, traffic is piling up behind them in your network queues, which causes your throughput to plummet and your latency to skyrocket.
OP also overly focuses on throughput in his benchmarks, IMO. I'd be interested to see the p50/p99 latency of the requests graphed against throughput - as you approach the throughput limit of an RPC system, average and tail latency begin to increase sharply. Clients are going to have timeout thresholds, and if you can't serve the vast majority of traffic in under that threshold consistently (while accounting for the traffic patterns of viral tweets I mentioned above) then you're going to create your own thundering herd - except you won't have other machines to offload the traffic to.
Source on this?
Anecdotally, the reaction of everyone I know working in law enforcement to the Musk purchase and subsequent layoffs was "this is going to make it infinitely more difficult to get CSAM taken down from Twitter".
Including an anecdote from finding a radioactive rock in the school in 1999:
"But when the teacher moved to an unremarkable, slate-gray, grapefruit-sized rock, the Geiger counter erupted like an alarm clock, sending the tiny gauge on the device to whiz to the highest reading levels, Gallo said."
(My hometown, but did not attend the high school, AMA)
Holding specific people "accountable" for outages doesn't incentivize reducing outages; it incentivizes not getting caught for having caused the outage.
As a result, post-mortems turn into finger-pointing games instead of finding and resolving the root cause of the issue, which costs the company more money in the long run when a political scapegoat is found but the actual bug in the code is not.
I will say, this comment section is the first time I'm hearing about "bare-metal" meaning "without an OS", but the above question is genuine curiosity.