Hacker News Reader

Later in that thread he says he's currently in sales, but used to be a "self-taught infrastructure engineer". If you look at the site that he links from his Twitter profile you'll see a half-finished site made with a site-generator. Just the type of talent you want running the infra for the company you just paid $44B for.

What kind of frontend skills are you expecting from an infrastructure engineer, self-taught or not?

It's about the optics of some rando, with a half broken site that he's sharing publicly, claiming he'll "come up tonight" to help run one of the biggest sites on the web. Using a landing page creator like ConvertKit shouldn't require any frontend skills.

...Ability to change title of the page from "Home - made with softr.io" to his name ?

Guy can't even use website generator

To be honest, this sounds like bloody red meat to any black or grey hats out there interested in having some infiltration fun. Why not show up at Twitter’s hq with a mouth full of smart sounding engineering babble and a backpack full of spycraft toys?

Yep, Elon's paranoid about his current employees sabotaging him, but he should be just as paranoid about new job applicants looking to gain access and capitalize on the internal chaos.

Not gonna lie, if my current contract was ending id take a stint at twitter for a while just to be a fly on the wall for the insanity even knowing id get the arse at any time.

I'm somewhat surprised that nobody bypassed the internal systems to tweet stuff using Musk's account. (Or maybe somebody has and we just can't distinguish?)

Hahaha. Even better, the twitterer specializes in digital detoxes (a worthy cause, but counter to Twitter's goals). Maybe his secret plan is to turn down services in a volunteer capacity

I'm appaled at how ignorant you are

At the end of the day, most software eats the same stuff and the levers are often rudimentary and grokable, especially if you focus on a specific area. I don't think someone could meaningfully implement a new feature in a night, but could they tackle a problem in a given service and identity a rough remediation to save the next 72h? Yes, this stuff happens all the time. I've jumped on outages like this and figured it out as I went. You're not going to build a new feature but it's probably possible to get oriented enough to avoid catastrophy.

People want it all to fail out of spite, and attempt to bring their dreams to life by merely doomsdaying repeatedly and with fervor. We had 4y of Trump and no amount of wishful doomsdaying actually changed anything. The media was full of drama and we told ourselves stories of incredible disfunctions, but at the end of the day the government kept on doing its thing, the country mostly didn't change too much on the fundamentals. Reality is most often more boring than that (save FTX-like events).

That’s very naive. We’re talking about a distributed system with thousands of servers, with code from people who might not even work there anymore. Yeah, maybe if a server gets a high disk usage alert because of runaway logs you might fix it… but maybe that fix causes another problem, or maybe the alert is in a system that you don’t even know its purpose. Maybe 72h is just what it takes to you understand the internal tooling and how to do any action there.

> code from people who might not even work there anymore

I think that's most of the code, at this point

Basic firefighting often revolves around the same set of problems. It's not that naive. Tell me I'm wrong? I work with systems like this. Realistically most problems that show up by themselves (ie not caused by bad code) often have very common mitigations/remedies. There's surely some super complex interaction going on but bugs showing up of this complex nature are usually due to code changes, or scaling events that reach a tipping point. Odds are those aren't the bread and butter of problems faced by Twitter _right now_.

> Realistically most problems that show up by themselves (ie not caused by bad code)

Well that goes the first assumption.

> or scaling events that reach a tipping point

Such as having applications running without restarts for a long time due to deploy freezes?

> Odds are those aren't the bread and butter of problems faced by Twitter _right now_.

Right now, exactly this moment, probably not. But if those were the only worrying events that could happen in the short term, I doubt anyone would need to offer any help to Twitter.

> due to code changes

Well, Musk has said that he wants to do "hardcore engineering" implying there will be a lot of changes coming up. That surely can't go wrong.

The World Cup is starting Sunday. That usually would be an all hands on decks event at least ops wise, but there are very few hands left and the door to the deck is closed anyway...

In most cases that's enough to keep it running, even if you need to learn 100x of that to start making changes.

But if infrastructure wasn't managed like shit it will probably just keep on chugging

Dude just getting access / logins you need, client side certs for auth, MFA that’s required by certain systems, SSH keys would take time and be arduous during a regular hire, can you imagine what this would look like trying to show up and have it all in a night for a sev1/p1, when so many (who would manage that access, as others have pointed out in other threads) have been let go or left already? Hell, who even manages IAM or PKI at Twitter at this point?

If the problems are "this link is saturated" or "we're out of compute capacity" and other run of the mill stuff, I think getting access, being able to measure, identify and effect change is indeed the tough part.

The problems themselves most often have simple short term remedies.

I really think people blow out of proportion the actual requirements to keep things running, probably because they wish to see him fail. But everywhere I've worked, systems were most stable when engineers weren't there to muck with them. There's a reason many people refuse to ship on Friday.

Are you going to be able to do it under the conditions of massive brain drain? Better hope the engineering and SRE teams had up to date SOPs that work well. Oh, and you’re new. So that means you need to probably start getting access granted to services on the fly first to even see dashboards. But it sounds like that team has also seen massive attrition…

Are you sure you can be effective coming completely cold into a company with massive attrition?

I think Twitter is becoming a place for high risk reward employment. Maybe it crashes and burns, maybe you quickly ascend the ranks in the turmoil and reach a point in your career that would have taken substantially longer in a more traditional route. If Twitter does something like what Tesla did, you might even get filthy rich. Or, hey, maybe it doesn't work out and you get nothing but an interesting story, Twitter collapses or Elon fires you or whatever.

Many companies say that they are like a startup, but I can see Twitter going back to that style. Far fewer people, far faster shipping, less thinking things through, etc. The bluechecks for anyone in a week and undoing it seems very startup like to me. Is that good? Probably not - but it is interesting, and, like I said, high risk high reward.

> high risk reward employment.

I honestly have no idea what you're talking about. Has there been literally *any* indication that this "hardcore" attitude will result in more reward?

You don't think you would have an opportunity to achieve a higher position at Twitter (and more money either there or by switching into an adjacent company) faster than at a FAANG? And the other indication you have, apart from common sense, is Tesla. If you had got and held on to a lot of Tesla stock you would have been rewarded. No guarantees obviously, but if Twitter could be as successful - same thing could happen there. It's a possibility.

Wouldn't it be more profitable to just make Twitter over again at a new company?

Yes, but more risk. In five to ten years I think there is more of a chance for Twitter to be worth 500 billion dollars than a random startup trying to recreate Twitter.

No.

People here on HN drastically underestimate the value of Twitter’s network and name. Musk could start 1,000 social media companies for 1,000 lifetimes and still not hit on Twitter’s timing and execution early on.

The age of social media is dead. No one is creating a Twitter competitor in 2022+. Either the site completely dies and people flee to existing platforms, or it lives on.

The idea that one can just build a Twitter clone shows little understanding of human nature.

>The age of social media is dead. No one is creating a Twitter competitor in 2022+

What happened in 2022 that all the doors are suddenly closed? Wouldn't you have said the same in 2016, before TikTok came out of nowhere and became the most popular website (per Cloudflare). I'm pretty sure people spoke similarly in 2006 about how Facebook has no chance against Myspace. The only constant is change.

The social graph is extremely valuable, but the amount of debt it’s in is extremely not-valuable, so I’m not sure how much these two cancel out.

You might be able to acquire the company from Elon for cheap in a bit Tumblr-style though.

Elon literally fired 50% of Twitter's staff in order to save on expenses. He's not going to suddenly turn around and dectuple the salary of the remnants.

What reward lol

What, is Twitter going to re-IPO?

Is that before or after paying off the billions in debt?

As an ordinary employee what is the potential reward?

Yes? That's what he has said he plans to do, take it public again. And, of course, they have to deal with the debt. Twitter might not be able to, but they might. That's the high risk part - you might burn out, get fired, or the company might collapse.

A turn around is clearly possible. Social media companies have been tremendously valuable in recent history and might become so again. Musk has had amazing financial success with other companies.

As an employee your potential rewards are what I've already explicitly called out. You can advance further and faster up the corporate hierarchy in the tumult. You can also earn and hold stock rewards that may be extremely valuable. And, of course, you may consider it intrinsically valuable to help shape a tool used by hundreds of millions worldwide, and, as the number of employees shrinks, the per-employee impact increases.

> That's what he has said he plans to do, take it public again.

For a mere hobby it is too expensive even for him. 44bn (+continuing operating costs, credit costs, ...) also is a lot to regain via profits. So the logical thing is to either sell it to some other corporation (who!?) or going public. But for that he has to somehow make it an attractive investment, while currently he is doing the opposite.

Twitter is becoming a place for high risk employment.

The reward is deathmarch with Elon holding the whip.

> As if you could just drive over to the Twitter HQ, grok how all the infrastructure works, and make/provide meaningful contributions/tweaks/upkeep.... tonight

The infra is just too clever for anybody to grok in a million years. These clowns probably don't even have fedoras, no way they'll ever figure it out.

Ever been on a team where a key person responsible for a piece of infra leaves suddenly, and next incident takes hours to resolve instead of minutes as everyone is running around trying to understand what the code is doing? It will be like that but way worse.

Yes, I have actually. Let it take hours to resolve, what of it?

Way better than keeping a person around who hoards institutional knowledge and admin access for job security. In fact that would be the first person I'd zero in on and get rid of, and have in the past. Not only are people like that not indispensable, they should never have been hired, anywhere.

A well engineered twitter has everything necessary to run it in properly documented internal wikis and can be handed off to a small new group of competent ops people at the drop of a hat. Ideally new hires should be able to fix and change things within a day of starting their jobs with little to no hand holding. I'm sure it is very far from that at the moment due to the unprofessionalism of everybody involved, but this is actually the best and fastest way to get it there purely from an ops perspective - usually takes years to convince management to yank off the bandaid because everybody is too scared to take responsibility.

Twitter used to have fail whales all the goddamn time, sky doesn't fall, it isn't amazon they aren't losing money every second it is down.

Whoever stayed on will be able to figure out eventually which things to restart in what order so long as they aren't locked out of the servers.

Unless Twitter is still using Zookeeper. :P

In the process of figuring it out they'll be putting in-writing what should have been documented all along and getting a feel for the brittle bits. Long term twitter is better off.

Good riddance to bad rubbish.

I'm sure the databases have recent known good backups at worst you'll get a bunch out of prolonged outages and some dataloss - since we are talking about tweets, nothing of value was lost.

> at worst you'll get a bunch out of prolonged outages and some dataloss

I hadn’t thought about what will happen next, but this is probably it. It’s fascinating watching the D-K effect play out at every level here.

It might actually benefit engineers in the long term too, since they can ultimately blame Elon ( after he fires them ) and hopefully learn from it.

Domain expertise does not imply 'hoarding institutional knowledge.'

The random spaghetti of deployment scripts, the right JVM parameters that got derived from trial and error over time so things don't OOM, and what format the git tag should be for the CI branch are problems of ones own creation.

Operating a Kafka cluster doesn't make one an expert at anything. Zilch.

The site is still up and in a month or two Elon should cut another 90% of them and it will still be up.

Some people will have to live down the discovery that they were never essential workers but rather seat fillers.

https://en.wikipedia.org/wiki/Seat_filler