chronid

Please no! When I last worked with physical servers we were replacing tpms all the time...

Some kind of cost shedding to the application owner (in many enterprises this is not the infra owner) is definitely needed otherwise everything becomes critical.

"Everything is critical" should sound a million alarm bells in the minds of "enterprise architects" but most I've discussed this with are blissfully unaware.

I do, and I'm not a security researcher (not really).

If nothing else, it's fun to see who pokes you, even if I don't actually follow up on it.

I think there's a bit of "they could" but also something that is considered very little in many contexts unless you have experienced the contrary: integration is costly and integrating properly sometimes is more work than doing something from "scratch", so you don't do it and then you have a mess that hurts you in the long run.

The same reason cloud is all the rage. It has its uses, hype is behind it and the marketing ignores its caveats.

The main issue I have with it is simple: in my experience you cannot really debug some classes of issues ("we had intermittent issues calling endpoint A from function B") and they kinda push on you a certain type of organization model so if it doesn't reflect your organization model (for example, you have centralized infra teams) prepare to be screwed.

At an higher level, it massively encourages atomization of systems, which makes them essentially spaghetti with a network you cannot debug as the sauce. Good luck, hope your company pays for honeycomb.

In infrastructure we build dashboards. And then we build systems to tell us something wrong, and when something is wrong we go and look at the dashboard.

Looking at dashboards all day to spot issues is a (pointless) nightmare.

This goes on every day even in our field, from small things (like infra being blamed for everything at first so the application teams can go home) to the biggest involving criminal investigation: if you have nothing to lose from not doing it and everything to lose from doing it, why care?

I am certain someone raised a hand at a certain point and was told to shut up and/or let go and/or his/her contract not renewed.

The amount of not giving a fuck is staggering in this world (and IIRC a primary source of enshittification), particularly when a billion contractors are involved...

The japanese rights holders for the logo and the japanese company (?) making it at least disagree I think.

The name and logo if nothing else are pretty recognizable and memorable. I know nothing about features, but they seem to be collaborating with institutions too...

The company behind it is called gehirn to add more fun to the story. :)

My 2c: you can get things right, but most of the time you won't, for many reasons - technical, logistical, cultural or merely political. Sometimes you don't control these reasons.

So you are now left with managing risk. It's trade-offs all the way down.

Spoiler: it won't.

I never worked too closely with TPMs, just looked at the code in a previous employer because it was adjacent to mine and helped tprefactor some of it. Settings them up was kinda nightmarish considering all the failure modes, and things were not getting much better over time and with new hardware.

I have been fighting this in my current position, with some success with some teams and far less success with others (who are now fighting with the distributed monolith they have created). Ironically the ex-faanmg folks are all pro-monolith.

I think most people fail to realize microservices are what you get from an iterative process. You pull out of the monolith the things you need to scale eventually.

Microservices need plenty of basic infrastructure that you don't want to mantain unless you need to. Most companies don't have an adequate platform to support the pattern anyway.

Definitely have multiple replicas, and recreate misbehaving ones, saving logs and data for later analysis.

If you can't have that (and budget allows) keep unhealthy replicas alive and pull them off the load balancers.

In my experience option one works best, and makes option two redundant. Unknown unknowns are a thing, but that's true even when the unhealthy replicas are kept around.

You need better observability or tooling then.

Operation teams (and automation) have usually a primary mandate of availability above all, not to investigate any possible failure.

I am in that condition, so personal experience here:

- i do most meetings with internal stakeholders to understand needs and changes, and document them

- i do most "architecting", as i have most context on the systems my team is responsible for (also I am the oncall escalation point all the time as I am the senior person)

- i write about 10% of the code, usually what other colleagues aren't comfortable with (legacy systems mainly these days)

"Development" is not only writing code, and senior roles tend to code far less because we get wrapped up with the rest of the company so other folks can code and deliver. My team, manager and customers seem very happy about the whole thing (but hey, maybe everyone hates me!). I do want to code more, maybe with a good PM i would be able to, but I have no illusions about that.

Multi-AZ is a requirement on production level loads if you cannot sustain prolonged downtime.

Datacenters do end up completely dying now and then, you really want to have a good strategy in that case. Or not, if that's not required.

Building your own datacenter is hard (think the logistics for just keeping the lights on).

Even if you don't own/manage the building but simply buy the hardware (and spares, and DC ops) from HP/Dell/EMC and friends things can become suprisingly even harder, because it leads to a rabbit hole of and tons of badly implemented ticketing systems and badly baked processes you don't really complain, which open the path for the "stale IT" developers complain about. Offloading that further to more consulting companies leads to an even bigger mess of staleness and sluggish movement/execution, with incredible lead times to expand the capacity for services and getting it for new projects.

Unless you have very good processes (I've never seen a company with those), iron clad contracts (your CTO chose the consulting company because he is playing golf in the weekend with the consulting company CEO. Oh sorry you don't even have a CTO, that was your CEO instead) and plenty (relative to company size) of good people in the right positions to drive things (no, you won't have those) managing your datacenter is massively harder compared to just using the cloud.

Then there are the benefit in financial sense (capex vs opex, opex always wins).

> Teams never seem to understand how to alert on stuff. Ive been paged for things going off, that might indicate a problem, then you get stuck sticking around because someone else wants to just wait and see what happens. "We should just be cautious" Its impossible to push back on these things, your just going against someones gut feeling, like maybe one day we will want to know, and everyone needs to protect them selves.

From an ops person: if an alert does not have: - clear, provable impact on customers (internal/external) - clear documentation (e.g. runbooks) on how to solve it

It should not be an alert. I took this path (successfully) when trying to remove spurious alerts that existed only for the ego of someone (most absurd example, something that started complaining when p99 for some endpoints went >500ms and happened every day when we downscaled the ASGs because business hours were over. No clear path to resolution, and impact was a couple pages opened a bit slower sure - but the number of customers using those pages after hours was <1%!

It sucks, definitely, but the best way to go around those alerts is to prove they're pointless or a waste of time or can be automated around and should automated around (and I've seen so many servlets leaking memory triggering OS alerts for OS teams or spawning infinite threads and never cleaning up after themselves...).

If the company does not want to do it, and pushes back, I would recommend starting to look for another company. It's sad, but it is what it is. 99.99% of software does not need a follow the sun rotation (or people damned to night shifts), just a bit of thought about what happens when things fail.

I would add a "there is time reserved to write automation to reduce toil".

I have a fair bit of experience in teams with developers hating oncall and the critical issues happen in two camps generally:

- In some cases the org was absolutely open to give them time to resolve issues and automate stuff away (PMs were proposing months for fixes only, and cleaning up boards, etc), there just was no interest until enough escalation happened (creating a massive conflict between ops and devs in the meantime). Even offering to do the work was met with a "stay in your lane" kind of response and no collaboration at all.

- In others the company just did not care, deprioritized tickets until things blew up completely, then finger pointing started and all that nice toxic bullshit.

Note those are anycast addresses, my guess is the DNS server gives out addresses for FB names pointing your traffic to the POP the DNS server is part of.

If the POP is not able to connect to the rest of Facebook's network, the POP stops announcing itself as available and that DNS and part of the network goes away so your traffic can go somewhere else.

The EU commission is voted in by the EU governments. The EU council is made by the EU governments. EU citizens elect MEPs.

There is no accountability because no one holds them to account and people repeat what you just wrote like gospel, including people writing for newspapers - and that has to stop.

You can pressure the EU as much as you can pressure your national government, all you need to do is put the pressure on your national government. Add your MEP on top, but the national governments are enough if you don't want to go the extra step.

My interpretation is that it's advocating for counting the thing that matter, not the consequence (e.g., the "SEV" event).

The problem is your availability being <X%, your API responding >Xs Y% of the time, not "we had X SEVs last half". People will gravitate to numbers that are talked about (because they at least appear important), so you should try to talk about the right ones.

It's hard enough to convince junior folks to open incidents without them feeling like they will mess up a very-important-management-eyes-on-it metric if they do...

> Is that not what the person who originally posted the content being moderated is doing?

The person that posted the content does not expect a random person from the internet to see a video of his children because it was accidentally flagged by an AI as child porn. :)

> IIRC, Netflix OSS published some tools quite some time ago to support multiple DNS providers but I don't know/remember if they tackled the availability problem.

The classic way of doing this is AXFR (your own DNS server is a "hidden master" and the DNS providers are the slaves).

The problem is you won't be able to have redundancy at the registrar level, but that has historically at least been less of an issue.

> If not, and you own the physical capacity yourself, wouldn't you do away with CloudFlare entirely?

Cost could be an issue. We had something similar (not in the same context) in a company I worked for before. We could shift traffic, but that would cost 2-3x more, so it was not the preferred path unless we had problems.

It surprises me that many (big) companies did not learn the lesson already. We had a similar thing happening already years ago with dyn in 2016 (https://en.wikipedia.org/wiki/2016_Dyn_cyberattack), and it was surprising how many companies relied on a single DNS provider.

Fun fact, this has been an issue since 2011:

https://support.google.com/calendar/forum/AAAAd3GaXpEE7zPvtA...

Why not asking them during the interview process?

Have them "drive" you through the resolution of one of the issues that you expect could happen. See how they behave and what they try - they may not necessarily have the complete correct answer, but you will see if they know where to look, the basic concepts, and so on.

I've interviewed people capable of writing ansible scripts, but with no real ability to dive into issues and fix them and/or investigate what's broken. Many would not even tell me "we could just provision a new VM to replace the one that went down, right?"...

Google also has all that and now and then their network explodes anyway when they do configuration changes. :)

Certain configurations at a big enough scale are dangerous, just because you could hit a terrible corner case when you rolled out the change on 50% capacity, and lose all of it so fast that your magic automatic rollback is pointless because your infrastructure is burning.

> It wasn’t Trump winning that did it, it was that Trump won with the assistance of data that Facebook was unable to properly protect.

I agree that Facebook is certainly a convenient scapegoat to hide all the other issues that plagued the Clinton campaign (from the DNC scandal, to the unlikeable candidate, through some controversial phrases she said during the campaign itself, and so on). Also it helps newspapers since they have to take no blame for what happened (and do no real analysis on why democrats lost).

I'm pretty convinced that if Clinton had won - even if Clinton had used CA data to do so - there would be far (far) less media coverage about the "evil of Facebook", regardless of the impact of their incompetence in the election. That was the case with Obama already after all.

In 2008 and 2012 FB got praised for being the platform that Obama used to win the elections - I remember multiple articles about it (and someone from the Obama campaign later admitted that they did the same thing that got the CA scandal started). In that period of time there were also multiple newspapers writing that "privacy is dead, and it's a good thing". There was not a lot of discussion about privacy when they bought whatsapp and instagram too, IIRC.

Frankly newspapers really changed idea only after Trump won the elections.

I find somewhat fun that some of the recent news about facebook being evil actually talk a lot about that period (ex: the recent news about the "friendly fraud" class action) and some are the consequences of the (naive, as we a lot of us were at the time - me included!) choices facebook made back them (wanting to be a platform, without really understanding what it meant) before pivoting to the current model (which is AFAIK far more similar to google's).