Comment by schleck8 | Hacker News Reader

Comment by schleck8 | Hacker News Reader

The HTTP traffic from Yandex is down to ~ 0 % after the third strike, after the first two strikes already degraded it. I wonder how this looks like at the network level because their search page still serves, but presumably their crawler is inactive now

Edit: Here is some more info, they do have availability issues as is to be expected

> Because three of its primary facilities were knocked offline within four days, Yandex’s failover mechanisms broke down[1][8].

> Outages have cascaded across Yandex Go (taxis), Yandex Pay, food delivery, smart home platforms, internal logistics, and third-party corporate services hosted on Yandex Cloud[2][7]. Problems have spilled into neighboring countries using these systems (Belarus, Kazakhstan, Armenia, Serbia)[2][7].

Edit 2: Here is the Yandex Cloud status page, their GCP/AWS equivalent. The entire compute core is offline, and even worse, Cloud Backup

https://status.yandex.cloud/ru/dashboard

> ru Compute Cloud, ru Kubernetes, ru PostgreSQL, ru ClickHouse, ru MySQL, ru YDB, ru Kafka, ru Application Load Balancer

Interestingly, Yandex tells its users on the status page to switch to competitors Selectel, K2Cloud, and VK Cloud. Which will probably be hit next

This will be a case study in catastrophic events for HA and Ops folks.-

I don't think there is much you can do about being targeted by a military, simply because of how unique the attack vector is.

Hacking is smart but happens through software, so if you have hardware (i.e. physical) barriers, that stops the hacking.

Natural disasters happen physically but they are dumb, so if you have software redundancy (i.e. by copying the data to multiple locations), that stops them from destroying everything.

A military is both physical AND smart. They can target multiple locations at once and destroy whole buildings.

You can learn to quickly diversify availability (or at least data safety) beyond the region of conflict when a kinetic war breaks out. The moment “kinetic sanctions” became the strategy, the risk to data centres should have been self-evident. And if it wasn’t immediately obvious, it should have been blindingly so when targeting expanded from oil refineries to Wildberries warehouses.

I don‘t think you can build a new data center in such a short time frame that passed since attacks on WB, definitely not when you can source hardware only from the black market.

I’m not describing what Yandex should do, but rather what their customers should do. Whether that means diversifying to eastern Russia, or China, or Belarus, I don’t know what options exist.

... and in the middle of the RAM-pocalypse, no less.-

Wildberries warehouses were filled with flammable items boxed up in cardboard, packed for maximum efficiency and no wasted space. They were tinderboxes.

A datacenter is filled with server racks that wouldn’t burn as readily as cardboard. The fire videos I see of this datacenter don’t look as good as the Wildberries videos.

That said, I wish the Russian people good luck with booking their taxis and Yandex employees luck with finding RAM.

> I don't think there is much you can do about being targeted by a military, simply because of how unique the attack vector is.

You could choose not to invade other countries.

I guess, but that doesn't sound like something in the control of a single individual company.

They control their own individual contribution.

Is it just me, or didn't we just see a single company cause a bit of a dustup when they refused to provide certain services to a military that wanted to bomb another country?

If it doesn't sound like something in the control of a single individual company, perhaps you have something distorting your hearing.

This (as with defense) is kind of outside Google or Yandex's purview.

Said every company serving fascists ever.

No, you can choose to collaborate or you can shut it down.

Collaborators don't get to point back in time and say 'oh we had no choice', even if the choice was a painful one. Sorry it does not work that way.

A very naive take from someone who clearly lives in a place where people have real choices, economic freedom, freedom of movement, and a government that protects them. "Just shut it down, bro" risks destroying your livelihood and leaving your family financially stranded, and yourself incarcerated or sent to the war zone. And it's not like Yandex develops weapons of mass destruction, you probably assume it's a collaboration even if Yandex simply hosts the Defense Ministry's websites. No sane parent will put their family at risk for this kind of stuff. "I'm going against the government for abstract ethical reasons" is something very few people/countries can afford.

And you can't shut it down anyway. The government will just nationalize it and keep it running. As long as you're the one in charge, you can at least throw a few hidden obstacles in the way, like dragging out legal processes.

No, a pretty informed take from someone who had family members that once upon a time made some very tough choices because they thought it was the right thing to do.

You can pretend at being ethical and moral as long as there is no price to pay, but when the time comes that it matters there will always be a price tag. Whether you are prepared to pay or not is what defines whether you are ethical or just pretending.

Company execs make decisions, employees make decisions, everybody carries a part.

Everything I don't like is fascist. Bed time is fascist. Blah blah. This is an incredibly naive take.

There's better definitions to use to identify fascism.

"At the core of fascism is loyalty to tribe, ethnic identity, religion, tradition, or, in a word, nation." - Jason Stanley

Also, a tendency toward or actual exercise of strong autocratic or dictatorial control.

Yandex is a part of Russia's military. Just the same as Google, MS, Amazon etc. provide services for the US military. Google et al. are probably in the critical path of defence for most European countries as well, insane though that is.

Not defending Yandex here, but I think their military involvement might be as accidental as of an American farmer whose beef landed in the cantina of aircraft carrier. Government and military procurement in Russia is its own universe, with strong players having specific IT expertise and capable of building DCs and tech. Assuming that everywhere in the world it’s the same as in America is wrong. Consumer tech companies aren’t operating this way usually, it is the scale of American Big Tech that allows them to compete with traditional contractors.

In general it’s the price of working with government, also see AWS Bahrain. I wonder if their insurance actually pays out because those are not officially wars there so hopefully they could recover some what from the insurance.

AWS plays both sides in that conflict since they're also a supplier for Israel. Presumably the aggressive side got them more profit than the defensive side cost them.

I don't think that qualifies as playing both sides.

I feel like this isn't even a particularly new thing. I worked at a medium-sized SaaS company and we did disaster recovery drills for moving our stack from eu-west-1 to eu-central-1 in case something catastrophic happened in Ireland. If you want HA, you kinda have to assume that any single country in the world can have some kind of a crippling disaster. Unfortunately for Yandex's clients, Yandex only has regions in two countries, and not that many regions either.

I don't know how useful it will be as a case study since an HA/Ops person in (let's say, the US, Canada, AU, western european countries) has access to do things like buy hypervisor capacity or colocation in just about any location. I could go buy colo in Brussels or Panama City or San Diego or Edmonton or Melbourne tomorrow by contacting the right people.

The Russians are much more restricted from doing anything geographically outside of their own borders (for very well justified sanctions reasons), so their limitations are not the same as an HA person in almost any other country. Maybe some Iranian HA/Ops people will be comparing notes with them.

Sometimes there exist data or processing residence requirements that bind you to a specific region (e.g. EU-only). Also, „all AZs in this region no longer exist“ is not on everyone‘s bingo card, this is a good reminder why multi-AZ deployment in the same region may be not enough.

For data center guys this is definitely a case study, because drone attacks are probably on the threat list for everyone now, for various reasons, including domestic terrorism and hybrid warfare. If you want to defend your investment, time to think what covers you from the air.

It's an open question at what point we'll start seeing small fiesty startups that want to sell you a privately owned air defense system for your datacenter roof. The tech to do it is probably far ahead of the regulatory regime to own and operate autonomous small radar+SAM batteries.

It will require some involvement of state, but it is plausible, that critical infrastructure will be required to have this covered and certain big, licensed private contractors may be able to operate air defense systems on such sites (do you smell the money and lobbying effort?).

Also this may influence the design of such data centers. A drone that a paramilitary group can build is not very sophisticated and won’t carry big payloads. Put an outer concrete shell or some better passive defense, and maybe that will be enough.

As a result of some of my work only slightly related to telecom/internet infrastructure, I happened to get on the marketing lists for about six different Chinese companies that will gladly sell you a full size Shahed-136 clone complete with 2 stroke petrol motor. I wouldn't say that small paramilitaries being only able to fly a 5-10kg max size quadcopter with a munition on it into a datacenter is the limit of possible future capabilities. These things are cheaply made in fiberglass molds and look to be dead simple. They're already showing demo videos of them flying around with very low cost ardupilot capable flight controllers.

They even advertise the airframe and motor packages on Instagram....

Well, what an interesting time we are living in…

In a scenario were data centres in China, the US or France are getting blown up you're in WW3 and your food delivery service no longer matters.

Just co-located datacenter next to a military base

Just run your app on three fast servers in your office.

Yes, spend 10-100x more on 24x7 IT support for it, fiber from two different providers and take care of physical security of the server room. And when there‘s fire, you still need a cloud backup.

> privately owned air defense system for your datacenter roof

Already here: https://vishkipvo.ru/

It heavily depends on "what kind of DC". A lot of current DCs are not really well protected against a lot of known threats - and that's just a design tradeoff.

High value data is already protected a bit better (or at least should be - the usual trend, management gets lazy when nothing blows up). I'm old enough to have sat in the meetings for "so, apparently a plane crashing into one or more of our locations is a viable risk now" - and a DC hardened for "somebody tries to land a 747 on that thing" should also survive a small drone with a bit of explosives.

While for the younger ones the "we lose several availability zones within days" is a new thing now - it really isn't. We were planning for that back then as well, just that we didn't call it "availability zone" or "cloud" yet.

The DR plan for pros, "I could go buy colo tomorrow by contacting the right people"

It was clearly meant as an example of the ease of a non-russian company buying geographically-distributed services in the global marketplace, not "I could go buy this with a few docusign signatures and SWIFT wire transfers tomorrow". I am not in the market for colo in Panama City or Melbourne. I don't think it's a bad thing that the russians are hampered by many limitations but whatever lessons their HA people are learning these days are probably very specific to a sanctioned Russia.

Unironically yes, if you're just a typical B2B SaaS who doesn't have "datacenter getting hit by a literal missile" in your threat model, this coupled with offsite backups is probably the logical DR plan for that scenario.

I think the complete destruction of the AWS Bahrain region got there first.

> The entire compute core is offline, and even worse, Cloud Backup

Their Kazakh regions are available according to the dashboard, so it isn't as catastrophic.

Their managed DBs and data processing services are heavily affected though with no Kazakh redundancies from the looks of it.

The KZ region wasn't intended for Russia's entire workload, that's a secondary region with better Central Asian access.

If what you deduct were correct, Yandex would not be telling its customers to switch to competitora like VK Cloud. They even have a "priority onboarding code" called "CloudBro" for three competitors.

I just searched "blyat" and it worked, so?

Given that blyad is just punctuation at this point, I am not surprised.

The username checks out!

("Ept" and "suka" are two other swearwords, that are also used much like punctuation.)

Get with the times, gramps

I predict a great many more videos coming out of Russia of Firepoint UAVs flying into refineries and such, punctuated by background audio commentary that can be summed up as "cyka blyat"

Reply on news.ycombinator.com