Sourcehut and Codeberg are both currently experiencing a DDoS attack
fosstodon.org
fosstodon.org
In one of the previous ones, they said there was high suspicion it related to 'hatred against a transgender developer in the company' or something like that. Like, who the hell targets a "company" because of an employee who works there you don't like ...
So in this case, that really means that it’s not controlled directly (or indirectly) by a FAANG.
Is your beef with people making money for their work? Or with Megacorps? Because sourcehut isn't one, by a longshot.
A corporation is just one method by which people coordinate their activities, not some separate entity that has its own independent existence or intentions.
You can certainly draw distinctions between the consistency, reliability, and scalability of some people's way of organizing their activities vs. other people's, but that doesn't distinguish the underlying nature of what they are, just how good they are at organizing.
I concluded he was right after I read about ongoing problems with the GMP project, at https://gmplib.org/ : "The GMP servers has been under serious load due to a barrage of clone requests from Microsoft/Github. ... In effect, Microsoft's computer cloud performed a DDoS attack on the GMP servers. ... Our servers are fully available again, but that's the result of us adding all participating Microsoft network ranges to our firewall. We understand that we are far from the first project to take such measures against Github."
Also, you can't build projects like Sourcehut by being only an antagonist, and I can think of some pretty antagonistic CEOs of billion dollar companies.
Too many. The internet hivemob is dangerous. Hell here in Germany we have a pretty nasty cyberbullying case around former Youtuber "Drachenlord" - his hater mob goes as far as to terrorize hotels where he's stayed (he lost his home as a consequence, it's utterly utterly insane).
We can't tell one way or another from outside in this case, but don't discount this as too unlikely.
It's a lot of resource to expend for a personal vendetta. Transgender people work in a lot of places, my next question would be "why this one?".
My guess would be that there is something that pays for this financially. It could be that it's a specific repository on Sourcehut that hosts a competitor, for example.
Of course it's all speculation.
Trans has become the new focus chosen out group of the culture war and we’ve had to deal with this fascist ideology shit for a while now.
Here’s a great real would descent into the insanity of these people, fueled by the pandemic.
After reading it, there’s little doubt that this bullshit is infectious https://www.reddit.com/r/singapore/comments/p6ovxr/an_indept...
I don't think such a comparison is useful here. That said, there is a clear difference between group and individually focussed hate. For some reason Sourcehut is clearly being singled out.
> Trans has become the new focus chosen out group of the culture war and we’ve had to deal with this fascist ideology shit for a while now.
I don't think it is a strictly fascist ideology at all. I see people on both the left and right with a range of opinions on this subject. You'll find the majority of people are resistant to any form of change, whether rightfully or wrongly.
> After reading it, there’s little doubt that this bullshit is infectious https://www.reddit.com/r/singapore/comments/p6ovxr/an_indept...
It's difficult to know who is the insane one there, the person who is committed or the person who dives so deeply into another person's life.
I think it would generally be a mistake to assume that this attack is based on the gender of a single employee at Sourcehut. To me it would be just as likely to be something they host, or something they refuse to host.
I've seen plenty of witch hunts of the sort the past years in tech, including attacking or boycotting the company due to an individual (usually due to them having conservative views).
That said, I'm inclined to think this is a form of "pay us or we keep on ddos-ing".
I wonder if something like the BitTorrent protocol could be used to keep a repo accessible without directing all the traffic to one site. Take a magnet link to a particular repo and commit, receive the complete branch eventually. Now serve these commits to other peers. That would be rather harder to DDoS or censor.
(This, of course, ignores other aspects of forges: CI/CD, releases, packages, etc. Issues and even code reviews can be branches, too; such tools exist, and e.g. Fossil has them built in.)
Is that about right? It actually sounds cool :-). I guess the next issue I see is that torrents are not mutable (I think there are some specs maybe, but it's not common/implemented? Not sure).
To do it in a decentralized fashion you could simply have a DNS pool with all of the known replicas and rely on round robin DNS (not great but better than a single point of failure)
Phabricator implements an interesting multi-master replication scheme with git that essentially keeps track of which node has the newest commit for a given HEAD, then edge nodes can automatically proxy incoming requests to the node with the latest change, or a random node if they are all in sync already.
The repos had the same hook and it just fanned out, unattended. Pretty neat, if you ask me.
Using the new HTTPS DNS record¹ you could simply list all the servers in the DNS.
1. RFC 9460; discussion here a few months ago: <https://news.ycombinator.com/item?id=38420555>
The main issue is that each commit would become a separate torrent, so you'd need some abstraction on top to clone a whole repository (you'd need to download all the torrent-commits).
OTOH, when updating a repository you'd only pull new commits (just like other transports). I've toyed with the idea in the past, but it's a substantial amount of work.
You still have the problem of incentivizing hackers. In the Github world, hacking the author of a major project is bad but not disastrous, somebody is probably going to notice, ping the Github security team, and the changes will get reversed in a matter of hours. Hacking a major project on radicle or similar means that the project is now out of its authors hands, potentially forever, and there's nothing you can do, short of informing every single consumer and asking them to change the project ID they depend on. You can give a trusted team a way to override ownership in such cases, but then these people are also subject to being hacked or compelled by a court. Requiring multiple signatures and best practices around hardware wallets is probably the best you can do realistically, and that dramatically increases complexity and expense on the maintainers' side compared to traditional Git workflows.
In all seriousness though, the reason why services end up under DDoS attacks are wide and sometimes complex.
There is a likelihood that someone is simply upset at these services, or that there was a coordinated effort against source control programs more generally but its only succeeding against these two (that we're aware of), but not say, GitLab / GitHub.
Another example: they may be targeted simply due to traffic and volume, and this is an attempt to "dry run" DDoS techniques that may be used against bigger sites down the road too.
There's alot of room for why these things happen. I've learned that motivations and execution is very wide.
Of course the efficacy of attacking a service with lots of content is questionable, but the plan is to put on so much pressure that the service removes the content.
https://www.theguardian.com/technology/2024/jan/11/ebay-fine...
tightens tinfoil hat
Same state actors might do the same here
I implemented this as a .NET lib (https://bvulpes.net/tarpit-a-proof-of-work-http-ratelimiting...) for obscure reasons, but an NGINX integration would really be ideal.
https://en.wikipedia.org/wiki/Hashcash
Uh. Ignore the "Cash" in the name. That's mostly a coincidence.
Human basal metabolic rate is around 100W. Actual energy use per capita averages up to 10,000W, depending on how wealthy and boreal you are. If you want to count "energy waste", a successful DDOS costs each blocked user multiple hours, at multiple kilowatts.
It's no more a waste of energy than using advanced video encodings, or running `bcrypt` on passwords. The energy cost is comparatively miniscule, and it solves or mitigates other problems.
You could make it less annoying by borrow from TeamSpeak and/or hCaptcha. On TS3, as a server admin, you could set a minimum [pow-style] security-level to prevent spam/flooding. If server had a higher than default number, clients were prompted to improve their security-level or disconnect. And alternatively, as a client, you could pre-emptively generate higher/highest security-level (up to "30"?) over night or whenever you had spare cycles.
You could borrow that concept to allow non-JS clients to generate extra-strong tokens to allow them to bypass the system entirely. Or combinee that concept with what hCaptcha does for their disabled users -- once you solve the special disability captcha [or in this case, generate a stronger token], rather than inflict extra pain on the users, you get granted ~20 tokens or so to allow you to avoid the normal captcha system for a decent period of time.
If you want to go the extra mile and account for no-cookie use in addition to no-JS, allow it to be sent in the auth header or something.
Anyway, I don't know how many things I've seen use the concept, but IMO, it's under-utilized.
As a kid, I was super into reverse engineering RuneScape and writing our own MMORPG servers based on the protocol. Most of us had no idea what we were doing or what the games industry was doing, and we were all mostly kids, so we all had to be our own developers, db admins, sys admins, etc, and DDoS mitigation was effectively non-existent until OVH democratized it (BlackLotus would charge like $550 a month for a mediocre dedicated server, Staminus was similar, AwkNet was cheaper but only did TCP so TeamSpeak etc was still vulnerable, etc - and god knows how much Akamai and similar would've wanted).
One of the billion issues people ran into often was just login flooding; if it was a new username, that'd mean forcing the server to go through the new account and serialization process, and if it were an existing account, it A. opened up bruteforcing but B. forced the server to, at the very least, do password hashing, possibly database queries, or commonly reading a character file from disk, etc.
Rate-limiting by IP isn't super effective, because proxies. Rate-limiting / straight-up banning certain ASNs worked better to deal with VPNs, but you can't exactly grind all of Comcast or Verizion to a halt without disrupting users. And rate-limiting the particular account sucks, because you run into situations where individual players get targeted and effectively locked out of their account because someone is griefing them.
Far more effective than anything else was the idea of just generating some sort of PoW token on login that took a bit of time for the client to generate but was quick for the server to verify. Significantly better was the idea of being able to scale the difficulty globally, per IP/block, and importantly per account -- nearly no difficulty if no flooding is occurring, ~1-3 second logins if a massive flood is ongoing, or a per-account ~1-10 second difficulty if someone is trying to bruteforce a particular account. Even if the latter is mildly annoying, it's better than being totally blocked from playing.
RuneScape itself implemented this a couple of years ago too, I believe. I haven't seen them actually do much with the difficulty factor yet but even their super naive simple version seems to have helped a ton.
[of course this doesn't fix the issue of giant IoT botnets and stuff, but most people didn't (don't?) actually bother -- it was almost always DDoS amplification for volume and one or two servers with proxies doing the layer-7 floods]
If you have deep enough pockets, sure, just pay anyone ASAP. But as far as I can tell, Codeberg is a non-profit entirely dependent on donations and, according to Drew DeVault, paying Cloudflare to protect SourceHut is prohibitively expensive even at his scale, so I doubt Codeberg is in any better position to finance this.
> More than 100.000 projects as of late on Codeberg. Codeberg e.V. has 401 members in total, these are 281 members with active voting rights, 113 supporting members and 7 honorary members.
[0] Joining: https://web.archive.org/web/20240106140505/https://join.code...
[1] Donating: https://web.archive.org/web/20240106140512/https://docs.code...
Afaik you can also do wire transfers in other countries but there are some fees involved, so isn't IBAN available to anyone with any bank account if they're okay seeing fees which the payment platform would otherwise hide?
https://en.wikipedia.org/wiki/International_Bank_Account_Num...
https://en.wikipedia.org/wiki/Single_Euro_Payments_Area
Only European bank accounts can do SEPA transfers.
Europe is many nations, not one.
SEPA is the Eurozone area that treats all transfers as domestic - usually free or near-free, and fast (max 1 day normally, max 10 seconds for instant transfer).
IBAN is just a numbering scheme to uniquely identify bank accounts worldwide. It has nothing to do with transfers. The closest thing would be SWIFT, though that's just an information protocol and says nothing about fees or times.
IBAN is also used by some countries outside of EU. It's basically just a way to assign an alphanumeric code to an account so that it also includes a country code, bank code, and checksum.
Also when getting money from Japan (mtgox), I had to put in the bank's branch address and some funny details one doesn't usually need, but it's possible to receive it on my IBAN account
.
According to SWIFT, the account is defined by a combination of bank identifier and account identifier. That is, account identifiers are supposed to be unique only within the bank. (Not sure about this part.)
There are many ways to identify an account, IBAN is one of them. Other methods require you to specify a "schema" (one of predefined codes 1-4 characters long), an "issuer" (string up to 35 chars), and "id" (string up to 34 chars).
I suppose banks know the values that make sense for their customers, and otherwise just forward the information to other banks.
IBAN is an alphanumeric string, where the first two characters are country code, and the meaning of the rest is country-specific. BIC is encoded in IBAN, so the information provided is redundant (in the payment you specify both the bank and the account, but the account also specifies the bank).
.
SEPA is a subset of SWIFT, with extra rules imposed. The currency is EUR, and the only allowed account identifier is IBAN.
That means, SEPA requires IBAN, but you can also use IBAN outside of SEPA as one of many possible ways of identifying the account. For example, if I pay from EU to USA, I am using my IBAN account in a non-SEPA payment.
(I can certainly see where you're coming from, I just wanted to provide some additional insight.)
They are just using sloppy language. SEPA is just a payment system for cheaper and faster payments. It uses still uses regular bank accounts to send from and to, it can be regarded as backwards compatible with normal bank transfers.
Instead of finding ways to isolate and blackhole the offending networks - granted that refining this data into small enough ranges would be quite difficult - they instead punish the targets, who are already suffering to begin with. This is mobster mentality.
You can bet that if random ISPs ranges would get blackholed when there are missbehaving clients, they would instantly come up with ways to detect and mitigate excessive traffic, net neutrality be damned.
This information can trickle down from the tier one that presently just blackholes the victim.
Nobody says that infrastructure that was designed in the late eighties can't be improved upon with additional functionality, namely passing top down information about missbehaving network ranges and or IPs.
As an ISP you already see devices on your network sending traffic in unusual patters, and when receive the warning that they are part of a DDoS you shut them down. If not, your whole range goes down, etc.
Let's add a fourth to the three-napkin protocol.
mystery solved: in the coming days github will announce obligatory 2fa linked to a phone number.
some Microsoft exec is betting his job that people will not just abandon github if they turn the heat on too hard on the slow boiled frog that is open source still hosted there.
to offset that theoretical exec fears that open source projects will jump ship, they spent the equivalent of one expensive dinner on ddos for hire to take the top competitors offline.
this is not some coordinate evil plan at Microsoft, just something that someone with even as little as M3@microsoft money and very little cryptocoins can easily do in under an hour.
This is what I've never understood about Cloudflare.
You're suppose to be able to purchase a business account for $200/mo [1]
But it seems like at some point, Cloudflare says you need to upgrade to a custom pricing plan.
What triggers Cloudflare to state you can't use the $200 Business Plan account?
[0] https://fosstodon.org/@arch@floofy.tech/111739294821803544
iirc one reason is needing proxying of non-HTTP protocols.
Oh? You want that attack to stop? It's going to be big number.
...we'd have to charge you for causing to much traffic to your server
> It's $200/mo!
>> Ok, but the other quotes we're receiving for $requirements are 10x that, are you sure?
> Yep, sure, it's $200/mo* (until it isn't)
>> Right, so we might pay more than that, what would the pricing be for $requirements?
> $200/mo*
>> Ok, cool, just to confirm, if we put $requirements of traffic through this service, we will only be charged $200/mo?
> Wellllllll, that might work out to more like $4000/mo.
Compared to all the competitors who would happily take our $requirements, stick them in some internal calculator, and give us a real figure.
1) Sales didn't know WTF they were doing. Incorrect info we had to waste time sorting out, confused when we asked them about a use case for which they had an entire-ass prominent landing page with a top level link on the home page.
2) They gave no fucks about our mostly-CDN needs. All their enterprise plans were oriented around selling us a bucket of other crap that we didn't need. We weren't able to get them under $4kish no matter what, and even at $5k there wasn't enough pre-allocated transfer to keep us from hitting metered billing. They were not interested in talking about adjusting the price down by removing all that stuff, or almost any other customization. We just needed to sling bits, and a few features directly related to that. The result was that they were not at all price competitive with our other options.
3) They were the least-hungry sales folks I've ever interacted with. It was a weird experience. They didn't act like we were annoying them or anything, they just acted like they were totally unmotivated to make a sale. Every question, no matter how simple, seemed to give them pause, and set them to musing as if they'd never even considered it. Never seen anything like it. Maybe mid-five-figures a year is too small-fish for them and we were talking to the not-even-actually-sales-folks screeners who keep the actual salespeople from wasting their time?
My guess (based solely on that experience) was that their sales organization had crazy turnover (even worse than a typical sales org), awful training/mentoring, and possibly not-great hiring. But IDK.
We did end up using the $200 "self-serve" plan because it's 1) the only self-serve plan with any kind of SLA at all, and 2) actually a pretty fuckin' good deal for the things it provides. They weren't even in our top-3 for our heavier CDN needs, though.
Like... if you can just go there and self-signup and move your DNS, etc... then you're going to be just fine in their standard plans.
There's a lot of big-orgs that think they need special stuff when they in fact do not.
> "we called [Cloudflare] and they quoted a very big number"
Why did they think they needed to even call Cloudflare or speak with a sales person at all? That was the first mis-step... their Business Plan ($200 monthly) says "unmetered DDoS Protection"...
I think specifically for Sourcehut, to survive the DDoS they likely don't need 100% feature parity. Right now you can't even load their webpage - the service is effectively entirely down and users are left guessing what's going on. Getting web services back online would be significant.
The point for them would be to get breathing room to figure out a better long term plan. When the immediacy is gone, they'll be in a much better negotiating position with Cloudflare (or alternatives) for non-standard features.
sr.ht does not support pushing to git repositories over HTTPS with a username+password — SSH keys are mandatory.
HTTP access is restricted to read-only operations (clone, pull, etc).
[0] https://web.archive.org/web/20231225152845/https://man.sr.ht...
It appears they restored HTTP services now, so perhaps they ended up taking that route. Their homepage lists details of the ongoing attack/outage and their current efforts. This is what should have been done immediately, I'm afraid to say.
Once the attack passes, they will then be in a better position to consider their options and negotiate with providers for long term solutions.
To be honest such approaches are not ideal anyways. It would most likely require a GRE tunnel with the DDoS washing service, increasing latency amongst other potential issues.
It's probably way easier (and possibly cheaper) to buy DDoS protection from your ISP/transit provider, assuming they're not so small that they don't have it themselves.