dotwaffle

There's an RFC for Peg DHCP! :D https://datatracker.ietf.org/doc/html/rfc2322

While it's an April 1st RFC, I've seen this implemented at a Hackspace in Amsterdam, works very well!

I'd have thought that by now, most would have been swapping to WebAssembly. It's really nicely sandboxed, you expose it to only what you want, and you can compile a lot of languages into a WASM form meaning you're not stuck with only Javascript or similar. Am I naive for thinking that?

I have done extensive research on CDC and it almost never works out because most utilities don't create compressed archives in an "rsyncable" (rsync does CDC) format, I actually saved a lot of storage using restic when I switched my backups of certain things so that files were stored in archives uncompressed, and sorted in a stable order. I know syncthing eventually removed CDC and just went with constant-size block sizes.

Bazel, on the other hand, is completely in control of this, and it makes perfect sense to do this at that point -- and it seems to be a relatively efficient implementation too, really nice to see!

If you're the author of https://github.com/hanwen/go-fuse/ -- thank you :D

As someone who predominantly writes in Go, cider-v was a massive step backwards compared to cider. I eventually moved entirely over to using vim (with the set of internal plugins for blaze etc) which became so much more useful, but I still missed the features of a proper IDE that cider just excelled at.

I imagine a lot of it came from that push to "use outside world tools more rather than writing our own" which is great in theory, but really felt like a huge leap backwards in terms of convergence.

> flow-crushing remote desktop latency

Yeah, I was working out of the Sydney office. Almost everything was incredibly slow due to that latency, not just chromoting but also just accessing most sites through beyondcorp.

/me shudders. cider-v...

Cider (and p4/g4c etc) was amazing when I left back in 2020, I loved it so much, and truly miss it. I rejoined Google last year, and they'd replaced it with a VSCode clone that truly was just a glorified text editor and most were all-in on mercurial as a piper/citc shim -- I was only there for 5 months before I decided not to stay, and I never managed to get Go type definition hints working.

That's what Locally Recoverable Codes (including Pyramid Codes) are designed to address: you can repair a missing block using only a subset of the blocks, which you can make sure are placed in just one zone, eliminating the cross-az bandwidth requirements. Sure, if you lose multiple blocks, you are going to end up needing that extra bandwidth, but the chances of having two blocks offline at once is very low -- if it's just for a failure rather than extended maintenance, then you have probably already repaired the first block by the time the second fails.

In fact, the RS configuration is often in the >50 data blocks and >10 parity blocks range (albeit with an LRC/nested RS config) for object stores because it's more important to have that recoverability than repair efficiency. While one large provider I worked at did have a system whereby they did effectively have two copies of the RS-encoded data (so that 130% turned into 260%) across two AZs, they were actively in the process of swapping to the blocks being evenly distributed across the AZs, near-halving the total required disk space.

As I said before, most object storage is not on SSD, it's on hard disk: it's 20% of the price per TB, and most objects are read very infrequently. I can promise you that they're not paying $200 for 1TB of SSD either... I realise prices are higher than sensible in the last 6 months, but it was fairly easy to pick up SSDs for under $50/TB at retail pricing (and hard disks for under $10/TB) only a year ago.

GCS / Azure can survive a total AZ failure too, think of 3 x Replicas as RAID-1, whereas Erasure Coding is more like RAID-6. Only it's actually more resilient:

Let's say you have 10 data blocks, and you have 4 parity blocks. You can now lose 4 servers containing a block and still be able to repair the data, whereas in 3 x Replica you can only lose 2, and have to store everything 300% of size, instead of only 140%.

And yes, it is unreasonable how much they charge for both storage and inter-az bandwidth.

Is it 3x redundancy forever? I always just kinda assumed it was RS encoded after a while, so only 30-50% larger than a single copy. Plus, almost all object storage is written to / read from hard disks, not to SSDs. Unless they're in a caching layer that is.

I know Azure has done a bunch of work around Pyramid Codes (essentially a locally repairable EC/RS variant), and Google obviously have the Colossus infrastructure that allows variable encodings, I'd be surprised if AWS is still triple-replicated everywhere.

Insanely high S3 storage charges too. $23/TB/month? Even with the insane HDD pricing that we see today, that's paying off a drive in 1 month (at retail) that will last for 50-100 months. Sure, there's probably some encoding overhead, but it's still mad.

> Just create separate AWS accounts for separate services

My understanding is that different AWS accounts have different mappings of availability zones, so it's very easy to suddenly find yourself with an unexpected bandwidth bill due to all the cross-az traffic.

I've been irritated at AWS (and the other large cloud providers) that they charge $0.01/GB for cross-az traffic. That's $3.24/Mbps -- about the same I was paying for internet transit (as in: from London to anywhere in the world) 20 years ago, and this is just between two datacenters in the same city controlled by the same organisation, markup must be 10,000x or more considering these places are cross-connected with massive bundles of fiber!

Commits or pushes? Commits aren't really a worthwhile source of measurement in terms of load.

Someone came up with a good theory a while ago that I'm inclined to believe: The social media companies (esp. Meta as I understand it) were looking at huge fines for showing adult content to under-18s, so they lobbied hard to ensure that the burden of proof for age verification was on anyone else but themselves, hence why the OS vendors are being targeted now.

Ultimately, they seem to have realised that they can't stop adult content from being shared, so the easiest way to get there was to mark anything even vaguely possible of being adult, and require age verification -- which comes with a lot of political cover vs. just deleting it.

Of course, if you stoke up the right people, you end up with lots of support from the puritanical brigades, and label all naysayers as putting children in harm's way.

That's the point, though. An SSH key gives authentication, not authorization. Generally a certificate is a key signed by some other mutually trusted authority, which SSH explicitly tried to avoid.

I've never quite understood why there couldn't be a standardised "reverse" HTTP connection, from server to load balancer, over which connections are balanced. Standardised so that some kind of health signalling could be present for easy/safe draining of connections.

... fair point.

They're 47 inches long. Amazon (UK) has 48 inch long zip ties for $14.45 (pack of 12), 60 inch long for $18. Not quite as thick or wide, sure... But that's not what was in the headline :P

> An extra 131 GB of bandwidth per download would have cost Steam several million dollars over the last two years

Nah, not even close. Let's guess and say there were about 15 million copies sold. 15M * 131GB is about 2M TB (2000 PB / 2 EB). At 30% mean utilisation, a 100Gb/s port will do 10 PB in a month, and at most IXPs that costs $2000-$3000/month. That makes it about $400k in bandwidth charges (I imagine 90%+ is peered or hosted inside ISPs, not via transit), and you could quite easily build a server that would push 100Gb/s of static objects for under $10k a pop.

It would surprise me if the total additional costs were over $1M, considering they already have their own CDN setup. One of the big cloud vendors would charge $100M just for the bandwidth, let alone the infrastructure to serve it, based on some quick calculation I've done (probably incorrectly) -- though interestingly, HN's fave non-cloud vendor Hetzner would only charge $2M :P

I started rewriting gcsfuse using https://github.com/hanwen/go-fuse instead of https://github.com/jacobsa/fuse and found it rock-solid. FUSE has come a long way in the last few years, including things like passthrough.

Honestly, I'd give FUSE a second chance, you'd be surprised at how useful it can be -- after all, it's literally running in userland so you don't need to do anything funky with privileges. However, if I starting afresh on a similar project I'd probably be looking at using 9p2000.L instead.

I know quite a few AFS systems that moved to AuriStor's YFS: https://www.auristor.com/openafs/migrate-to-auristor/auristo...

As I understand it, it mitigated many of those issues, but is still very "90s" in operation.

I've been flirting with the idea of writing a replacement for years, about time I had a go at it!

I know a lot of people who use it, in fact I'm one of them.

I have an @gmail.com account with about 20 years of stuff associated with it, from purchases to YouTube subscriptions, from calendars to GCP accounts.

However, I use a vanity email (me@somedomain.example) that everyone I know uses to get hold of me. Until about 10 years ago I could just forward emails but that slowly became unworkable as more and more stuff just broke due to SPF etc. So, I've been using POP pickup (and accepting the 5-30 minute delay) ever since.

As I understand it, I can't move all my gmail.com data into a GWork profile easily, and POP has worked for years. This is very frustrating.

Interesting... It's been ~15 years since I last used BitTorrent personally, and I had asked a friend before replying and they swore that all their traffic was TCP -- though perhaps that may be due to CGNAT or something similar causing that fallback scenario you describe.

Thanks for the info, and sorry for jumping to a conclusion! Though my original point stands: Residential ISPs are generally not built to handle BitTorrent traffic flows (customer to customer or customer to other-ISP-customer across large geographic areas) so the bursty nature would cause congestion much easier, and BitTorrent itself isn't really made for these kinds of scenarios where content changes on a daily basis. CDNs exist for a reason, even if they're not readily available at reasonable prices for projects like OP!

... What? You realise BitTorrent runs over TCP/IP right?

My example was to show locally what happens -- the ISP does not have control over how many connections you make. I'm saying that if you have X TCP connections for HTTP and 100X TCP connections for BitTorrent, the HTTP connections will be drowned out. Therefore, when the link at your ISP becomes congested, HTTP will be disproportionately affected.

For the second question, read the section on choking at https://deluge-torrent.org/userguide/bandwidthtweaking/ and Deluge appears to set the maximum number of connections per torrent of 120 with a global max of 250 (though I've seen 500+ in my brief searching, mostly for Transmission and other clients).

I'll admit a lot of my BitTorrent knowledge is dated (having last used it ~15 years ago) but the point remains: ISPs are built for "North-South" traffic, that is: To/From the customer and the networks with the content, not between customers, and certainly not between customers of differing ISPs.

Exactly. As it happens, this is an area I'm working on right now -- instead of using a star topology (direct), or a mesh (BitTorrent), or a tree (explicitly configured CDN), to use an optimistic DAG. We'll see if it gets any traction.

If you download 2 separate files over HTTP, you'd expect each to get roughly 1/2 of the available bandwidth at the bottleneck.

With 1 HTTP connection downloading a file and 100 BitTorrent connections trying to download a file, all trying to compete, you'll find the HTTP throughput significantly reduced. It's how congestion control algorithms are designed: rough fairness per connection. That's why the first edition of BBR that Google released was unpopular, it stomped on other traffic.

> bittorrent will make best use of what bandwidth is available.

At the expense of other traffic. Do this experiment: find something large-ish to download over HTTP, perhaps an ISO or similar from Debian or FreeBSD. See what the speed is like, and try looking at a few websites.

Now have a large torrent active at the same time, and see how slow the HTTP download drops to, and how much slower the web is. Perhaps try a Twitch stream or YouTube video, and see how the quality suffers greatly and/or starts rebuffering.

Your HTTP download uses a single TCP connection, most websites will just use a single connection also (perhaps a few short-duration extra connections for js libraries on different domains etc). By comparison, BitTorrent will have dozens if not hundreds of connections open and so instead of sharing that connection in half (roughly) it is monopolising 95%+ of your connection.

The other main issue I forgot to mention is that on most cloud providers, downloading from the internet is free, uploading to the internet costs a lot... So not many on public cloud are going to want to start seeding torrents!

> ISPs arrange peering accordingly

Two eyeball networks (consumer/business ISPs) are unlikely to have large PNIs with each other across wide geographical areas to cover sudden bursts of traffic between them. They will, however, have substantial capacity to content networks (not just CDNs, but AWS/Google etc) which is what they will have built out.

BitTorrent turns fairly predictable "North/South" traffic where capacity can be planned in advance and handed off "hot potato" as quickly as possible, into what is essentially "East/West" with no clear consistency which would cause massive amounts of congestion and/or unused capacity as they have to carry it potentially over long distances they have not been used to, with no guarantee that this large flow will exist in a few weeks time.

If BitTorrent knew network topology, it could act smarter -- CDNs accept BGP feeds from carriers and ISPs so that they can steer the traffic, this isn't practical for BitTorrent!