745 karma · joined June 12, 2019
Also, relevant tool for similar applications (albeit not SQL based): https://github.com/cloud-custodian/cloud-custodian
> ~1.5x reed solomon is the default these days, again, unless you need read throughput performance
I'm not surprised that Reed-Solomon is the "default these days" given that it exists since the 1960's, and that the most widely available and deployed open-source distributed filesystem is HDFS (which uses Reed-Solomon). However I don't see how that is to be taken as a blind endorsement for it, especially given that the paper in reference explicitly compares itself to Reed-Solomon based systems, including concerns regarding reconstruction costs, performance, and reliability.
> Also, these days the storage of the data doesn't have to be at the same machine that processes the data
Even though what you said here is correct, I don't see how that's relevant to the referenced paper, nor do I think I implied that I hold a contrary belief in any way from what I said.
> Also2, this is completely independent of consistency guarantees
My comment about consistency referred only to the fact that you cannot "simply" spin up more replicas to increase read throughput, because consistent reads often have to aqcuire a lock on systems that enforce stronger consistency, so your comments regarding throughput are not universally true, given that there are many systems where reads cannot be made faster this way, as they are bottle-necked by design.
> Properly done Reed-Solomon has the same durability guarantees as plain old replicated setup
This is not true unless the fragments themselves are being replicated across failure domains, which you seem to address with your next comment with "you need geographic replication if your data cannot be accidentally lost without serious consequences". All of this, however, is directly addressed in the paper as well:
> The advantage of erasure coding over simple replication is that it can achieve much higher reliability with the same storage, or it requires much lower storage for the same reliability. The existing systems, however, do not explore alternative erasure coding designs other than Reed-Solomon codes. In this work, we show that, under the same reliability requirement, LRC allows a much more efficient cost and performance tradeoff than Reed-Solomon.
However for cold-data, there really hasn't been (or at least I am unaware of) any system that can achieve the combined durability of 1.5x Reed-Solomon codes + 3x replication, with such a small penalty to storage costs.
Like you said though, it's definitely not the thing you'd be doing for things that prioritize performance as aggressively as the use-cases you've suggested.
This is a combination of erasure coding AND replication, whose combined storage amplification is dramatically lower than previous SOTA.
I gave a longer explanation in a sibing comment to yours [1]
[0] http://fireless.cs.cornell.edu/publications/erasure_iptps.pd...
I actually found out about this paper because it was referenced in a slide presentation from Google about Colossus (which is the successor to GFS). GFS indeed uses erasure coding with a 1.5x factor, but erasure coding alone does not guarantee durability, and thus needs to be combined with replication to satisfy that requirement, and erasure coding is not the same thing as replication.
The innovation here is explicitly the combination of a new erasure coding algorithm (LRC) AND replication, with a combined storage amplification that is much lower than the previous SOTA.
The paper explicitly compares the new algorithm (LRC) with GFS and other alternatives, and explains why it's better, so this is really not something that is comparable to the 2003 GFS paper in any way (or to any other prior art really), as this is not just a trivial application of erasure coding in a storage system.
There's also this paper [0] from 2001 which digs a bit deeper into the Erasure Codes vs Replication idea that I can recommend if you're interested
[0] http://fireless.cs.cornell.edu/publications/erasure_iptps.pd...
The idea that you can achieve the same practical effect of a 3x replication factor in a distributed system, but only increasing the cost of data storage by 1.6x, by leveraging some clever information theory tricks is mind bending to me.
If you're operating a large Ceph cluster, or you're Google/Amazon/Microsoft and you're running GCS/S3/ABS, if you needed 50PB HDDs before, you only need 27PB now (if implementing this).
The cost savings, and environmental impact reduction that this allows for are truly enormous, I'm surprised how little attention this paper has gotten in the wild.
[0] https://www.microsoft.com/en-us/research/wp-content/uploads/...
That is the meaning everyone that does not primarily use the Northern European alphabet (aka most people in the world) assume it has.
Maybe suppress your instinctive outrage next time, or if you really need to be angry at something, at least direct it at the right place: the dead mathematician who proposed it [0]
Linux already supports offloading even arbitrary eBPF/XDP programs to compatible NICs, which is already extensively used for DDoS mitigation (Cloudflare) or even Load Balancing (Facebook).
I'd be surprised if it didn't leverage these mechanisms to offload the kernel's own network stack as well, but haven't really checked to confirm...
Leaving the implementation of basic functionality, that everyone needs in any language if you're trying to do anything even slightly non-trivial, to the "organic process of letting things naturally arise from the community", has so far been a very effective way to ensure no packages can naturally talk to each other ever, or have any composability.
Everyone re-implements their own "personal abstractions library of crap I need often" like we used to do with C because package management in C is horrid and the stdlib is as crude as it comes, or use one of the many crates of people who decided to publish their 10 LOC crate, and then you get a proliferation of microscopic packages that implement trivial functionality and the ecosystem starts looking a lot like NPM.
Rust could really use some more top-down leadership. Say what you want about Rob Pike (and I'm not saying that Golang doesn't have pain-points with the leadership), but almost everything in the ecosystem composes like magic.
On top of that, everytime I try to use Rust I'll need a feature and will run into some issue from 8 years ago, where everybody admits it's a problem, there are good and obvious solutions on the table, but all those 8 years were spent circlejerking and bikeshedding and no one can make up their minds; it's not even a "this is what the future of this issue looks like but it will take some time", it's more like a "after 8 years consensus hasn't been reached and no progress has been made on something a lot of people agree is a problem".
Sometimes the need for group consensus serves only to corrupt the conceptual integrity of systems, and dilute any sense of vision into oblivion... I wish we could do without it sometimes
I thought we were still tied to `bcc` and that python frontend thingy, but I'm glad the ecosystem has evolved significantly while I was not paying attention! It's kinda like finding a $20 in your back pocket that you didn't know was there when you put your pants on...
[0] https://ebpf.io/
Federated distributed computing relates to what you're thinking of, but it is an extremely difficult topic, and veers into the realm of blockchains and things like that. There is also the whole area of Zero-Knowledge Proofs/Compute that deals with these matters: https://eprint.iacr.org/2013/229.pdf
If your threat model falls under the uber-paranoid category, you have to on-prem everything, preferrably inside your very own bunker. There is no getting around this afaik, and if there is there will probably be major tradeoffs (like the blockchain thingy) instead of a "drop-in" replacement for traditional compute.
Setting aside the fact that BitTorrent came about long before BitCoin was a glint in Satoshi's eye, it's still massively used for content distribution, on top of being a foundational technology to several industrial applications: Apache Spark, for example, uses BitTorrent to shuffle/broadcast data around cluster nodes.
Even when BitTorrent alternatives are used in its place (such as IPFS, Dat, or Kademlia), BT was still a massive influence in all of their designs, and to the design of any other DHT that came about after its release.
All of these, plus other systems that are (in abstract) similar constructions, have several industrial applications for asset distribution (Netflix uses IPFS to distribute container images internally [0], Uber and Alibaba do similar things) and network management (DNS for example, is really just a DHT, as are several other network protocols)
Ever since BitTorrent came about, it has unequivocally been a resounding success. It never for a moment had to look for applications, there were already plenty since day one.
[0] https://blog.ipfs.io/2020-02-14-improved-bitswap-for-contain...
On your first point: Was aware of this, and this is the part that is most puzzling about the whole incident to me. I just checked my $PATH and I am indeed running the system binaries, so not sure how to explain this one.
On your second point: Was aware of this too, but I assume a lot of the slowness comes from JITted programs, for which there will be phoning-home for any new executable memory page (AFAIK), and the policy decision caching semantics for things that are not on disk are not as clear to me.
On your third point: While this is true for the connection that gets established to Apple's servers, I think this might have had to do with DNS being UDP based by default (AFAIK), so there is no explicit refusal, and it hangs on a timeout because of that, even though DNS can be done over TCP as well. Haven't investigated this though, just a hunch...
On your fourth and fifth points: I don't use XCode, but my terminal (Kitty) was already on the Developer Tool list when this happened, which makes the situation with ps and grep even more mysterious to me...
I guess the questions I'd have regarding this that I'd need to validate would be:
- Does this setting persist across reboots?
- Does it stop phoning-home altogether, or will it still phone-home, but always authorize regardless of answer?
- Is there a more granular way to disable just phoning-home, rather than nuking all of Gatekeeper? (Some of its things like disallowing access to personal files by default are still nice to have)
Will report back with an update after testing...
Easier to automate setup through Homebrew, no need to store/activate the license or anything, doesn't have the "draw the window position" popup thingy, but I don't really care about that and gotten used to the (very simple) default shortcuts.
I highly recommend it, give it a try if you're interested!
[0] https://rectangleapp.com or https://github.com/rxhanson/Rectangle
I almost locked up my entire computer this week, presumably because of this... here's the story for that:
I had setup unbound listening on 127.0.0.1:53 as the only DNS resolver for all network interfaces, so that I hijack several responses that I might want to (e.g. adnet sinkholes) and forward the rest to Google/Cloudflare over TLS.
When I rebooted the Mac, the entire computer was so fucked that I panicked cause I thought my SSD was failing. Turns out unbound failed to start so all DNS was down. I suspect it to be the "phoning home" because when I opened the Terminal, running `ps aux | grep unbound` took like 5 seconds to return anything (as did everything else that tried to run), and that behavior has been reported by other users as Gatekeeper's Phone Home checks being at fault.
So that's why I was wondering if anyone has found out how to keep macOS from doing this (preferably without disabling all of SIP), cause that'd be one hell of a tip...
Haven't tested or validated that it can be done though, just an idea...
Notable diferences that might influence your decision are:
- RPMs instead of DEBs;
- Flatpaks instead of Snaps;
- Podman and Buildah instead of Docker (although you can get Docker if you really need to use that);
- SELinux enabled by default (some people don't like this for non-server usage, but I dig it);
- firewalld comes enabled by default, which may be annoying and unexpected if you're trying to get some iptables rule to work (I personally always remove firewalld and install ufw for the things I need);
- Fedora 33 (due for release next month) will be switching to btrfs as the default filesystem, whose features are definitely welcome for home usage (but I'll wait a bit before upgrading and see if people run into any issues);
- I'd also highly recommend installing Pop!_OS's Pop Shell [0] to add great tiling support for GNOME, but that goes for anyone using GNOME really
[0] https://github.com/pop-os/shell
If you like gaming though, I never tried setting up Steam or a ProtonDB game on Fedora to be able to report on that (I think it would be complicated enough to make me wanna switch distros), but if you'll be doing this a lot, Pop!_OS (Ubuntu based, snaps disabled) has a great out-of-box experience with Steam, as does Manjaro (Arch based) which has an excellent hardware detection and driver auto-installer tool called mhwd, and makes setting up NVIDIA cards and other finicky hardware a breeze.
Yes, but only if you assume that an adversary's power is stable over time, and that it's safe to amortize the risk over an extended period of time, both of which are wrong assumptions.
> If your users have strong passwords (e.g. a 20 alphanumeric random password from a typical password manager) it makes no difference at all
Yes, except that is not the world we live in, which makes this a pointless argument.
> If your users have very weak passwords then once again it makes no difference. Scrypt password hashing doesn't prevent me guessing that Steevo412's password might be something obvious
> None of this makes you any safer from inadvertently leaking the plaintext passwords, which your system unavoidably needs to know during authentication
Yes, if you frame the question as "being able to defend against threat models you can't effectively defend against under this authentication model" then indeed none of this matters, but that's a self-propelling argument, and also not the discussion we're having.
The discussion we're having is that given we know password-based auth is a bad security model, and that people have shit security practices, how do we squeeze the most value out of whatever entropy is given to us, in a way that will last as much as possible, since passwords are often recycled, rarely rotated, and you can get pwned decades into the future because of a leak that happened in 2004.
Password-based auth is a reality, and will keep on being for a long time. It's just not a defensible position to argue that there is no point in leveraging given entropy to the max because "we're all doomed anyways", or that since it's a bad model we shouldn't care.
The goal is explicitly to protect knowingly reckless users from themselves. Saying that "if they all used random passwords from a password manager it wouldn't matter" is not realistic or helpful.
> If security actually is important you could deploy something that's actually a clear improvement such as WebAuthn
I don't think anyone at any point in this thread defended passwords as a solid threat model, and I made this same point in another comment. WebAuthn is awesome, but passwords are a reality and will keep on being for a foreseeable future, we just have to deal with it
There are differences among them between classes of attacks they're susceptible to, provable security properties, and how paranoid should you should be in respect to advances cryptanalysis.
I tend to weigh these as significant factors given that the dynamics of password-based auth can very well lead to a database leak screwing someone over 20 years into the future, although I admit that you can't get too picky nowadays and I'd be very glad if we, as a species, could have what you described as a lower-bound (with the exception of PBKDF2, of course :P)
> QUIC is what happens when corporations define standards so their own services can be run cheaper
1) Google is not the only one who would be able to run their services cheaper. That is as plain a fact as it could be. Do you think Google is the only company who needs to serve massive amounts of content? In what way is energy and computation efficiency "harming" the internet?
> The transport layer shouldn't be "aware" of what it is transporting
2) In what way is QUIC "aware" of what it's transporting?
> It shouldn't be a gigantic heap of things
3) QUIC is not a "gigantic heap of things" any more than TLS or SCTP is. It is an L7 protocol hosted on UDP, just like TLS is an L7 protocol hosted on TCP.
> It is already impossible for anyone but Google (or another mega-corp) itself to make a browser that works with "modern" sites. This piles on to that
4) This has nothing to do with browsers, and QUIC has nothing that relates exclusively to browsers. Also, the idea that competing in the browser space is will be harder because you need to implement a new transport is laughable. Browser complexity derives from HTML/CSS/WebApps/etc, and QUIC is not only insignificant by comparison, but also a much less volatile feature that wont need to be "evolving" all of the time.
> Protocols should be generic
5) QUIC is absolutely, 100% generic? I'm not sure where you're getting the idea that it isn't from...
> Layers should be layers
6) Except when the layers are horribly misplaced of course; then you get into the situation we're in with TCP where we have hard-coded definitions for how control flow will work during a given session, what the session is, how you always need a new handshake to start a new one even if on the same host, and then you need to work around all this bullshit by starting a crapton of TCP connections to multiplex streams like HTTP/1.1 does...
7) In what way is QUIC not a layer?
8) That is a horrible argument. Nothing is keeping the IETF from standardizing QUIC at the same level as UDP/TCP/SCTP, you just have to look at IPv6 and SCTP's adoption to understand what happens when you try to change a protocol that deep in the stack.
> There's more to the web, and the internet, than fixing mobile latency issues
9) Sure... If you ignore that unlike when TCP was standardized, mobile devices represent 50% of all publicly routed internet traffic, and that all of these devices are dropping packets, triggering retransmissions, and congesting the network for everyone (including non-mobile devices), then there is no probelm at all!
That is also known as the tactic 5 year olds use when faced with a problem: closing their eyes and pretending it doesn't exist.
10) I'm not sure where you got the impression that QUIC only brings benefits to mobile latency, but that is also patently false
That being said, for these cases you could use Argon2 which is mostly (entirely?) based on ARX constructs and will work well on embedded/underpowered devices
There are also some symmetric encryption shenanigans that can be done, but if you can use something other than password-based auth you're free from pretty much all of these problems, with probably much better security
If you're on Golang, Rust, or C though: leverage your options!
> For the vast majority of developers, bcrypt, scrypt, argon and PBKDF2 provide functionally equivalent security
1) The "vast majority of developers" should not be implementing login systems, period. The chances most people have of not falling for any OWASP gotcha, making sound security choices, and implementing them correctly, is pretty much nil. Leveling the argument to this makes many things that should not be done sound passable.
2) They do not, categorically, provide "functionally equivalent security" (especially not for bare PBKDF2). This is a myth people believe in because they normalize their perception of deviant behaviour[0], and frame the situation as "if my database never gets pwned, any one of these is fine", which is just an argument based on wishful thinking, akin to "I can drive recklessly as long as I don't crash", but we don't use this reasoning to nullify seat-belts: the choice of algorithm is important precisely, and perhaps exclusively, for when all your other security mechanisms failed.
The reality though, is that more often than not, when databases gets pwned you never find out about it because monitoring and security practices is often lackluster, and then you keep believing that these things don't matter.
[0] https://en.wikipedia.org/wiki/Normalization_of_deviance
As a general rule for their security properties: Scrypt > Argon2 > Bcrypt, and PBKDF2 should be avoided. You should prefer the first one of these you can find with a robust implementation (which as @lanecwagner pointed out may not always be Scrypt, and that's fine, as is Bcrypt if you have implementation constraints)
Also, it's not a good idea to base your security parameters on the compute power of the most underpowered device in the chain.
Step 2) There is no step 2
QUIC is a vastly superior transport to TCP in every technical regard. What exactly makes it a "purely corporate needs fulfilling step backwards for human people", if you don't mind adding some context to this extremely vague statement/opinion?
> A lot of the challenges that are part of implementing HTTP/2 correctly are also present in QUIC
I'd dispute this because it doesn't have nested flight windows, and doesn't have to multiplex and interleave streams over the same tunnel (it does that over the same socket/port, but it's way easier because UDP is a stateless packet machine-gun, so you don't need to keep both TCP and SPDY's concept of a stream at the same time).
It has about the same challenges of implementing TCP correctly (which is what you'd expect really), because it needs to implement congestion control and etc pretty much in all the same places, the only "extra" it shares with SPDY is stream multiplexing (which is still much simpler because it doesn't need to worry about TCP's semantics on top of it).
> resource wise, it might never catch up with HTTP/1.1 + TLS (I think you meant TCP here) due to missing hardware support
I have two comments about this:
1 - This is actually more of a software problem than a hardware problem. Hardware acceleration for QUIC depends mostly on evolving the tooling we have for "talking" to our network cards across different platforms than anything else really.
For a while now, most NICs (especially those you'd see in data-center gear) use a Motorola 68k based chip, or something else equally programmable, and offloading arbitrary programs to those is already something that happens quite often. For example, Linux XDP programs are automatically offloaded to the NIC if it supports that.
This is also not a problem for middle-boxes, since they just treat UDP blindly as UDP and will just do AQM on top of it like they always did, no need to implement flow control.
This might be a problem for legacy consumer hardware, but then again those are already inefficient in all kinds of ways, and for these cases being able to run the entire networking stack from userland is actually a benefit: being able to fallback on full-software implementations means we can make Windows XP support QUIC if we wanted to. Can we even measure how much shit we'd break if we tried to "upgrade" TCP?
2 - This might be underestimating the amortized gains from switching to a more efficient protocol: having to retransmit less data, 0-rtt handshakes, and such things add up.
Specifically for HTTP1.1, being a text based protocol doesn't help either: how many CPU cycles do we waste with parsing CRLFs, lowercasing headers, and crap like that? At least HTTP2 doesn't suffer from these problems, but it still screwed up transport big time.
> I feel like at least in some areas the QUIC specification became less complex
This I agree with, wholeheartedly.
So far, the only thing I've found absolutely detestable about it is mandated encryption, as that prohibits me from outsourcing some L7 intelligence to a proxy/sidecar/service-mesh/whatever, and having to shoehorn some bullshit certificate between my application and the proxy so that it can have some visibility into traffic will be error prone and annoying.
Maybe we'll make a variant of QUIC with TLS disabled for gRPC and other API shennanigans? We'll see when the time comes...
Damn, I was not aware of this, but sure enough there it is [0]
I suppose that's just one more thing to reaffirm my belief that SPDY is a horribly over-engineered protocol, that tried to shoehorn fixes for TCP's shortcomings on top of the wrong abstraction (which, ironically, is TCP itself).
Had HTTP2 only introduced the binary frames encoding, instead of bolting this horrible new transport along with it, we wouldn't be so deep in this mess. QUIC can't come soon enough.
[0] https://www.chromium.org/spdy/spdy-protocol/spdy-protocol-dr...