voidmain

I wonder if this would be a good candidate for the basis of a persistent ("functional") data structure?

O(1) pointer changes for insert/remove is a good sign, since pointer changes tend to become space overhead in a persistent data structure.

If you are using Raft for replication, at any given time your replica set has a leader and it is located somewhere, and I would assume that writes from near there are faster than from anywhere else. In FoundationDB this is handled at a somewhat higher layer of the system, and since our transaction pipeline is (very roughly) resolve conflicts -> transaction log -> storage rather than transaction log -> resolve conflicts -> storage, we are also doing conflict resolution in that region.

Moreover, most current users of FoundationDB aren't willing to accept even 1xRTT latencies anywhere in their transaction lifecycle, so they can't abstract away which region is the fast one. A common way (though not the only way) to set things up is that your whole application (not just the database) is active in region A, but prepared to fail over at a moment's notice to run in region B. Ideally this comes at basically no performance penalty relative to a single-region setup in region A. Alternatively, it's possible to read or write the database from passive region(s) (and individual reads will normally happen from the local region with low latency), but each transaction has to pay a geographic latency at some point (or, in the case of read-only transactions, accept reduced consistency).

I think it would be possible to implement something analogous to satellite replication for Raft. It's a really nice set of tradeoffs for many applications, and if your cloud provider or equivalent has their act at all together instantaneous failures of all the datacenters in a region or their networking should really be pretty rare.

FoundationDB has long supported both synchronous and asynchronous replication across regions, and its major users use multi-regional configurations. In the former mode, you will see 1xRTT latency for commits [1] from the active datacenter and 2xRTT latency for commits from other datacenters. In the latter mode, commits from the active datacenter are fast (0xRTT) but the durability property must be sacrificed if a region fails. I believe almost all users currently use asynchronous replication because the performance is so much better (than any geographically replicated synchronous database).

The new "satellite" replication model in FoundationDB 6.0 allows the best of both worlds: if you have two or more datacenters in each region (each storing transaction logs, but only one storing actual data) then you can do synchronous (0xRTT) commits to the regional datacenters and asynchronous replication to the other region(s). If there are failures in one region, the database can quickly and automatically fail over to the other region, while maintaining durability (by getting the last bits of transaction log from at least one datacenter in the failing region). Even if a whole region fails, as long as the failures of the datacenters and network links in it are spread out over time the database will complete a full ACID recovery. And in the worst case you can fall back to ACI recovery as in asynchronous replication.

The question you are linking to is asking about something different, the ability to have different parts of a single database have subgeographic latencies in different regions.

[1] You will also see 1xRTT latencies when you start transactions, but you can solve this by setting the causal_read_risky transaction option on all read/write transactions. This is not actually risky because if the reads in the transaction turn out not to be externally consistent, the transaction will not commit.

I think the future defense against this sort of thing is more situational awareness on the part of the phone. You won't just have to fool the biometrics, you'll have to do so without making the phone suspicious at any point. My guess is that phones already have enough sensory data to pretty reliably distinguish everyday usage from being stolen or confiscated, and it's "just" a matter of fitting a model. Nor does the phone have to ignore what happens to it after it's unlocked.

No one tries this sort of thing on people or animals, and it's because they don't shut their perceptions off at all times except for a half second when they are authenticating someone.

If someone you know has had bladeless laser eye surgery (with a "femtosecond" pulse laser), it probably used chirped pulse amplification.

FoundationDB uses a consensus algorithm for reconfiguration, but not in the (happy path) transaction pipeline. It provides (by default) strict serializability (i.e. serializability and external consistency/linearizability) for arbitrary, ad hoc, interactive transactions, and it's expected to provide excellent performance when every single transaction is cross node (so e.g. indexes can be efficiently updated this way). It provides better fault tolerance than consensus replicated databases typically can because it needs only N+1 replicas instead of 2N+1 to survive N faults (it keeps 2N+1 replicas of tiny configuration for consensus, if course). It has the best testing story in the industry and is used at scale by, among others, the largest company in the world. Because it doesn't have consensus in the datapath it can have lower latencies than consensus replicated databases in multi region deployments, it also supports asynchronous replication, and an upcoming feature will provide a unique option for multi region failover with sub geographic write latencies, maintaining full transaction durability in failover, as if in synchronous replication, except for the exceptionally rare case where the failure of multiple datacenters in a region are exactly simultaneous.

Besides its extensive documentation, you can read its source code and run its deterministic simulation tests yourself if you are interested (it's Apache licensed). Skepticism on these points was reasonable when we originally launched it in 2012 but is getting a little silly in 2018.

This article lacks any reference to FoundationDB, which offers external consistency and serializable distributed transactions without trusting clocks in any way. We designed it starting in 2009 and so its lineage is independent of either Calvin or Spanner.

FDB doesn't have an actively developed SQL layer at the moment, so I guess you could say it isn't a "NewSQL" database, but none of the properties under discussion have much to do with the query language.

No need to apologize, I thought your comment was perfectly civil. It can be hard to read tone on the internet!

In my admittedly very cursory skim of the paper, I gave it the benefit of the doubt on this point, because this seemed convincing:

"for the purpose of prediction, we are, of course, not so much interested in the probabilities issued by the measure functions, as in the conditional probabilities that give the corresponding predictors’ outputs."

Do you disagree?

(Also, my original summary is wrong at least in that I should have said limit computable where I said semi computable!)

The argument of this very long and technical paper seems to boil down to this:

Solomonoff induction is (in the limit) as good as any computable prediction method, which sounds pretty "universal". But it is not itself a computable prediction method (it's just semi-computable). And if you are satisfied with semi-computability in a predictor, then you should demand instead that a "universal" predictor always be as good as any semi- computable predictor, and it can't meet that more demanding requirement. And so on in infinite regress.

I find this kind of underwhelming. Solomonoff induction represents, in practice, an unattainable ideal that prediction systems can aspire to approximate. This argument shows that if you somehow could reach the ideal, there is an infinite regress of super-ideals beyond it. So what?

One day through the primeval wood

A calf walked home as good calves should...

https://m.poets.org/poetsorg/poem/calf-path

> I think the coolest fact that I didn't grok until undergrad is that gravity relates to the stress-energy tensor and is not just due to rest mass alone. Which means that anything that has energy associated with it (i.e., everything we've ever observed) affects the gravitational field.

And not just energy, but tension. And tension is a signed quantity!

Imagine that you go out into deep (flat) space and build a large disk out of unobtanium, an extremely strong and rigid material. The ratio of the circumference to the disk to the radius is exactly pi. Now imagine trying to bring the disk to Earth's surface. Here space is curved by gravity, so the radius of a disk is just a bit longer relative to its circumference. So the disk has to flex! If it is rigid enough, it will want to spring back into flat space to minimize energy. You could imagine adding some gearing that would let you control this (anti-)gravity effect at will.

Calculating the material strength needed to make this effect usefully strong is left as an exercise to the reader.

(Disclaimer: this makes sense to me, but I am not a licensed antigravity engineer)

Others have pointed out that shadowbanning doesn't really work against sophisticated and coordinated adversaries, and that HN has probably never been attacked by an army. But there's also another problem: someone has to pay for the decision, and the system to make sure that the decision is made fairly, that moderation powers aren't abused, etc. HN is subsidized by YC, and it's not very important in the scheme of things so the injustice of a few bad decisions about bans is not the world's most pressing problem. As networks scale up, it becomes much more expensive to moderate them and much more important to get it right. If abusers of a system pay for their abuse, they can also pay for careful and fair enforcement.

If most people's willingness to pay for non-abusive communications media is $0, then... I expect they will continue to get abuse.

I don't think the problem is federation. The problem is media where participants have nothing at stake, so that abusers can't be made to pay the costs of their abuse. Changing this isn't incompatible with anonymity - for example you could require anonymous participants to post bonds in a cryptocurrency. But e-mail spam, for example, would never have been a problem if the recipient of an e-mail could push a button and destroy $1 of the sender's money. Getting the incentives right for a social network is undoubtedly harder, but you can't even start until people have something to lose.

I'd like to see a transparent ledger for software releases, to make targeted malicious update attacks detectable.

Similarly, pinning the hash of the index resource for a web site using a transparent ledger could let web apps be as secure as native apps with auto update.

Not all kinds of query plans can be expressed as nested loops, though. Index intersections and other ordered merge operations are very useful, and how do you do them with only nested loop plans?

I said "not speculatively executing AVX-512 instructions _when the units are powered down_".

Powering up the full AVX-512 datapath supposedly takes something like 500 microseconds, and then slows the clock speed for something like 2000 microseconds. If so, you could literally just make every AVX-512 instruction fault when the units are powered down, let the operating system (after an appropriate fence) explicitly authorize powering up the AVX-512 datapath, and have no noticeable performance degradation relative to these enormous costs. And once the units are powered up, there is no side channel problem.

On the AVX thing, I think maybe you can eliminate it as a channel for speculative execution attacks by just not speculatively executing AVX-512 instructions when the units are powered down, which also sounds more efficient (it doesn't sound good that, apparently, if (expression_that_is_false) { do_avx_instruction() } can drop your clock speed for several milliseconds if branch prediction guesses wrong! The cost of powering up the AVX units is, I think, much greater than the cost of failing to speculate once.)

In general, I think the problem is that we probably don't know about all possible side channels, and might not for many years. So the approach you suggest - eliminating side channels one by one so that you can't extract information from speculative execution that way - is inherently risky.

We need a certificate transparency type solution that makes pushing compromised firmware to individual devices impractical.

That still leaves forcing manufacturers to insert backdoors in all devices, which as far as I know can only be made tamper evident by open sourcing (and using reproducible builds for) all security critical software and hardware.

This could certainly be a boondoggle, but reducing flow through a dam is limited to storing excess power at the natural flow rate of the river, whereas there is no limit in principle to how fast you could pump water from a big reservoir below a dam to a big reservoir above the dam, or how fast you could draw energy from greatly increased flow through your generators. So pumped hydro storage isn't an inherently silly idea.

> If Amazon wants to publish their assumptions for S3 for comparison, we're all ears.

The 2010 S3 calculation is obviously also wrong! I totally feel your pain in terms of wanting to have a directly comparable answer, but the reasons you give that "it doesn't matter" (and others, including correlated faults, software bugs, model risk, and security breaches) are actually reasons why the stated durability number is wrong. Honestly IMO a probability of 1-10^-11 is the wrong answer to pretty much any question; model risk is going to dominate that for any problem more complex than 1+1=2.

That said, although neither your system nor Amazon's should be expected to have anywhere near "eleven nines" of durability in reality, if as I understand it S3 is split across availability zones in a region and your product has all splits in a monolithic DC, I would expect S3 to come out ahead in a more careful analysis. (But note that S3 is not really a seamless multi region product, though there is an option to set up cross region replication.)

The Crucial M4 SSD had a firmware bug [1] that caused it to fail repeatedly starting after 5,184 power on hours.

Disk failures could also be triggered by datacenter environmental factors shared among many drives like temperature or noise.

[1] https://www.storagereview.com/node/2676

Here is the "most vivid" page of the "most vivid" book:

"giants, and they were impaled by spear, lance, and crystal shard. A series of explosive reports echoed across the battlefield as the giants stumbled upon the Mistcloak’s tripwires, sending lethal blossoms of sharpened steel twisting through the air. Fell and his minions moved through the giants like an avalanche. The Under-King shifted his form to a flowing slab of stone and crashed down upon giant flesh, pulverizing it to blue powder and red ash. Even the animals, though weak and weary, tore into the giants with the primal fury of the wild. Claw and fang stood with horn and hoof, wounding with equal enmity. Beak and talon darted and gouged. The entire island of Mistgard stood united against the foul armies of frost and fire. Devastation was rampant on the mountain, but it was nothing compared to the wrath of the Storm Speaker. Even the stoic Under-King was surprised at the power of the Oldest of Cubs. At the back of the Pandyr’s armies, high atop the tallest of Fell’s battlements, stood the lone figure of the Storm Speaker. He called forth and charmed the very storms from the clouds beneath him and sent electric green-and-blue arcs of lightning into the giants’ lines, blasting hundreds of their bodies off of the battlefield and into the mist below. The world above burned. The radiant morning light was blackened by acrid smoke, making the golden skull radiate a brown and bloody glow. The Aesirmyr lay strewn with broken bodies: blue and red"

I am no expert in German copyright law! But your own source says:

"It is worth noting that the result would stay the same if CC0 would not even contain such an explicit fallback rule. According to the prevailing opinion of the legal scholars, public domain licenses (which cannot be interpreted as a waiver of rights under the German copyright, see above) are reinterpreted as unconditional, unlimited, non-exclusive (i.e. public) licenses."

which appears to contradict your statement that public domain dedications are treated as no license at all.

That looks to me almost exactly like a (1,1) evolution strategy [1]. Our field frequently has invented almost the same ideas multiple times in different subfields, with different mathematical analogies and different terminology. It's useful to identify these connections, and it helps not to be automatically dismissive of people who seem to be using the "wrong" terminology.

[1] https://en.wikipedia.org/wiki/Evolution_strategy

Hopefully the FDA will not move to block this.

In other words, they encourage^Wallow you to use their service to break the law, and then disclaim responsibility when caught?

I'm not saying there is no imaginable ethical sales activity that involves a telephone in some way.

But despite do not call lists etc, I can no longer answer my cell phone when I don't recognize the number, because of all the unwanted commercial calls, scams, surveys, etc. Many are falsifying their caller ID data because they know that everyone would block them if they could.

This product really really looks like it will be used to make that situation worse. "90% MORE LIKELY TO CONVERT A LEAD if you reach out minimum six times". Blech.

Somewhere, people are failing to communicate about something important because these people are pissing in the well.

Telemarketing is wrong. You are working to destroy the value of one of the most marvelous accomplishments of the human race.