cscheid

> Spectre is a bug in the processor, not in the software.

It's a bug in the processor that causes a bug in the software. It's not a bug in your idealized mathematical model, but try telling that to the people who paid you not to leak private keys.

I see my job as an engineer to be to create a product that satisfies the user's expectations (which in this case are eminently reasonable). It matters not one bit that I can point the finger to the chipmakers. I'm still selling something that I now learned doesn't do what I said it would. It's still on me to fix it the best I can. If I care about the product quality, that is.

> how different is the impact on the manufacturer when the software is used by 1k users, 1M users, and 1B users?

_very different_, when the user's environment is different. And 1) you haven't seen shit if you think you can perfectly control the user's environment. 2) every new user is a chance for the environment to bite you.

> Do they make the product unprofitable or unsuccessful in anyway?

You do your engineer best to try and fight that. But there's absolutely a marginal cost, which is what I was responding to.

> And for good enough software you should consider that maximum already for any number of users.

I don't believe such software exists. (And, to be clear, I'm writing from direct, day-job experience.)

EDIT: I take it back. SQLite, cURL. Maybe.

EDIT2: I can't reply to the SEL4 response, so here goes. I'm a huge fan of verification tools, but consider the Spectre class of bugs. Verification is always done wrt a mathematical model that you've defined after inspecting the world and writing down the properties you want to track. But the world changes, and the chance that the world changes increases with the number of users of your software. That's the nature of the beast.

> Software has no marginal cost.

Maybe you've never experienced the difference between writing software for 1000 people and writing software for 1M people, or (I imagine) 1B. The marginal per-person cost of software is not on shipping. It's on "what kind of weird shit will I now have to do because 1M is a lot of chances for my software to break weirdly, and people have paid for it"

> You don't have to worry about quality control and returns.

You don't have to worry about quality control and returns if you don't care about quality control or returns.

On LaTeX:

> Little context sensitivity

... this is close to a disqualifyingly wrong statement.

(I literally work with Markdown on a daily basis as my actual day job. Markdown really does suck.)

The only way in which (La)TeX is not context sensitive is that it's sensitive to the entire global TeX state at all times. There's no notion of grammar in TeX, _at all_. It's hard to trust the rest of the content in there after this.

I don’t quite know why, but my combination of first and last name on gmail is such that I get email directed at other people with the same name as me, including financial documents. Wild stuff. I would reply with “um you probably should check before sending” but after a while I just started ignoring it.

Take a look at https://automerge.org/ and the stack those folks are building. You're exactly right that it's a difficult balance (specifically the trick is proving commutativity for the domain-specific data of your application). But automerge (and then https://github.com/inkandswitch/peritext) show it's at least possible. Good stuff.

One thing that saved an estimated couple of million lives comes to mind.

This is relatively subtle stuff, but here's an attempt at describing the general problem. I'm going to describe the deterministic case, but the probabilistic case is effectively the same.

Let's say you have a bug you suspect is from an interaction of any one pair of 10 features being "on" or "off", but you don't know which specific pair causes the problem. Encode each of the states you could set up your code by a 10-digit binary string: 0000000000, 0000000001, 0000000010, 0000000011, etc.

We could try the 45 possibilities in some order, and we would expect that on average it'd take us 22.5 tries to find the bug. But notice how your "target set" is smaller than the universe of strings: there's only 45 pairs of features, but 1024 strings.

What happens if we try a random string of ones and zeros? Now, instead of catching just one possible pair, we are covering many pairs. The only problem is that we now won't be able to know exactly which pair caused the problem when it does. But we can build a corpus of strings that don't trigger the error vs. strings that trigger the error, and a random sampling soon converges on the correct pair.

If you think about why this works, it's because any of these random strings has about a 1/4 chance to trigger the bug: wlog we can reorder the bits so that the buggy feature are the first two digits, and then we see that we have a 1/4 chance of hitting "11" on those two digits.

The problem is that as you increase the size of the subset that needs to be active, the probability that your random strings will actually catch the bug decreases exponentially. For any _fixed_ target size k (the number of features that need to be active), the overall complexity is still polynomial in n (the number of existing features). But if k is a constant fraction of n, then this technique takes exponential time in n.

All-pairs is for _pairs of features_. For subsets you're in much deeper trouble because of the exponential dependence on N. For a fixed polynomial dependence, you can get clever and let tail bounds eventually work for you, but for exponentially growing hypothesis sets, that won't work.

> After reading your comment I dug around to see how many Chapman Stick players there were (What's the TAM for anything targeted at Stick players..if you pardon my startup-speak). The stick subreddit seems exceptionally small: 596 members. I'm guessing there are non-reddit niche community forums with much bigger numbers to boast of.

So, each Stick has a unique serial number. I got mine in 2021 and it's 6596. So that gives you an idea. I totally understand if the market isn't there.

"There's dozens of us!"

I added my email to your mailing list, but also please feel free to reach out directly to me over email (it's on my account info).

I put together an observablehq notebook with a Stick fretboard (https://observablehq.com/@cscheid/hanon-01-diagrams-for-chap...) for me to work on some fingering patterns, but something like fretboardfly would be so awesome (even if it were just one side at a time). A configuration file like "string tunings + fretboard markers" would totally do it.

omg, I would pay $100 for something like this on the Chapman Stick fretboard. ("which Stick fretboard and which tuning" of course are the hard questions on said weird instrument... but 10-string Baritone Melody please? :) )

> dummy argument aliasing

Thanks for the precise Fortran terminology; that's what I meant to say in my head but you're correct. From the linked website for everyone else:

"Fortran famously passes actual arguments by reference, and forbids callers from associating multiple arguments on a call to conflicting storage when doing so would cause the called subprogram to write to a bit of that storage by means of one dummy argument and read or write that same bit by means of another."

The language semantics matter greatly.

GHC with `-fllvm` is not going make Haskell any easier to compile just because it's targeting LLVM. Fortran is (relatively!) easy to make fast because the language semantics allow it to. Lack of pointer aliasing is one of the canonical examples; C's pointer aliasing makes systems programming easier, but high-performance code harder.

> No subgroup is inappropriate unless you know all of the values of all of the parameters

I don't know how to put it less bluntly: you're incorrect.

> But sure, the intent they tried to demonstrate (take all sub-group analyses with a grain of salt)

That's not what they intended to demonstrate. They intended to demonstrate that you need a _reason_ to want to split, and that reason needs to be given _ahead_ of the analysis. If you see the results _and then choose_ a new data analysis (that includes new subgroup analyses), your procedure is no longer statistically sound.

This is a specifically bad statistical practice called HARKing https://en.wikipedia.org/wiki/HARKing

Yes, exactly. There's a real-life example of an author group who got sufficiently annoyed at a reviewer requesting an inappropriate subgroup analysis: https://www.thelancet.com/journals/lancet/article/PIIS0140-6... Reviewers asked for the subgroup analysis; authors said "no, this is statistically nonsense"; reviewers said "yes, but we'll reject the paper otherwise"; reviewers said "ok, but only if you let us also split on astrological sign".

Result: paper reports that aspirin has an effect, but only if you're not a Gemini or Libra. Too good.

Sorry. You do mention a linear systems response, and that's what I meant.

In that setting, the eigenvectors work as a generalized forward and inverse fourier transform, and the eigenvalues form the transfer function you allude to in the bold sentence

"The attention mechanism’s role is the same as that of a transfer function in a linear time-invariant system, namely it calculates the frequency response of the transformer model,"

Specifically, it seems to me that this requires a _symmetric_ attention matrix. Which you get from the self-attention mechanisms (two of the three places where they're used in transformers), but not all of them, notably not the one that combines the output of the first two attention mechanisms (one input, and one output)

Interesting paper.

I have a question about the claim in 6.2 that attention matrices are SPD, if you don't mind my asking.

It seems to me that accepting the empirical result that the eigenvalues are positive isn't enough to get a Fourier Transform interpretation. Specifically, I don't understand the assumption that all attention matrices are symmetric. (I'm sure you know that positive eigenvalues are not enough by themselves, but for other folks reading, [[1 1/2] [1/3 1]] is a simple concrete example.)

Consider Fig. 17 here: https://lilianweng.github.io/posts/2018-06-24-attention/ (this is Fig.1 in Attention is all you need). I understand that you get symmetric attention matrices for the self-attention matrix in the input stream, as well as the masked attention matrix in the output stream (the first block). But I don't understand how you claim symmetry for the final attention mechanism that combines input and output.

And if you don't get symmetry, you don't get the Fourier Transform interpretation and all the nice algebra that follows.

I don't know why you're getting downvoted. I read the methods section and the stats model doesn't seem to adjust for "calories consumed after dieting stopped". Those calories, as you point out, can be correlated with PFAS because of their association with high-calorie foods. This seems to me like a potential confounding factor.

> There used to be all sorts of cardboard "slide rules" that basically used slide rule-like principles to do calculations specific to some particular industry or function

Those are nomograms, and agreed - they're really freaking cool: https://en.wikipedia.org/wiki/Nomogram

> Does it know? Is it coincidence? Is it malicious?

In case you haven't read it yet, you might enjoy Peter Watts's Blindsight.

Ask it the other way around, "Why is the 8K context only twice as cheap as the 32K context?" and your original answer is clearer: because they think the demand curve supports it.

Price is not determined by cost, but by how much people are willing to pay.

Not _any_ problem, but the _specific_ problem of determining how many arbitrarily placed points (cherries) can be split by a given shape of hypothesis classes/classifiers (knives) is literally the definition of VC-dimension, yes :)

And if you think sufficiently hard about curved knifes and cherries on high-dimensional pies, you end up studying machine learning theory (https://en.wikipedia.org/wiki/Vapnik%E2%80%93Chervonenkis_di...)

Read the whole paper! Seriously, do it. The author is being cheeky with showing how an academic urban legend gets made and it takes the whole paper to get the punchline.

Looks like a larger one than usual: https://www.githubstatus.com/incidents/t2xwk9mz56f4

You can't do graphs without numbers, so the history makes sense. Finite graphs are relations over finite sets. You can't do much with finite sets before inventing numbers.

That would make it not really what distill was about. Concretely, the article my students wrote wouldn’t have been possible - it was literally proposing a new visualization method and showing a real implementation on the web.

Distill was one of the best experiments in publishing of the last decade, no irony. Unfortunately, it’s worth reflecting on why they are on a hiatus that I fear will be (understandably) permanent.

When I was an academic, I had the privilege of participating in the process of producing one article for Distill, and the amount of work was equivalent to 3-5x the work of any one single publication in other venues. I’m not sure that’s avoidable to achieve the quality that Distill strives for, but it means the incentives are all pointed against it.

A direct consequence of working on an environment with bad incentives is that people there will burn out, which is part of what I think happened.