rwilson4

Counterpoint: regularization is just a way of specifying a Bayesian prior for maximum a posteriori estimation.

Anecdotally, SpaceX pays less than other aerospace companies. Source: I used to work at Boeing, I have friends at SpaceX and various other aerospace companies.

I love how it’s from 2016 and still has NOTESSSS comments.

Very cool! Any stats you can share about increased performance switching to go?

I was skeptical about the “applied” part, but glancing through it, the examples are really interesting! Discussing magnetic fields in the same chapter as lambda calculus, very cool! Seems a bit light on details though...

Using the techniques described on my blog [0], which are ideal for KPIs like conversion rates and small sample sizes (since no Gaussian approximation is made), I get a p-value of 0.177, which is not significant. The observed treatment effect is a 36.8% lift in conversion rate, but a confidence interval on this effect has endpoints -10.7% and +97.4%. Anything in that range would be considered consistent with the observed result at a 0.05 significance threshold.

With 6000 total impressions and a 50/50 split, the experiment is only able reliably to detect a 74% lift in conversion rate (with power = 80%).

If you want to rigorously determine the impact, decide what effect size you hope to see. Use a power calculator to decide the sample size needed to detect that effect size. Administer the test, waiting to acquire the planned sample size. When analyzing, be sure to compute a p-value and a confidence interval on the treatment effect.

[0] https://www.adventuresinwhy.com/post/ab-testing-random-sampl...

Discussion from last time: https://news.ycombinator.com/item?id=19075599

Lots of practical challenges for sure!

You're right, and in some instances it is possible to draw causal conclusions from observational data. See [0] and [1] for two pretty different perspectives. But for this to work, you need a lot of data: both lots of units (e.g. people), and a lot of information about each individual unit.

[0] Causality, Judea Pearl

[1] Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction, Guido Imbens and Donald Rubin

Yes absolutely! Random assigment along with statistical power and significance considerations does indeed allow one to draw causal conclusions. It’s the gold standard for causal inference.

It’s a paradox because many people find it counterintuitive. It’s the mathematical statement of why correlation does not imply causation. The existence of a confounding variable correlated both with the purported cause (eg gender) and the purported effect (school admissions) can lead to reversals in observed association when grouped or broken out. Thus it is challenging to draw causal conclusions from observational data.

Short, to-the-point, and I wasn't familiar with any of these options! Love it!

Differences in notation between different fields abound. It used to be (probably still is) the math department didn’t talk to the engineering department, so people would reinvent basically the same solution in different contexts. Now people maintain the notation for internal consistency, instead of switching to “universally standard notation”.

Basically, historical consistency trumps cross-disciplinary consistency.

That’s a good point! I’ve typically piggy-backed off these types of systems to do that kind of analysis. But you’re right it is a distinct use case.

Events are the cornerstone of product analytics. If you want to understand what your users are doing on your platform, and to look for opportunities to improve the user experience, events are a big part of that.

Super cool! Out of curiosity, is this an academic group or a (potentially hiring?) company?

Update: After some internet stalking, might be related to JPL? I’d love to hear some meta info about this project: why it was created, what you’re doing with it etc.

This is all good info; I'll add that a confidence interval on the effect size[0] is more informative than a p-value. (This is subtly different than a confidence interval on the response in each group, which is discussed in the article: Warning Sign I5: taking p too seriously.)

[0] "It's the effect size, stupid" https://www.leeds.ac.uk/educol/documents/00002182.htm

We definitely do study specific instances. For example, the other day an article was posted here on Support Vector Machines, which are a convex optimization problem having a special structure that is exploitable for fast solutions. For more examples, see Convex Optimization by Boyd and Vandenberghe, and the course notes for EE364a/b at Stanford. Modern Convex Optimization by Bertsekas is also good. I had the privelege of taking the convex optimization sequence at Stanford from Prof. Boyd, and one of the major take aways is that many practical problems can be solved in quadratic or even linear time. A general purpose solver typically runs in cubic time. For large problems (lots of variables), a special purpose solver can solve problems in seconds that would take a general purpose solver hours!

The book posted here is very much an introduction; even the EE364 sequence was merely a jumping off point for being able to explore the literature. Mathematical optimization, and especially convex optimization, is a deep and beautiful field!

Yes, all constrained optimization problems can be converted to unconstrained problems. For example, consider:

minimize f(x) subject to x \in C

Let g(x)=f(x) if x \in C and infinity otherwise. Then

minimize g(x)

has the same solution as the original constrained problem. However, this conversion often conceals some of the structure of the original problem which can be exploited to solve the problem more efficiently.

Solving point problems as you have called them typically involves solving a sequence of least squares problems, which are simple to reason about and computationally efficient to solve. Solving a calculus of variations problem typically involves solving an integral or partial differential equation. Although there are theoretical similarities in practice they are pretty different.

SVMs are a convex optimization problem, which means any local optimum is also globally optimal. In practice that leads to fast, provably optimal solutions.

With deep learning, the problem is non-convex, so the results are locally optimal but not necessarily globally optimal. We get around this with different techniques such as randomized initial weights, but with few exceptions, non-convex problems are NP-complete. Therefore there is no polynomial-time algorithm for finding the global optimum in deep learning unless P==NP.

I occasionally hear people claim deep learning finds the globally optimal solution, but that’s just not correct (or they mean they are doing an exhaustive search e.g. branch and bound, which has exponential worst case run time and isn’t practical).

That said, locally optimal solutions are often “good enough” for practical problems.

I think people end up reaching for straws because a lot of this stuff works well in practice and they’re trying to make sense of it. 20 years from now we’ll have a much clearer picture of why the techniques work! Until then, a lot of the explanations, maybe even some valid ones, may sound like baloney!

Science is a combination of theory and experiment. Sometimes theory advances faster than experiment, sometimes vice versa. Right now in ML, experiment aka practice is advancing faster than theory. Theory will eventually catch up.

I do donate to homeless shelters and cancer research foundations, and I hope other people are also donating to causes important to them! But I don’t have enough money to make a dent in the problem. Collective action is the only way to make progress.

I would gladly pay higher taxes if the money were spent on homelessness or education or healthcare instead of the military.

Or increasing taxes on the wealthy and using that to provide a better safety net.

Then we'll find out when it comes online! If the new theory is compatible with past observations, and correctly predicts something that GR gets wrong, that will be a huge finding! But the press has gone wild prematurely, as they often do. As others have mentioned, that seems to be the real issue to which Dr. Hossenfelder objects.

It's not disqualified for not being simple, it's disqualified for not making testable predictions. Given two theories, neither of which make testable predictions, the simpler one (and the one that has been studied more thoroughly) is the more reliable.

Otherwise, why not just say that the missing mass is unicorns? And the energy driving the accelerated expansion of the universe is the energy of their love? There's no shortage of radical explanations for dark mass/energy. The scientific community takes them seriously to the extent that they are compatible with past observations and make testable hypotheses that distinguish them from other theories.

I believe the author's argument is: however simple or complicated dark matter and energy are, negative (inertial) mass just makes things even more complicated, without offering any new (testable) insights. The author herself points out that she (and the GR community) may be wrong about a lot of this stuff (e.g. that gravity is mediated by a spin-2 field), but absent a testable hypothesis, negative masses don't actually move the field forward at all.

Seemingly "crazy" ideas sometimes turn out to be correct! But in science, we demand radical ideas at least make testable hypotheses.

...and I’m bankrupt.

I spent about a year studying Italian on Duolingo and had a positive experience. But I also augmented Duolingo with Anki: I made flashcards for every single word I learned in Duolingo, and went over them every day. That really helped me memorize the vocabulary. I also got a workbook on verb conjugations to practice those (I didn't end up finishing that workbook and my verb conjugations are weak).

The real benefit of Duolingo is that I can do it while I'm waiting for my coffee order, or on the train. My only free time is spontaneous, and in short increments; e.g., 5 minutes here and there, unexpectedly. It's hard to learn anything in that context! I'm surprised I learned Italian as well as I did! Taking a class in Italian with a human teacher would no doubt be superior, but I unfortunately don't have time for that!