Hacker News Reader

steppion Song stuck in your head? Just hum to search (2020)

I think even timbre matters. I got a 95% match singing the first verse of Smells Like Teen Spirit using vocal fry to add some rasp, but couldn’t get over 80% singing in an unaffected soft head voice, though I got a 79% match for a cover by Malia J which wasn’t even originally in the results. Maybe I unconsciously matched the melody more closely when trying to do my best Kurt impression?

I just tried it belting from my chest voice like Michael McDonald and only matched covers; the best match only 33%. None of them sounded like Michael McDonald.

steppion Song stuck in your head? Just hum to search (2020)

Very cool. I tried it humming the choruses from the batch of my own songs I’m working on at the moment to see if I might have committed unconscious plagiarism. There were no close matches and I was feeling pretty good. Than I hummed the chorus of My sweet lord, 29% match to a cover, 18% match to the George Harrison original, He’s so fine by the Chiffon’s wasn’t in the results. I guess this wouldn’t have saved George.

steppion You don't need to work on hard problems (2020)

Largely, it seems top modern researchers are able to free themselves from many of the perverse incentives of the academic game. You might be surprised how many who have not reached such a level also value simplicity, but simply must play the game. It's very common to see a clear and simple (in the good sense) paper grow increasingly complex in response to reviewers demands, going through a sequence of one or more rejections before finally getting accepted.

steppion The First 6 Books of the Elements of Euclid with Coloured Diagrams and Symbols

Oliver Byrne’s 1847 edition of the elements did have colored designs [0]. This website features a digitization of Byrne’s book and defaults to the text as it appears in the original.

[0] https://maa.org/press/periodicals/convergence/mathematical-t...

steppion Introduction to Hilbert Space (2022) [pdf]

Agreed that this is pretty terse. The sequences they're talking about are called Cauchy sequences [0]. A sequence a_i is Cauchy if for any epsilon, there exists an N such that if m and n are both greater than N, then |a_m - a_n| < epsilon. A classic example, suppose your space is the set of rational numbers, and consider the sequence a0 = 1, a_1 = 1.4, a_2 = 1.41, ... a_n = sqrt(2) up to n digits after the decimal place. You can verify that this is a Cauchy sequence, successive points get arbitrarily close to each other. This means the rational numbers are incomplete, because this Cauchy sequence of rationals doesn't converge to a rational number. It's the real numbers that forms a complete space.

Completeness is required for nice results like the spectral theorem for self-adjoint operators [1] to hold, which is pretty essential for Quantum Mechanics.

[0] https://en.wikipedia.org/wiki/Cauchy_sequence.

[1] https://en.wikipedia.org/wiki/Spectral_theorem

steppion Transformers as Support Vector Machines

My apologies for the ambiguity; I assumed it would be clear from context. The article at the link, https://saturncloud.io/blog/what-is-the-random-seed-on-svm-s..., is incorrect. Whoever wrote it seems to have confused support vector machines with neural networks.

For the D-Wave paper, I'm not sure it's fair that they are comparing an ensemble with a single classifier. I think it would be more fair if they compared their ensemble with a bagging ensemble of linear SVMs which each use the Nystroem kernel approximation [0] and which are each trained using stochastic sub-gradient descent [1].

[0] https://scikit-learn.org/stable/modules/generated/sklearn.ke...

[1] https://scikit-learn.org/stable/modules/sgd.html#classificat...

steppion Transformers as Support Vector Machines

The soft-margin SVM which can handle misclassifications is also convex and has a unique global optimum [0].

[0] https://stackoverflow.com/a/12610455/992102

steppion Transformers as Support Vector Machines

The article you’ve linked is incorrect. As Dr_Birdbrain said, fitting an SVM is a convex problem with unique global optimum. sklearn.SVC relies on libsvm which initializes the weights to 0 [0]. The random state is only used to shuffle the data to make probability estimates with Platt scaling [1]. Of the random_state parameter, the sklearn documentation for SVC [2] says

Controls the pseudo random number generation for shuffling the data for probability estimates. Ignored when probability is False. Pass an int for reproducible output across multiple function calls. See Glossary.

[0] https://github.com/scikit-learn/scikit-learn/blob/2a2772a87b...

[1] https://en.wikipedia.org/wiki/Platt_scaling

[2] https://scikit-learn.org/stable/modules/generated/sklearn.sv...

steppion Full Speed Ahead: EV Study Reveals Impacts of Fast Charging

I’m with you on the inefficiency of wide-scale at home solar and your three list items but don’t understand the connection to the Great Leap Forward unless you’re just saying they’re both bad and involved (or would likely involve) ill-calculated intervention and mis-allocation. If anything I think the right analogy is that home solar is to small scale private farming as large solar farms are to large scale agriculture. I think a modern Great Leap Forward more worthy of the name would be an aggressive failed attempt to build massive solar farms and dismantle home solar systems through forced labor after home solar had already achieved ubiquity.

steppion Brand-new Linux release, which I'm calling the Debian Linux Release (1993)

While a college student, Murdock founded the Debian project in August 1993, and wrote the Debian Manifesto in January 1994. Murdock conceived Debian as a Linux distribution that embraced open design, contributions, and support from the free software community. He named Debian after his then-girlfriend (later wife) Debra Lynn, and himself.[3] They later married, had three children, and divorced in January 2008.[4]

https://en.m.wikipedia.org/wiki/Ian_Murdock

steppion How to be successful as a research mathematician? Follow your gut

Tao’s advice is characteristically excellent but I think Eugenia Cheng’s advice (not just the article title but as a whole) is actionable, just not so much for someone who is ready for specific advice like Tao’s. Her message seems to be targeted towards those who are just getting started and who may not otherwise have been interested in studying math.

steppion Math Exposition on YouTube

It hasn’t been nearly so long for me, but I’ve managed to hold on to much of the math I’ve learned by re-proving theorems I’d learned and solving problems in my head during downtime, while doing household chores, and while falling asleep. Another benefit is that I am almost never bored since there’s always more math to do.

steppion GPT-4 details leaked?

I don't think things are quite as dire as that. I have experience as a machine learning scientist, have actually never worked on the kinds of papers you're talking about, but have helped apply machine learning to scientific problems. First in a systems biology lab for tasks upstream of drug discovery, and then in industry for credit risk modeling. I've mostly just used classical machine learning or Bayesian machine learning models though.

My interest has always been in using machine learning as a tool to help understand some underlying phenomena, not in trying to push to the top of the leaderboard on benchmark datasets. I think this kind of attitude isn't uncommon in academic labs focused on doing real science, though the methods used aren't necessarily state of the art, and ML typically plays only a supporting role.

Curiously, I'm going to present at a conference later today and will give a toy example showing why those leaderboards are not necessarily reliable for distinguishing between the quality of different models.

steppion Lev Landau's “Theoretical Minimum”

In undergrad I took a graduate class out of Calculus of Variations by Gelfand and Fomin [0] that went deep into the math of classical mechanics but assumed students already knew the relevant physics. I hadn't taken a proper classical mechanics class, and working through Volume 1 of Livshitz-Landau in parallel with the lectures helped to ground the material for me. A brilliant classmate who’s now a successful mathematical physicist had recommended the series.

Later I tried to learn Quantum Mechanics while taking a course on Operator Theory, but gave up on using Volume 3 of Livshitz-Landau because it was just too impenetrable. I ended up going with Sakurai instead.

I found Volume 1 to be beautiful read and felt like I was learning something profound. Volume 3 was beyond me. The only thing I'd say these books have in common with the Feynman Lectures is a dearth of exercises. This wasn't an issue for me reading Volume 1 of LL because that Calculus of Variations course had many exercises which I just wasn't in a position to appreciate without knowing more Physics.

steppion XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

I did some old school NLP before, but don’t really work in the field anymore. As a generative model. As a general purpose generative model, maybe this isn’t very useful. As a foundation to make models to perform text classification and information extraction tasks, this could be very useful. For these kinds of tasks, you can still get good results with the classic bag of words type approaches people were using 30 years ago even. I remember when transformers first came out, limitations in sequence size made them unusable for some classification tasks.

steppion Every factorial approximates a power

Replying because I can no longer edit. A * got gobbled. The factor should be

factor(x) = ((x + lanczos_g - 0.5)/e)**(x - 0.5).

steppion Every factorial approximates a power

I work on numerical evaluation of special functions. Fyi, typically a better way to evaluate Gamma ratios is to use the Lanczos approximation. See the Boost math documentation [0] for good explanation of the problems with calculating n! / k! as

exp(lgamma(n+1) - lgamma(k+1)).

The form I use for the approximation is:

gamma(x) = factor(x) * lanczos_sum_expg_scaled(x)

where

factor(x) = ((x + lanczos_g - 0.5)/e)*(x - 0.5)

and lanczos_g is a constant.

When dealing with Gamma ratios, you can often cancel the factors nicely, giving a highly accurate approximation which doesn't overflow. It's curious I saw your comment when I did. I was just tinkering with the Lanczos approximation earlier today.

If you use Python, the Lanczos approximation is available in SciPy using

from scipy.special._ufuncs import _lanczos_sum_expg_scaled

and you can use lanczos_g = 6.024680040776729583740234375

[0] https://www.boost.org/doc/libs/1_82_0/libs/math/doc/html/mat...

steppion Brains on Drugs: How tinkering with consciousness became a societal sin

Yeah, industrial alcohols were (and still are) "denatured" with additives to make them toxic or unpalatable and industrial alcohol was often used as a precursor for bootleg spirits during prohibition. I think modern denaturants lean more towards unpalatable than toxic.

steppion Brains on Drugs: How tinkering with consciousness became a societal sin

Not that ethanol isn't also neurotoxic (to a much lesser degree), but bootleg moonshines common during prohibition often contained trace amounts of methanol, the number of people irreversibly injured by methanol poisoning greatly exceeded the number who died.

By 1926, according to Prohibition, by Edward Behr, 750 New Yorkers perished from such poisoning and hundreds of thousands more suffered irreversible injuries including blindness and paralysis. [0]

[0] https://content.time.com/time/specials/packages/article/0,28...

steppion Hello, Perceptron: An introduction to artificial neural networks

Possibly, but I struggle to reason about Bayesian nets at that scale. I think the level at which a Bayesian net could “know what it doesn’t know” would be regarding uncertainty in what text to generate in a given context, not whether or not the generated text is saying something true. One example could be a prompt in a language not seen in the training data. It could be that some plausible sounding made up thing is likely in a given context. Also, at the end of the day, what you’ll get out of a Bayesian LLM is a sample of several generated texts which would hopefully have more variation than multiple samples from the same standard LLM. I can see it being helpful to see if the different outputs agree or not, but I can’t tell at a glance how well it would work in practice.

steppion Hello, Perceptron: An introduction to artificial neural networks

I’m not sure if you are aware, but Bayesian neural networks can be actually be well approximated by appropriate ensembles of standard neural networks [0]. The strength of Bayesian nets (including the approximating ensembles) is that they are able to estimate the uncertainty in their own predictions (by generating a probability distribution of possible predictions), at the cost of more computation needed for training and inference. I don’t think it’s ever going to be a matter of Bayesian nets outright outcompeting standard nets though, it’s just another tool in the toolbox if you want a model which “knows it doesn’t know something” and don’t mind the extra compute needed.

[0] https://arxiv.org/abs/1810.05546

steppion Pierre Menard, Author of the Quixote

Borges’ depth of knowledge of philosophy and mathematics far exceeded what would have been covered in the BA he received in 1914. He was an autodidact who quietly studied a great deal of topics [0]. Beyond being an educated man of his time, I think he was a genius.

[0]https://en.m.wikipedia.org/wiki/List_of_autodidacts

steppion Pierre Menard, Author of the Quixote

Something I find fascinating is that many of Borges’ stories demonstrate a solid grasp of fairly advanced mathematics of his time [0] but he had little formal education. I like to try imagine the work he must have put in to attain this knowledge. I think of Pierre Menard as an example of the extremal principle [1], bringing insight into how works of literature are interpreted by considering extreme cases. A good counterpart to Pierre Menard is The Library of Babel [2], also in Ficciones.

[0] https://en.m.wikipedia.org/wiki/Jorge_Luis_Borges_and_mathem...

[1] https://brilliant.org/wiki/extremal-principle/

[2] https://en.m.wikipedia.org/wiki/The_Library_of_Babel

steppion Learn x86-64 assembly by writing a GUI from scratch (2020)

Yeah. I'm going to be helping to work on expanding CI for OpenBLAS and have been diving into this stuff lately. See the discussion in this closed OpenBLAS issue gh-1968 [0] for instance. OpenBLAS’s Skylake kernels do rely heavily on intrinsics [1] for compilers that support them, but there's a wide range of architectures to support, and when hand-tuned assembly kernels work better, that's what are used. For example, [2].

[0] https://github.com/xianyi/OpenBLAS/issues/1968

[1] https://github.com/xianyi/OpenBLAS/blob/develop/kernel/x86_6...

[2] https://github.com/xianyi/OpenBLAS/blob/23693f09a26ffd8b60eb...

steppion Learn x86-64 assembly by writing a GUI from scratch (2020)

Another example is writing hand optimized matrix and vector operation routines tailored to specific hardware for BLAS libraries [0].

[0] https://en.m.wikipedia.org/wiki/Basic_Linear_Algebra_Subprog...

steppion Learn x86-64 assembly by writing a GUI from scratch (2020)

This is a really cool little example. I've been teaching myself assembly recently and have found Learn to Program with Assembly (2021) [0] by Johnathan Bartlett to be really valuable. I had initially looked through his freely available book Programming From the Ground Up (2003) [1], which covers x86 assembly, and ended up buying the updated book after finding the old one to be well written but out of date. I've been programming in C for a long time and it's been very cool to dig a little deeper and understand better what's really going on under the hood.

[0] https://www.bartlettpublishing.com/site/books/learn-to-progr...

[1] https://download-mirror.savannah.gnu.org/releases/pgubook/Pr...

steppion Everything you always wanted to know about mathematics (2013) [pdf]

It's not one book, but for everything before calculus it would be difficult to beat the books in Israel Gelfand's High School Mathematics Correspondence Curriculum [0]. These are designed for self study and give a fresh perspective on topics they cover.

[0] https://www.goodreads.com/series/318605-gelfand-corresponden...

steppion Fake scientific papers are alarmingly common

Amazing. Reading more carefully, as FabHK pointed out above, they aren't even applying the obvious correction. They're just reporting the positive rate of the imperfect test. I've implemented Diggle's method [0]. When I have time, I'll see if they've provided enough data to do a proper analysis, and maybe write a blog post about it or something.

[0] https://github.com/indralab/opaque/blob/761572ed1b0d601271f0...

steppion Fake scientific papers are alarmingly common

I've worked on research estimating prevalence from imperfect tests, and something that concerns me about this study is that they aren't showing the error bars for their estimates. Typically, you would report a confidence interval for prevalence rather than just a point estimate, and the confidence intervals can often be fairly wide. There's two sources of uncertainty here, the assumed probabilistic nature of the diagnostic test, and uncertainty in our estimates of the sensitivity and specificity.

I think this paper by Peter J Diggle [0], gives a solid methodology. Instead of treating sensitivity and specificity as fixed values using sample estimates, you can model them as each having a beta distribution. In this case these beta distributions can be found using a Bayesian treatment of Bernoulli trials.

[0] https://www.hindawi.com/journals/eri/2011/608719/

steppion An end to pop?

I get it, but I see so many possibilities here. Fitter Happier and Climbing up the Walls have string arrangements performed by an orchestra. Can you imagine writing a score and spinning up a virtual orchestra to play it? There are so many subtleties in phrasing, dynamics, and intonation that make the difference between just Ok and excellent. It doesn’t seem impossible to imagine giving your score to your virtual musicians and communicating with them in natural language and hummed phrases in an iterative process until they get things just right. You’d have to agree this would be democratizing, right?