HNHacker News
TopNewBestAskShowJobs

Majromax

3,557 karma · joined February 1, 2019

submissionscomments
Majromax··on Show HN: Tiny Diffusion – A character-level text diffusion model from scratch
The basic MLP block in this model uses a ReLU^2 activation function (x <- ReLU(x)^2). That seems to be copied from the nanochat project, and it's not present in nanoGPT. Is there some documentation on the choice of this activation function?
Majromax··on Leaving Meta and PyTorch
> You just write down your math equations as JAX and you're done.

It's natural when your basic unit is a whole vector (tensor), manipulated by some linear algebra expression. It's less natural if your basic unit is an element of a vector.

If you're solving sudoku, for example, the obvious 'update' is in-place.

In-place updates are also often the right answer for performance reasons, such as writing the output of a .map() operation directly to the destination tensor. Jax leans heavily on compile-time optimizations to turn the mathematically-nice code into computer-nice code, so the delta between eager-Jax and compiled-Jax is much larger than the delta between eager-Pytorch and compiled-Pytorch.

Majromax··on Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model
> If OpenAI or Anthropic could squeeze the same output out of smaller GPUs and servers they'd be doing it for themselves.

First, they do this; that's why they release models at different price points. It's also why GPT-5 tries auto-routing requests to the most cost-effective model.

Second, be careful about considering the incentives of these companies. They all act as if they're in an existential race to deliver 'the' best model; the winner-take-all model justifies their collective trillion dollar-ish valuation. In that race, delivering 97% of the performance at 10% of the cost is a distraction.

Majromax··on A bug that taught me more about PyTorch than years of using it
> Could this really invalidate research? Managing to produce a model that works (assuming you check all of the myriad modeling correctness checkboxes) is sufficient on its own.

In the academic sense, a model that happens to work isn't research; the product of research should be a technique or insight that generalizes.

"Standard technique X doesn't work in domain Y, so we developed modified technique X' that does better" is the fundamental storyline of many machine learning papers, and that could be 'invalidated' if the poor performance of X was caused by a hidden correctness bug avoided by X'.

Majromax··on Don't Force Your LLM to Write Terse [Q/Kdb] Code: An Information Theory Argument
> When I(short proof)=I(long proof), per-token average surprisal must be lower for the long proof than for the short proof. But since surprisal for a single token is simply -log P, that would mean that, on average, the shorter proof is made out of less probable tokens.

This assertion is intuitive, but it isn't true. Per-token entropy of the long proof can be larger if the long proof is not minimal.

For example, consider the "proof" of "list the natural numbers from 1 to 3, newline-delimited." The 'short proof' is:

"1\n2\n3\n" (Newlines escaped because of HN formatting)

Now, consider the alternative instruction to give a "long proof", "list the natural numbers from 1 to 3, newline-delimited using # for comments. Think carefully, and be verbose." Trying this just now with Gemini 2.5-pro (Google AI Studio) gives me:

"# This is the first and smallest natural number in the requested sequence.\n 1

# Following the first, this is the second natural number, representing the concept of a pair.\n 2

# This is the third natural number, concluding the specified range from 1 to 3.\n 3"

We don't have access to Gemini's per-token logits, but repeating the prompt gives different comments so we can conclude that there is 'information' in the irrelevant commentary.

The author's point regains its truth, however, if we consider the space of all possible long proofs. The trivial 'long' proof has higher perplexity than the short proof, but that's because there are so many more possible long proofs of approximately equal value. The shortest possible proof is a sharp minimum, but and longer proofs are shallower and 'easier'.

The author also misses a trick with:

> Prompted with “Respond only with code: How do you increment i by 1 in Python?”, I compared the two valid outputs: i += 1 has a perplexity of approximately 38.68, while i = i + 1 has a perplexity of approximately 10.88.

… in that they ignore the equally-valid 'i = 1 + i'.

Majromax··on Understanding Cultural Differences: The Michigan Fish Test (2013)
Not just the scientific replication crisis, but also the scientific media replication crisis.

For all of the p-hacking and file drawer effects that modern research (noting this is an older piece) tries to avoid, the incentives for popular scientific media including blog posts all run in the other direction. Even if limited to just good, replicable studies in journals, anything we hear about via popularization is likely to be attached to a stronger-than-real effect size.

Majromax··on Making Florida More Flood Resistant Is Forcing Hard Choices for Homeowners
> And also, dont count on the government, like FEMA, to help. Now with phrases like "climate change", and "green" now banned from federal discourse, we're not permitted to say the obvious.

It's a cold calculus, but a government that is ideologically opposed to the science of climate change is more rather than less likely to help bail out flooded properties. With such a… fixed opinion, the hidebound government agencies can't allow themselves to think that the overall risk profile has increased, so they'll be surprised by a hundred-year flood every 10 years.

The confounding factor is that the current US political mix combines ideological opposition to climate change science with opposition to well-functioning assistance programs, so on balance there may be nobody left to write the bailout cheque.

Majromax··on Anthropic raises $13B Series F
Not in the United States, but it is illegal in some jurisdictions.

Additionally, the entire "payment processors leaning on Steam" thing shows that it might be very difficult to monetize a model that's known for generating extremely controversial content. Without monetization, it would be hard for any company to support the training (and potential release) of an unshackled enterprise-grade model.

Majromax··on The sisters “paradox” – counter-intuitive probability
In all of these questions, you're making an assumption about the data-generating process. In Q1 and Q2, you're assuming that you had a 0% chance (a priori) of hearing that 'neither is a (girl/boy)', and in Q3 you're assuming that there's a 0% chance of hearing that the envelope doesn't match the family.

Take a look at this problem beginning with no assumptions. We have two kids, and an envelope that contains 'B' or 'G'. Our probability space is (B,G)^3, with each having probability of 1/8.

Now, we add information about the match as conditioning. Conditional on being told that the envelope matches the family, we can exclude the BBG and GGB cases. That brings us down to 6, of which we have BBB, GGG, and (BG,GB)(B,G). With this additional information, the probability of matching genders becomes 1/3. This probability is still 1/3 if we open the envelope to find B or G, since we exclude all three cases where the envelope doesn't match our observation of it.

In my view, this is related to the Monty Hall problem; we have to realize that we're given additional information with the statement/envelope.

Majromax··on Why are anime catgirls blocking my access to the Linux kernel?
It's civil cases that have the lower standard of proof. Civil cases arise when one party sues another, typically seeking money, and they are claims in equity, where the defendant is alleged to have harmed the plaintiff in some way.

Criminal cases require proof beyond a reasonable doubt. Most things that can result in jail time are criminal cases. Criminal cases are almost always brought by the government, and criminal acts are considered harm to society rather than to (strictly) an individual. In the US, criminal cases are classified as "misdemeanors" or "felonies," but that language is not universal in other jurisdictions.

Majromax··on US Wholesale Inflation Rises by Most in 3 Years
> I dont think it’s the tariffs. I really think the entire category of merchants looked at each other and figured it was fine to raise prices.

Those statements aren't contradictory. Pricing in non-commodity goods is always a human phenomenon, driven ultimately by the idea that a business can 'get away with it' without losing too many customers to competitors.

Tariffs, on the other hand, provide a good reason/excuse to raise prices. Affected firms see their cost structures blown up, so they have to raise prices at least somewhat. Unaffected firms see that their competitors are raising / must raise prices, so they know that they can raise their own prices in the now less-competitive market. That's how and why tariffs work: they give domestic firms protection to have profits in excess of what they would have under a free trade counterfactual.

Any argument that tariffs don't raise prices is a very long-term one, relying on the idea that it will be easy to open new firms and expand domestic production to ultimately return to a competitive market. This has nearly no effect on short-term pricing, and it's why the least-damaging changes to tariff policies are long-run and well-telegraphed plans, slowly implemented.

Majromax··on Ozempic shows anti-aging effects in trial
> Yet women have a longer lifespan despite shorter reproductive years.

After industrialized nations have reduced both maternal mortality and violent death (warfare) to small fractions of their pre-modern levels. The life tables of 1500, 500, or 5000 BC would look very different than their modern versions.

Majromax··on Scarcity, Inventory, and Inequity: A Deep Dive into Airline Fare Buckets
Was it really interesting? To me, it has certain hallmarks of an AI-generated article. In particular, it introduces the same concept several times, in different sections. For example, fare classes, nested booking, and the SABRE system each get two different introductions.

The content seems legitimate, but I felt like my time was being wasted through at minimum a lack of editing.

Majromax··on NIH limits scientists to six applications per year
> The solution shouldn't be then to fill them with LLMs but strip them entirely away.

You don't need language models to identify useless processes. The problem, however, is that people tend to be more comfortable with a process that exists whose product is ignored rather than no process at all.

For example, in the case of the grants here it's easier to imagine giving money to someone with a Gantt chart – even if that chart will never really represent reality – rather than someone who says 'trust us to use the money effectively.'

For an alternative view, a lot of the information supplied in such processes isn't related to the happy path, but rather it creates a paper trail for blame when things go wrong.

> I tend to think the same about everyone freaking out about LLM misuse in education.

The difference for education is that students need to practice, so the repetition is the point. The AI might ultimately be better at writing the book report, particularly compared to a student in 6th grade, but there's few other ways to train skills of reading comprehension and analysis.

Majromax··on Conspiracy theorists unaware their beliefs are on the fringe
> Here's an idea: treat conspiracy theory as theory, an explanation of a set of facts, rather than as social contagion or pathological disorder. Allow open discussion of all sides of an issue, admit controversial facts, and structure the discussion in such a way that truthfulness and explanatory power rises to the top.

That's well worth doing when evaluating the merits of a potential conspiracy theory for yourself. However, it's not evident that this approach will convince people who incorrectly believe conspiracy theories of their errors.

Eponymous conspiracy theories have a built in 'immune system' to reject logical, scientific argumentation, since most if not all refuting facts can be attributed to the untrustworthy or malicious conspiracy itself.

Ultimately, scientific argumentation depends upon all involved parties having a shared understanding of what observed facts are true or false, and if that agreement is not there then otherwise rational parties can still reach different conclusions with no persuasion possible.

> Sabine Hossenfelder handled "flat earth theory" in a way I found refreshing. She said, "Huh, this is interesting. I appreciate that they aren't afraid to question fundamental assumptions. That is a scientific attitude. But here's why I think they're wrong."

Was this approach persuasive to any 'flat earthers', or was it instead interesting to watch as a 'round earther'?

Majromax··on Jane Street barred from Indian markets as regulator freezes $566M
> The alternative claim - that large traders do not make decisions based on how their activity will move the market, is of course absurd.

No, that's de riugeur for any large firm. You or I can buy a stock without changing the market, but if a bank or hedge fund wants to buy or sell millions of dollars of a stock they very much have to worry about execution. The price they receive for a trade will be worse than the market price, simply because the price will move once it's clear that someone is making a large block trade.

This is also why hedge funds worry about their 'alpha'. Even when they have found a good basis for a trade (e.g. an as-yet unexploited correlation), taking a position to profit on that trade moves prices in ways that eliminate that edge. That's the efficient market hypothesis in action.

The very odd thing here is the allegation that Jane Street could make perfectly ordinary market transactions in liquid securities and somehow have the market move with their net position. This is extremely unusual.

Majromax··on Bitcoin's Security Budget Issue: Problems, Solutions and Myths Debunked
> Counterparty risk asks whether you'll collect on that bet.

You can make those bets on regulated exchanges, thanks to bitcoin-trust ETFs. IBIT is the largset of those, with $74bn assets under management, and it looks like options contracts are available. Correlated bets would also be easily available with companies like Microstrategy.

I think the biggest risk to such a move would be legal, since executing a demonstration 51% attack would plausibly result in a market manipulation investigation.

Majromax··on Bitcoin's Security Budget Issue: Problems, Solutions and Myths Debunked
One can take short positions on bitcoin almost as easily as long positions. With enough leverage, a well-connected firm could probably make a low 11-digit bet fairly easily.
Majromax··on Solving `Passport Application` with Haskell
> Do you have any insight as to why, seemingly, there is no contractual obligation on contractors to make a working system.

Often, you'll find that procurement is handled by someone working off of an incomplete set of written specifications. The procurement officer probably has no expertise with or informal knowledge of the problem to identify problems in advance, and even if they do legal requirements (ironically designed to ensure fair and transparent procurement) might force them to disregard informal knowledge over the formal specifications.

After the contract is signed, sunk cost fallacies and optimism biases take over. If the project goes over budget it might be easier to cut back on features than admit failure or demand more money, and the project will go over budget once real users start seeing the in-progress work and change requests pile up. Project managers want to be seen managing a successful project; nobody's rewarded for being the first to predict failure.

This is a kind of principal/agent problem that the private sector partially manages with equity or deferred compensation, but the profit motive has a very limited role in government and you'd probably not want to give stock options anyway. Beyond that, even the political overlords have short time horizons of 'the next election', and announcing a grand new project is almost as good as delivering one.

A high-profile example of this problem is the Canadian federal payroll system (https://www.oag-bvg.gc.ca/internet/English/att__e_43045.html), which has led to billions of dollars in fix-ups over a decade or so.

Majromax··on MCP: An (Accidentally) Universal Plugin System
> It would have been much better if the description language had been more formalized.

To speculate about this, perhaps the informality is the point. A full formal specification of something is somewhere between daunting and Sisyphean, and we're more likely to see supposedly formal documentation that nonetheless is incomplete or contains gaps to be filled with background knowledge or common sense.

A mandatory but informal specification in plain language might be just the trick, particularly since vibe-APIing encourages rapid iteration and experimentation.

Majromax··on Third places and neighborhood entrepreneurship (2024)
Original paper link: https://www.nber.org/papers/w32604
Majromax··on Maximizing Battery Storage Profits via High-Frequency Intraday Trading
> seems dumb to have electricity needing to be wasted when there is seawater to desalinate

That's a much more complicated problem. On an energy market, you have only one price to look at, and the battery operator can always buy, sell, or hold energy. The article here talks about optimizing this problem at 5-minute to several-hour intervals.

If you drop excess power into desalination, however, now you have two prices to worry about: energy and water. I also doubt we have 5-minute spot markets for water, so the operator must probably commit to some medium-term water delivery regardless of price.

This means that a desalinating firm takes on much more risk. This might still be profitable, but it's a long-term play based on a deep model of expected energy prices (i.e. knowing that energy is "always" almost free at noon in summer) rather than short-term time-shifting.

Majromax··on We’re secretly winning the war on cancer
> Adjust for those factors and the increased incidence disappears.

'Adjusting' for those factors builds in the assumption that they're independent of the thing you're trying to measure. If living near a smokestack is undesirable, then poorer/marginalized people will live there even if it also causes cancer.

Majromax··on AI Saved My Company from a 2-Year Litigation Nightmare
> There isn't clear caselaw on this yet, but given that material you share with third parties other than your lawyers almost always is and sharing your strategy musing would probably be disastrous, you should plan accordingly.

Wouldn't this have largely been settled already with cloud computing? If I use gmail to write an e-mail to my lawyer, then I can't imagine that e-mail is unprivileged because it's "shared" with Google.

Majromax··on US cancels funding for Moderna bird flu vaccine
> First they were supposed to prevent COVID. That turned out to not be true,

In fact, the vaccines did a fantastic job at preventing covid's spread. The problem was that they were developed for (and had clinical trial results for) the variant-free covid-classic. By the time they were widely deployed, the virus in circulation was already one of several more-easily-spread variants.

Had covid not mutated into variants, it seems very likely that widespread vaccination would have given us true 'herd immunity'.

> That turned out to not be true, so then they were supposed to lessen the symptoms.

They were also always supposed to do this, by making infection less severe and more easily fought-off. Many vaccines operate this way, including the seasonal flu vaccine.

Remember that until clinical trial results came in, we weren't sure exactly what the vaccine was going to do. Something that "merely" helped keep elderly people alive would have been good enough, and by that standard what we ended up with was a fantastic success.

> Everyone I know who still frequently gets COVID had the shot.

As phrased, you've walked right into selection bias; perhaps those people you know choose to get the booster vaccines because they recognize they're more likely to be exposed to covid.

> There were tons of surprise side effects especially relating to women’s periods getting screwed up.

Treatments have side effects, but for pandemic covid the right comparison was "vaccine plus none-to-mild covid" versus "no vaccine plus unmitigated component."

Infection with covid has plenty of 'side effects', and women frequently report disrupted menstrual cycles from infections.

> The official story was that we were trying to make COVID die out before it mutated again, but to do that would have required vaccinating all animals as well.

As far as I'm aware, every circulating covid variant that we've traced has likely come from mutations inside humans, not zoonotic intermixing.

> Despite the above, there was tremendous pressure from left wing institutions to vaccinate everyone, proving that their values were about pharmaceutical profits instead of my body my choice.

Now you're just being disingenuous. "Left wing institutions" were at the forefront of vaccine equality among nations, arguing strongly for coordinated buying approaches to ensure access to developing nations. If anything, that contributed to early politicization of the vaccine as the counter-reaction was "if this vaccine is so good, we need to have it for ourselves first."

The values involved were simply that of the common/societal good against individualist freedom, the same ones at play in lockdown/distancing rules.

> Anyone who investigated the origin story became spontaneously racist. But then later morality reversed and it was fine.

Note that this has absolutely nothing to do with vaccines.

Majromax··on Getting a paper accepted
> Many times it is very difficult to “get into” a new field if you don’t have an author known by the editors.

Although there's plenty of critique to go around about the review system, machine learning here typically uses double-blind peer review for the major conferences. That blinding is often imperfect (e.g. if a paper very obviously uses a dataset or cluster proprietary to a major company), but it's not precise enough to reject a paper based on the author being an unknown.

Majromax··on Getting a paper accepted
> The reason could be that clever titles add "novelty", but not much substance.

Another reason might be that clever titles stand out as bold claims, working counter to the common practice of academic humility. If a paper seems to be downplaying its own significance, then why should a casual reader (or reviewer, at first impression) give it the benefit of the doubt?

That's not to say that papers should over-claim, and I suspect that doing so might lead to a harsh counter-reaction from reviewers who feel like they've been set up to have their time wasted. Nonetheless, "project confidence" might be good practice in academia as well as one's social life.

Majromax··on Getting a paper accepted
Looking at the differences between the rejected and accepted papers, I don't think it's quite a matter of 'avoiding citations'. The changes seem to break along two lines.

1. Avoid overly general citations. The rejected paper leads with references to image captioning tasks in general and visual question-answering, neither of which is directly advanced by the described study. The accepted paper avoids these general citations in favour of more specific literature that works directly on the image-comparison task.

2. Don't lead with citations. The accepted paper has its citations at the end of the introduction, on page 2.

I think that each change is reasonably justified.

In avoiding overly-general citations, the common practice in machine learning literature is to publish short papers (10 pages or fewer for the main body), and column inches spent in an exhaustive literature review are inches not spent clearly describing the new study.

Placing citations towards the end of the introduction is consistent with the "inverted pyramid" school of writing, most commonly seen in journalism. Leaving the review process out of it for the moment, an ordinary researcher reading the article probably would rather know what the paper is claiming more than what the paper is citing. A page-one that can tell a reader whether they'll be interested in the rest of the article does readers a service.

Majromax··on What do wealthy people buy, that ordinary people know nothing about? (2015)
> The cumulative inflation for the past 10 years

Without taking a stand on which metric is better for social and lifestyle comparisons, the grandparent poster said 'valuation of assets' rather than inflation. Per Yahoo Finance (https://finance.yahoo.com/quote/%5ESP500TR/), the S&P500 is about 340% of its mid-2015 level on a total return basis (reinvesting dividends).

Today's $700-millionaire would have been a 'mere' $200-millionaire in mid-2015 if invested fully in equities. That allocation is probably ballpark reasonable, since with that kind of net worth investment horizons are very long (multiple generations of inheritors) and the investors themselves can be nearly risk-neutral rather than risk-averse.

Majromax··on No-boom supersonic flights could slide through US skies soon
> And is this a real thing that can be done? How do you make the sonic boom go away?

You can't completely make the sonic boom go away, but you can change how it's experienced on the ground through careful design of the airplane shape. The X-59 Quesst (https://en.wikipedia.org/wiki/Lockheed_Martin_X-59_Quesst#De...) is an experimental supersonic aircraft that reportedly reduces the sound of the sonic boom to something comparable to a 'car door closing', mostly by ensuring that different shockwaves created by different parts of the plane don't combine into a single, stronger boom.

> can it be used to make non-supersonic planes quieter, too?

Sadly no; the principles of design are very different for sonic booms and ordinary plane noise. For ordinary aircraft, the big causes of noise are the engines themselves and turbulence over the airframe.

That being said, the problem of ordinary aircraft noise is usually limited to the areas near an airport. Sonic booms are noticed underneath the entire flight path of a supersonic aircraft, even when it's flying at altitude.

← PreviousPage 6 of 27Next →