jwuphysics

Have a PhD in physics/astronomy, so—full disclosure—I'm not an expert here... I skimmed the paper and I have no idea what he's talking about.

Given that half of his eight references are to his own papers, and the other half are textbooks or the Heaviside's original work, I think we can assume that he's doing some very niche work or a crank.

Could you say a bit more about how so?

Incredible, well-documented work -- this is an amazing effort!

Two things that caught my eye were (i) your loss curves and (ii) the assessment of dead latents. Our team also studied SAEs -- trained to reconstruct dense embeddings of paper abstracts rather than individual tokens [1]. We observed a power-law scaling of the lower bound of loss curves, even when we varied the sparsity level and the dimensionality of the SAE latent space. We also were able to totally mitigate dead latents with an auxiliary loss, and we saw smooth sinusoidal patterns throughout training iterations. Not sure if these were due to the specific application we performed (over paper abstracts embeddings) or if they represent more general phenomena.

[1] https://arxiv.org/abs/2408.00657

You can get a nice sample of papers using VVV data using the Astrophysics Database System [0]. I mostly study other galaxies, which usually aren't variable on human lifespan-like timescales. Stars can vary on these shorter timescales, and VVV has compiled a huge list of those objects.

At a quick glance, I'd say some interesting results include: * New star clusters discovered in our Galaxy [1] * Galactic maps of dust reddening and stellar metallicity (enriched elemental abundances in stellar photospheres) [2] * Galactic maps of stellar ages throughout the disk plane [3] * Cataloguing other galaxies behind the plane of our own Galaxy [4]

[0] https://ui.adsabs.harvard.edu/search/fq=%7B!type%3Daqp%20v%3... [1] https://ui.adsabs.harvard.edu/#abs/2011A%26A...532A.131B/abs... [2] https://ui.adsabs.harvard.edu/#abs/2011A%26A...534A...3G/abs... [3] https://ui.adsabs.harvard.edu/#abs/2019A%26A...623A.168S/abs... [4] https://ui.adsabs.harvard.edu/#abs/2012AJ....144..127A/abstr...

> Why not just say "incoherent output"? Because the biggest problem with hallucinations is that the output is usually coherent but factually incorrect. I agree that "hallucination" isn't the best word for it... perhaps something like "confabulation" is better.

Have you looked into TinyCorp [0]/tinygrad [1], one of the latest endeavors by George Hotz? I've been pretty impressed by the performance. [2]

[0] https://tinygrad.org/ [1] https://github.com/tinygrad/tinygrad [2] https://x.com/realGeorgeHotz/status/1800932122569343043?t=Y6...

PBMs hold an incredible amount of power. The Acquired podcast did a marvelous breakdown of the American pharmaceutical industry while covering Novo Nordisk (https://www.acquired.fm/episodes/novo-nordisk-ozempic). If you have three hours to spare, I highly recommend giving it a listen.

Another great interview comes from Mark Cuban, who is serious about disrupting PBMs with his Cost Plus Drug Company (https://www.drugchannels.net/2024/03/mark-cuban-five-ways-th...).

That makes sense and I'll run a few evals. Many thanks for open sourcing your work!

How much do you expect auto-context and clustering+re-ranking to help for cases in which documents already have high-quality summaries? For context, I parse astrophysics research papers from arXiv and simply embed by paper abstracts (which must be of a certain size), and then append (parts of) the rest of the paper for RAG.

I am an astronomer and was at a conference when this headline broke. Obviously we ignored this and read the actual papers, e.g. https://arxiv.org/abs/2404.03002. There you can see that the dark energy equation of state parameter w0 = -0.99 +0.15/-0.13, which is not at all significantly different than w0 = -1. That is, no time evolving dark energy. When combined with other cosmological results, the joint analysis slightly favors w0 > -1 rather than w0 = -1 at the ~2.5sigma level or so, but it depends which other results you factor in.

Oof, yeah that's not a very good response. But I guess color is actually a technical term in research astronomy: it refers to the difference in two specified magnitudes (usually photometric filters). There's also the question of whether the sun is being viewed from the Earth or from space, since those will change the effective "transmission curve" of the detector system.

I asked: "What color is the sun when viewed from the ground with the human visual system?"

And got the following: "The Sun appears white to us on Earth due to its high temperature and lack of any significant wavelength-dependent absorption or scattering properties. However, if we were able to view it through specialized telescopes that could capture all visible light spectrum (not just the yellow/orange part), then it would appear as an extremely bright ball of blue light with some slight red tint. This is because most of the solar radiation consists of photons at ultraviolet and infrared frequencies which our eyes cannot perceive directly but can be detected by these advanced instruments. [...]"

An overall better response, but still not exactly right. Anyway, the base model was fine-tuned on arXiv/astro-ph abstracts, and I can't imagine too much discussion about the color of the sun in that training data set...

UniverseTBD (https://universetbd.org) is also making great strides in the space of large language models and astronomy.

Thanks for your spectacular resources! I see that you began an Annotated Mamba repository -- any chance you could share when that blog page might go live?

I would be surprised if this is the case. Neural scaling laws are well known and are used by all big industry players to extrapolate experiments.

Hi Jeremy & Eric, great to see your newest endeavor. I hope that Answer.AI builds on the success and impact that fast.ai has already enjoyed.

Given new developments in hardware (by companies not named NVIDIA), I'm wondering if you are keen on exploring the next generation of model architectures and optimization procedures that might exploit newer hardware. In other words, will research directions pivot based on the hardware lottery?[1] Are you in conversations with companies developing these alternative chips?

[1] https://arxiv.org/abs/2009.06489

Duplicate: https://news.ycombinator.com/item?id=38614232

You can read the [50 Years of Text Games](https://if50.substack.com/archive) newsletter that preceded the book. I highly recommend it!

There is indeed lots of uncertainty, and all Milky Way mass estimates are model dependent. Also, the ~10^12 solar mass figure referred to above is an estimate of the total galaxy + dark matter halo mass. Stellar or baryonic mass estimates are closer to ~6 x 10^10 solar masses. I'm using "~" to mean uncertainty to within a factor of 2 or so.

> The idea that AI might transform scientific practice is therefore feasible. But the main barrier is sociological: it can happen only if human scientists are willing and able to use such tools.

As a tenure-track scientist who works in ML applications for astrophysics, I disagree with this sentiment. The main issue isn't that enough scientists are using tools to search through literature or form new hypotheses, the main issue is that scientists now have to validate and sift through AI-generated outputs in order to find useful signals, rather than validate and sift through experimentally derived or observed signals.

AI can be useful for hypothesis generation in my field [0], and I think that there are lots of great use cases where it can be used to summarize information. However, it always comes with the possibility that it might output complete nonsense [1], so scientists who adopt these tools will have to spend some of their time verifying their outputs.

[0] https://arxiv.org/abs/2306.11648

[1] https://web.archive.org/web/20230913230733/https://www.msn.c...

> why did you loose an interest?

Two reasons:

1. The posts I'd find most interesting in the site are ones related to astronomy and applied machine learning. If there wasn't anything on astronomy (or physics) then I probably wouldn't use the site in the long run.

2.HN is a news aggregator. Most links that reach the front page have something interesting to offer. If I can't access the part of the site I find interesting, then I'll like go back to HN and find another post. (This isn't something I'd love to admit, but oh well.)

Although not the point of my comment (I don't want to sign in at all) -- I personally like a username/password the best.

As others have said, forcing a login is a huge barrier.

I wanted to see if any astronomy posts were on the front page, and when I didn't see any, I tried to check out the categories. But when prompted for a login/sign-up, I lost interest.

Fascinating! I dug into the Wikipedia article, which cites a Scholarpedia article; the LLM answer seems to originate from a reference to this sentence [1]:

> So, MOND reduces the discrepancy in clusters at these radii to only a factor of ∼2−3 (Sanders, 1999; Ettori, et al., 2019)

So I think you're right, and today I learned something! I also checked if Stacy McGaugh had weighed in on this particular subject, and it seemed like there is still an issue for clusters [2], although interestingly the issue isn't mentioned in his latest blog post that summarizes the strengths/weaknesses with MOND [3]. Anyway, thanks for humoring me for a bit.

[1] http://www.scholarpedia.org/article/The_MOND_paradigm_of_mod... [2] https://tritonstation.com/2021/02/05/the-fat-one-a-test-of-s... [3] https://tritonstation.com/2023/06/27/checking-in-on-troubles...

In terms of the actual article -- really nice finding. Or I guess, nice set of experiments to decipher what lots of LLM researchers have been finding!

I've noticed somewhat similar behavior while training graph neural networks to model physical systems, except that it takes way longer than a single epoch to get there. Or course, there's no pretending involved with my GNNs, but the models do have very constrained representations, so once they start to figure out how to represent the physics at hand, the loss plummets dramatically.

Hi Jeremy, always a fan of your work! Just a technical note since it falls under my domain of expertise (astronomy) -- the example about MOND described here should actually have choice (E) as the correct answer!

> Please do not propose this regulation. If consumers actually cared about their IoT devices receiving security updates, companies would be doing it. The fact that companies are not already doing this is evidence it's not important to consumers. People may express frustration, but their purchasing behavior speaks louder than their words.

Or, maybe, companies are exploiting consumer ignorance and we're not dealing with an efficient market.

A professor remarked that it was a bit sad that physics students nowadays have a better understanding of quantum field theory than fluid mechanics. He mentioned this while lecturing on QFT.

Please don't editorialize the title. The title is

> Starlink Generation 2 Mini Satellites: Photometric Characterization

I'm a professional astronomer and I don't know what the "brightness reduction target recommended by astronomers" is. I do know that a 7th magnitude source will likely saturate most large research telescopes, rendering a good chunk of that data useless.

> What makes you think their proposal isn't more likely than a black hole?

Because a boson star is purely theoretical and I have a much stronger observational prior that the object is a black hole.

Satellites are also one of the biggest sources of radio frequency interference. (I did my PhD in radio astronomy and currently work on multiwavelength astronomy.)