bobmarleybiceps

I think before I was born, there was just a ton of (closed-source) C compilers for different systems. So people were like "we should have a standard, so our safety critical application doesn't break when we decide to use a <new hardware vendor's> C compiler."

I also think since compiler infrastructure has improved so much, and everything is open source now ... so idk if it matters as much anymore.

I think a lot of past examples of "emergent capabilities" have at least partly turned out to be that more stuff was included in the training data that (intentionally or not) covered the capability.

Like i.e., I think the "large models exhibit emergent capabilities that small models don't" turned out to basically be "larger models are often trained on more extensive data"

oh that actually helps a lot. I was tying myself in knots trying to think it through.

yeah you're right. I wrote that poorly and seems I can't edit it now.. In the link I shared, he does propose the similar theta and omega notations and provides the existing definition for big O.

Knuth originally defining them https://dl.acm.org/doi/10.1145/1008328.1008329

that's not the convention I know... (from CLRS algorithms book).

Big O is sort of like <=. little o is sort of like <.

https://dl.acm.org/doi/10.1145/1008328.1008329

dang, that's exactly what it is. I was not expecting someone to recognize it! This also made me smile :-)

Agreed. I generally feel pretty comfortable with letting LLMs implement stuff like individual functions or algorithms. But I feel like it takes some effort to make sure separate things stay separated... (I do think it also helps LLMs if the code to stays nicely abstracted.)

On the other hand, I do some stuff with physics simulations and LLMs will spit out arrays like

static constexpr std::array<int, Q> cx{0, 1, 0, -1, 0, 1, -1, -1, 1};

static constexpr std::array<int, Q> cy{0, 0, 1, 0, -1, 1, 1, -1, -1};

static constexpr std::array<real_t, Q> w{ real_t(4) / 9, real_t(1) / 9, real_t(1) / 9, real_t(1) / 9, real_t(1) / 9, real_t(1) / 36, real_t(1) / 36, real_t(1) / 36, real_t(1) / 36};

static constexpr std::array<int, Q> opp{0, 3, 4, 1, 2, 7, 8, 5, 6};

and it's like ??? I know what these are for, but I have no clue if any of the numbers are right :-)

at least IME, chrome was unusable on Mac because it would regularly use an absurd amount of memory and freeze everything up.

but we only download the giant binary blobs from trusted actors, such as sam Altman and Elon musk!!

I think nature sort of has a reputation for sensationalism / probably overhyped (or outright wrong) stuff in the sciences now anyway... would not surprise me at all that they're desperate for ml stuff :-I

I think eventually companies won't get as much stuff that's usable for marketing, so they'll stop investing so much into ai for math, so eventually cheap and poor graduate students will be able to do relevant work again without worrying about getting scooped by a company with a million GPUs :-/

I feel like ultimately the journal / conference system of "submit pdf and never be able to update it" doesn't make sense in 2026.

- In a lot of fields, the "deliverable" could just be a git repo. - Results could / should be more reasonable to update as possible issues arise. - Peer review is pretty busted in a lot of fields, so a humongous number of papers get through that have pretty glaring issues.

The combination of being incentivized to spew out nonsense papers + academic publishing institutions liking to get more papers to sell to libraries + government funding being partly chosen by academics who are also part of the problem.

I think part of the issue is that there is basically no downside to having bad papers. In ML conferences, I think the only paper I have ever seen retracted was because some of the authors had an affiliation with a Russian institution. There are LOADS of bogus papers at the big ML conferences and none get retracted afaik ... The worst that can happen is someone tweets about it and then everyone forgets about the paper.

never occurred to me to write a comment there, but it actually looks kind of clean imho. Really old C code looked like

void func(var1, var2) int var1, char* var2. { ... }

perhaps some legacy from that? Probably not, but just first thing that popped into my head since it feels similar :shrug:

IMO, yes a lot of the "nvidia pays lots of people to make non-portable, tightly coupled backends to open source projects" is potentially going to be less of a moat?

(Though it could turn into "nvidia pay lots of people to use LLMs to make non-portable, tightly coupled backends to _even more_ open source projects")

A recent review of mine had a LLM-ism at the very end, "would you like me to format this into a formal peer review report?" So they very likely copy-pasted their whole review :). I'm pretty down on academia atm ;-;

I can't help but feel like all the tech glasses are basically a way to get more ai training data where they can get like a constant stream of human POV stuff.

it's depressing how small google's / meta's fines are. should be a % of revenue or total stock valuation or something.

I know in electrical engineering / chip design, it's actually quite hard to do "real" research that's useful for industry simply because of how expensive it is to make chips... I think most research for improving LLMs will go in a similar direction.

I've personal almost stopped reading papers in my area, which is in ML but not related to LLMs or CV. I do look for work related to whatever I'm doing, but it's kind of depressing how uncommon it is for (say) neurips papers to actually have anything useful...

(It's also kind of annoying how basically all funding agencies are only funding research into or using AI, but don't provide enough funding for lots of gpu time lol)

Yeah it's very short, but I can't remember if I read all of it, or just flipped through pieces. It has good advice and I think fairly "famous." (edit: this reminded me I still have my copy :-D)

I know someone in a profession that does a lot of writing, and it blew my mind how clearly feedback was communicated by a superior. Made me wish tech people had better written communication skills. ;-;

yeah, it would be almost shocking if an open source benchmark was NOT used ~somewhere in training. Perhaps just pre-training, but still. Neural networks can be fairly robust to some mistakes in their training data, so maybe it doesn't even matter if some of them are incorrect. Who knows.

I agree the combined masters + phd is weird tbh. I think almost everyone treats it like "just focus on research and spend as little time on course work as possible." I would prefer it if courses were more flexible.

I'm not a big fan of the US application setup. IDK how it is in Europe, but in the US, it feels like there's a lot of not-very-meritocratic "secret" stuff you need to know to up your chances.

1. US a lot of international PhD applicants, so traveling before even being accepted is difficult, and 2. lots of people don't do a masters.

when I applied to PhD programs (not in math) it was basically CV + personal statement + recommendation letters + short chats with interested faculty :shrug: Maybe it was because my CV was "strong" but the chats were more see if interests were aligned, rather than actually interviewing me.

I think people probably assume that openai / anthropics use of their data is probably like google's """limited""" use, in the sense that historically google wouldn't trivially be able to just take something from google cloud or someone's search history and insta-convert into some competing project... But LLMs are quite strong at approximately "memorizing", so I think that risk is wayyy higher.

numpy is pretty much all C and python. They may dispatch to some Fortran libraries, but I think basically all the internal implementations they have are C. IIRC, scipy does (did?) actually use a lot of fortran fwiw.

I sort of wish they used c++ for some of the template stuff, especially since the code base already seems to have some C++ iirc. I also contributed a tiny bit in the past, and their C template system was a surprise. (but this is cool and I love numpy anyway lol)

does this sort of fall into the bucket of counter-examples we've been seeing recently? I understand it's a construction causing blowup and that implies that the navier-stokes isn't regular / smooth, so sort of a counter example?

if there's anything that would convince that LLMS are one of the biggest innovations ever, it would be this :-D

guaranteed, up to lean itself having bugs that are exploited by the LLM :shrug: