deerstalker

Not strictly on topic but related to this -- I wonder if there is any effort/research being done with LLMs when it comes to finding low dimensional weight space representations/manifolds. If you could drastically lower the dimensionality with some algortihm X, you could efficiently use global optimization algorithms to find sparse but accurate networks (Lottery Ticket Hypothesis etc etc.)

National Labs in the US have been doing this for a while now. I feel like the private sector will take the lead soon.

I have been doing some research on this topic, and found that for some budget regimes (really expensive objective function evaluations) and some applications (HPC code parameter autotuning), the frontier LLMs can even outperform classical optimizers. Even open-weight models can perform well on certain applications but one some they fail abysmally (Of course this is limited to a bunch of niche applications).

Very cool. Evolutionary Algorithms have kinda been out of the mainstream for a long time. They are good when you can do a lot of "black-box" function evaluations but kinda suck when your computational budget is limited. I wonder if coupling them with ML techniques could bring them back.