Yep, loss = residuals*2 + lambda*sum(|a_i|), and we take the gradient of that with respect to the a_i to guide our gradient descent steps.
898 karma · joined May 24, 2024
meet.hn/city/us-Boston
Yep, loss = residuals*2 + lambda*sum(|a_i|), and we take the gradient of that with respect to the a_i to guide our gradient descent steps.
Matching pursuit is essentially a greedy algorithm, if I recall correctly - please do correct me if I am wrong, where you conceptually find the component that explains the most data at each iteration, remove it, and then repeat the process on the residual data. Pardon if that isn’t quite the right explanation, but it’s what my intuition is recalling right now…
What I was describing was a simpler algorithm that can be done with gradient descent or any other vanilla optimizer.
Your model parameters are the coefficients a_i over all basis functions in the frequency domain representation. Run them through the synthesis function to get a time domain signal, and select the values of the time domain signal where your target data is known. Compute the squared error at each of those locations, and take the mean. This is your reconstruction error, and should be trivially differentiable with respect to the coefficients a_i. Compute an additional error term which is a standard L1 regularization, ie sum(|a_i|), which can be added to the reconstruction error term with some weight λ (λ=1 is even fine here, at least for simple problems), and then is also trivially differentiable (provided you haven’t initialized any of the coefficients to 0). As with any L1 regularization term, the resulting solution should be sparse in the L1 regularized parameters (look up visualizations of problems with only 2 model parameters to see how this emerges from the L1 contour lines of equal loss forming “diamonds” with the points on the axes).
The classic learning example is to take a couple of sine waves at high frequencies, add them together, sample the summation at a very low frequency (so that the signals are way above the nyquist sample rate) or better yet just randomly at a handful of points, and then turn it into an optimization problem, where your model parameters are the coefficients of the DCT of the signal; run the coefficients through the inverse transform to recover a time domain signal, and then compute a reconstruction error at the locations in the time domain representation for which you have observed data, and then add an L1 penalty term to request that the coefficients be sparse.
It’s quite beautiful! All sorts of fun more advanced applications but it’s quite awesome to see it happen before your own eyes!
My personal signals & systems knowledge is pretty novice level but the amount of times that trick has come in handy for me is remarkable… even just as a slightly more intelligent data imputation technique, it can be pretty awesome! The prof who taught it to me worked at Bell Labs for a while, so it felt a bit like a guru sharing secrets, even though it’s a well documented technique.
I’m also curious to think about how the “time travel” might interact (if at all) with (1) basic algebra and (2) building off that, systems of equations and (3) building off of that something like implicit/crank-nicholson finite difference schemes.
Without having thought about it much, it feels intuitively like there might be an elegant way of expressing such algorithms using this notion of information traveling backwards, but perhaps not.
As an aside, although I’m not really a fan, the representation of time travel in Tenet seems like a natural analogy to include in your docs for fun if desired. More relevant, talking about things like backprop in standard ML packages might be one way of providing a shared reference point for contextualizing information flowing backward through a computational graph.
Finally, echoing what others have said about not burying the lede in the docs, as well as wanting to see more motivation/examples/discussion of interaction nets.
As far as I can tell, I did not… this is what I said:
> it’s very interesting to see how a practice that is fundamentally about exploiting generally unintended mechanisms within a system design still develops its own notion of ethics, legitimate and illegitimate exploits, etc
Removing extra clauses:
> it’s very interesting to see how a practice […] develops its own notion of ethics, legitimate and illegitimate exploits, etc
Sheesh, didn’t expect to re-litigate WS‘17 on HN today… I concede that there was at least some gray area in the codified official rules and the 2001 directive around electronic transmission, but it very clearly was a violation of the norms, IMO, and the Giants-Dodgers binocular incident in the 50s and its reception is good precedent.
> There is no “own notion” of ethics here, and it’s presumptuous and incorrect to suggest that exploiting games during a game exploit competition somehow reflects on ethics and makes the cheating “interesting”.
That’s not what I was suggesting was interesting to me. What interests me is the social phenomenon that is what the actual members of the community deem to be acceptable or in this case unacceptable, how the the community works to identify violations of norms, and then handle with the fallout.
Yes, you are mostly right, but it’s still interesting (and positive!) that the community landed on an “anything goes so long as you are honest” set of norms.
As someone without much (well, any) interaction with speedrunning communities, it’s still fun to see how other internet communities have codified or tacitly established their operating procedures, expectations, norms etc.
That is, precisely, a set of ethics!
> There really isn't an "ethics" of what exploits are allowed or disallowed.
It sounds like there is: disclosed exploits are allowed, while non-disclosed exploits are disallowed. This is very clearly a set of ethics. It is different than that of industry, where generally speaking trade-secrets and non-disclosure of methodology is often considered critical to business success, short of establishing protected IP; in academia, disclosure is putatively the norm but there is plenty of partial disclosure of methodology rather than complete… etc.
> The problem isn't the part about what exploit was used; it's the part where he lied to the community about which category it should be classified.
That puts it very precisely, and also highlights his unethical behavior in the context of speedrunning’s norms around ethical representation of achievements.
Edit: children have helped point out the precise notion of ethics - no exploit is unethical on its face, so long as it is disclosed, which allows the speedrun to be correctly classified and compared to others in its class.
We can attempt to think through how things will play out, given what we already see happening and simple assumptions about power dynamics. People will live through it all, especially the wealthy (cutoff threshold: unknown; or rather, where is the elbow/knee between suffering and resilience? I don’t know) - some of whom will experience some degree of climate shock (cf. the palisades, Malibu etc for a recent example) but will largely find ways to continue exploiting their advantages to survive, even thrive, while others will suffer the incrementally normalized dangers of simply being alive, like frogs slowly boiling in someone else’s kitchen. People will keep fucking, having kids, trying to work shitty jobs, trying to work fun jobs, doing science and research, making music and bad reality television shows and brilliant novels and bad novels, crossing continents for better lives (whether that’s economic opportunity or merely to escape climatic incompatibility with life) while others resent them for it, and people will continue to post on Hacker News long diatribes lamenting the state of things.
I do think it is accurate to say we are *doomed* to this in the sense of being *fated* to it though.
I guess what I am saying is that to call what will happen a doom horizon is just a hood we put over our own eyes because it is too upsetting to really think with nuance about the most probable trajectory.
One of the other easy ways to overcome it is to provide as many templates as possible for journals. I’ve used LaTeX for years, but would by no means consider myself an expert in LaTeX, as I’ve almost exclusively been able to grab a template from a journal or from my university, and then just draft in the relevant blocks, write equations, add figures, and, rarely, add a package. I would guess that there are a huge amount of LaTeX users like me out there. I do all my drafting on Overleaf. I love TeX (and curse my PI whenever he requires that we use Word/365 instead of LaTeX/Overleaf)… but so much of the benefit, for me at least, comes from the fact that templates are readily available for any journal I would want to submit to; my masters thesis was built in a template provided by my university; etc. I don’t have to deal with any of the cognitive overhead of styling and formatting (except for flowing the occasional figure) and can just focus on drafting.
For me to even consider typst, it’s pretty much a requirement that there is some degree of template parity actively being worked on. The most natural way to approach that would be to just sort every journal by impact factor and start working top to bottom; given that so many journals share templates due to being within elsevier, springer etc, it should be straightforward to reach a reasonable degree of parity relatively quickly.
Getting the major publishers to support and offer their Typst templates would make me try it out immediately for what it’s worth.
That’s probably a feature not a bug (from OpenAI’s perspective…).
My point is that assessing 98% reliability qualitatively depends on whatever kind of baseline system you are comparing against.
Obviously this reliability (and speed) of an LLM with 98% accuracy is atrocious compared to any basic automated arithmetic computation system. If you need to reliably add together 100 digit numbers, of course an LLM is a bad choice.
However, comparing it to the general class of systems which an LLM belongs to - ie “things that can add 100-digit numbers together quickly and also tell you about this history of the Ottoman Empire and a bunch of other stuff too, all within a relatively short amount of time” - this reliability is (potentially) impressive. We don’t have many such systems - a moderately educated human; a generic bespoke piece of software which process natural language requests and attempts to decide whether or not to retrieve information from a database/the internet or write and execute code, but could conceivably have been written in a crude manner 5 years ago; and LLMs (which might also take advantage of tool use as suggested in the previous example).
In this context - comparing to other systems with similar capabilities - the 98% reliability might be considered impressive, especially when you consider that it likely is much higher for more common simpler arithmetic problems.
My intention in bringing up “an average HN user” was to bring up an example of a reasonable benchmark to compare to, in my opinion. A HN user is a stand-in for a moderately educated person, which could reasonably respond to a variety of generic natural language requests in a short amount of time. My point was simply that adding together 100 digit numbers reliably and quickly while also being able to chat about Shakespeare or Kalman filters or planning a wedding is likely a more difficult task than we tend to give credence to, due to our familiarity with specialized systems which do arithmetic extremely well and extremely quickly.
This is not about comparing to reliability “of a computer” but rather to a process which responds to a wide array of natural language queries (both human processes an algorithmic processes)… and obviously the reliability goes up once tool use is included.
For instance, my ML work is almost entirely in the context of engineering simulation regression/surrogate development, where data quality/cleaning is almost no issue at all - all of the work is on the dataset generation side and on the model selection/training/deployment side.
Every job is different!
This is your most accurate/relevant point:
> All of this is ignoring the fact that it’s easy to create a tiny personal heated environment around an individual (it’s called a woolly jumper).
Whereas this is plainly wrong:
> It’s much easier, and consumes less additional energy, to heat an occupied space, than to cool it.
And then the following is correct but the marginal reduction in load is minimal except in relatively crowded spaces (or spaces with very high equipment power densities):
> Thanks to the fact that your average human produces 80W of heat just to stay alive.
The truth is it is generally easier to cool not heat when you take into account the necessary energy input to achieve the desired action on the psychrometric chart, assuming by “ease” you mean energy (or emissions) used, given that you are operating over a large volume of air - which does align with your point about the jumper to be fair!
Generally speaking, an A/C uses approx. 1 unit of electricity for every 3 units of cooling that it produces since it uses heat transfer rather than heat generation (simplified ELI5). It is only spending energy to move heat, not make it. On the other hand, a boiler or furnace or resistance heat system generally uses around 1 unit of input energy for every 0.8-0.9 units of heating energy produced. Heat pumps achieve similar to coefficients of performance as A/Cs, because they are effectively just A/Cs operating in reverse.
Your point about a jumper is great, but there are local cooling strategies as well (tho not as effective), eg using a fan or an adiabatic cooling device (eg a mister in a hot dry climate).
> So cooling a living space is always more costly than heating a living space.
Once you move to cost, it now also depends on your fuel prices, not just your demand and system type. For instance, in America, nat gas is so cheap, that even with its inefficiencies relative to a heat pump, if electricity is expensive heating might still be cheaper than cooling per unit of thermal demand (this is true for instance in MA, since electricity is often 3x the price of NG). On the other hand, if elec is less than 3x the cost of nat gas, then cooling is probably cheaper than heating per unit of demand, assuming you use natural gas for your heating system.
In general, we already have plenty of mechanisms for specifying interfaces/api specs, tests, relationships, etc in a declarative but more formal manner than natural language which probably all work , and I can only imagine we will continue to see the development of more options tailored to this use case.