817 karma · joined December 22, 2018
https://twitter.com/PMayrgundter
https://sites.google.com/site/pablomayrgundter
I think there's an important general principle about the role/capabilities of the external driver/operator that's party addressed here in the blog and video. You can see in the video a somewhat complicated manipulation to step the computer through one cycle and the author says:
"Because there is this feedback loop between memory and control logic, you need a fairly careful timing mechanism so that the output of the control holds steady long enough to set the appropriate state of the memory. That said, this control mechanism is basically missing from my computer. Instead, it relies on the operator (me) performing several different motions in the correct order to advance to the next state (as you can see in the video with me flipping several controls)."
This is importance bc it has to do with the thermodynamics of work. The more intelligent the operator, the less work efficient (less intelligent, so to speak) the computer need be. I'm impressed with the level of efficiency here.. his hand motions are rote and not particularly delicate.
The mechanism I had tried was to push a lego "program" (a flat block with protrusions) thru an "interpreter" (a chute with friction gears to drive a system of switches actuated by the program block's protrusions). This meant the driver was just a "push" of the program, which drove the system. Much nicer in theory, but also way harder and so didn't work. I also didn't have an interpreter of any sort. I had a few-bit display driven by the levers, so it was really just an analog machine.
Anyways, if anyone's playing with this, that's a suggestion for one area to focus.. to simplify the role of the operator to the point where there's no intelligence being used.. ideally just linear or rotational force.
* I was once into Quines and thought I was hot stuff after writing a self-printing program that also played the Game of Life[sg]. At work we had a table of legos, and I foolishly bet some colleagues I could build a self-printing program running on a lego machine. But the the computer is no simple feat itself. I'm now freshly inspired. I'd been exposed to the idea of a latch in school, but wasn't clear until now how important it is for state and control of the interpreter. [sg] https://github.com/pablo-mayrgundter/freality/tree/master/fu...
Yeah, IFC is neat. It's actually a dialect of STEP, so able to carry the geometry and tons of BIM metadata. We've estimated the spec is ~7k pages printed. It's like an abstract toolkit for working with building codes
Happy to chat more too. We have our project Discord and DMs on Twitter
I'm not seeing any IFC files on Wikihouse repos, but they look old (https://github.com/wikihouse). Do you know where they're hosting?
1) bilirubin (from red cell breakdown) from bloodstream excreted into the gut
2) gut biota convert it to urobilinogen via bilirubin reductase
3) urobilinogen -> urobilin spontaneous breakdown (still in gut)
4) urobilin reabsorbed into bloodstream, filtered by kidneys, excreted in urine
If so, why not just leave it in gut for excretion with feces?
or perhaps 2.1) urobilinogen reabsorbed into bloodstream first, then 3). Makes a little more sense
Link to PDF and some screens from intro here..
The bio-inspiration was via Frank Rosenblatt, who is referred to in that article tho yeah, the history is over in his article:
https://en.wikipedia.org/wiki/Frank_Rosenblatt#Perceptron
"Rosenblatt was best known for the Perceptron, an electronic device which was constructed in accordance with biological principles and showed an ability to learn.
He developed and extended this approach in numerous papers and a book called Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms, published by Spartan Books in 1962.[6] He received international recognition for the Perceptron.
The Mark I Perceptron, which is generally recognized as a forerunner to artificial intelligence, currently resides in the Smithsonian Institution in Washington D.C."
Your Juergen page is interesting, tho no direct comment on Rosenblatt there. He does cite the work on this page:
https://people.idsia.ch/~juergen/deep-learning-overview.html (refs R58, R61)
My reading is that a long-known idea, about multi-variate regression, was reinterpreted by Rosenblatt by 1958 via the bio-inspired Perceptron, and then that was criticized by Minksy and others and viable methods were achieved by 1965. When I was taught NNs by Mitchell at CMU in the 1990s (lectures similar to his book Machine Learning), this was the same basic story. Also reminds me of a moment in class one day when a Stats Prof who was surveying the course broke out with "but wait, isn't this all just multivariate regression??" :) Mitchell agreed to the functional similarity, but I think that helps highlight how the biomimicry was crucial to developing the idea. it had laid hidden in plain sight for a century.
Agreed, and I was aware, there has since been criticism of the biological plausibility of backprop.
Your further links with refs to backprop in transformers are interesting; I hadn't seen these. It's clear the term is being used like you say, tho I still see ambiguity of it utility here. Autodifferentiation, gradient descent, multi-variate regerssion etc. are ofc in common use and scanning these papers it's not clear to me the terms aren't simply to a point of conflation. What had stood unique for me with backprop was a coherent whole-network regression. This to me looks like a piecewise approach.
But anyways, I see your point. Thanks!
"For example, when doing the backpropagation (the technique through which the models learn), the gradients can become too large"
But I think this is more of a borrowing and it's not used again in description and may just be a misconception. There's no use of the Backprop term in the original paper nor any stage of learning where output errors are run thru the whole network in a deep regression.
What I do see in Transformers is localized uses of gradient descent, and Backprop in NNs also uses GD...but that seems the extent of it.
Is there a deep regression? Maybe I'm missing it
For Backprop, I'm basing this off the development of the Perception. Wiki supports this and its bio-inslired origin[1].
As for its use in Transformers, if you mean simple regressing of errors or use of gradient descent, I'd agree, but that's not usually called Backprop and the term isn't used in the original paper. The term typically means back propagating the errors thru the entire network at a certain stage of learning, and that's not present in Transformers that I can tell.
Happy to see any support for your claims tho.
Transformers do have coefficients that are fit, but that's more broad.. could be used for any sort of regression or optimization, and not necessarily indicative of biological analogs.
So I think the terms "learned model" of "weights" are malapropisms for Transformers, carried over from deep nets because of structural similarities, like many layers, and the development workflow.
The functional units in Transformer's layers have lost their orginal biological inspiration and functional analog. The core function in Transformers is more like autoencoding/decoding (concepts from info theory) and model/grammar-free translation, with a unique attention based optimization. Transformers were developed for translation. The magic is smth like "attending" to important parts of the translation inputs&outputs as tokens are generated, maybe as a kind of deviation on pure autoencoding, due to the bias from the .. learned model :) See I can't even escape it.
Attention as a powerful systemic optimization is the actual closer bit of neuro/bio-insporation here.. but more from Cog Psych than micro/neuro anatomy.
Btw, not only is attention a key insight for Transformers, but it's an interesting biographical note that the lead inventor of it, Jakob Uzkereit, went on to work on a bio-AI startup after Google.
The studies say they are looking at doses of 10mg/kg[2]. Similar dosing to a pain pill.
On a first read, looks like a partial solution as it only delays onset of venom pathology for a day, tho not sure I'm interpreting that correctly.
Otoh, it looks like the pills can continue to be taken for many days.. so perhaps there's promise for the venom to be slowly metabolized somehow that isn't inhibited. Might also bring down the cost of antivenom if it's now feasible to funnel patients into central treatment over greater distances, so stocks could be better used.
[1]https://pubchem.ncbi.nlm.nih.gov/compound/155815#section=Che...
As for how much greening benefits offset costs, I was mildly frustrated by the downvoting :) and went researching on the economic effects in particular. The only study I can find on CO₂ fertilization is recent and stunning:
"we apply the CO₂ fertilization effect... backwards to 1940, and, assuming no other limiting factors, find that CO₂ was the dominant driver of [American agriculture] yield growth"
https://www.nber.org/system/files/working_papers/w29320/w293...
As they note, the extrapolation is from a study period of 2014-2020, but it's expected they have underestimated the prior role of CO₂ fertilization, as CO₂ response becomes saturated as levels rise and that's not represented in their model.
This is 2021 research funded by DOE & USDA, run by NBER, with PIs from Harvard and Columbia who are well cited within their field of environmental economics. Given the pedigree and that conclusion, this should be front-page news in science mags at least. I couldn't find any reporting on it except for Epoch Times (https://www.theepochtimes.com/science/nasa-satellite-data-su...), which gets an eye-roll from me, but guess they got this one :)
I read the full thing. It's great. Crosses high-resolution satellite CO₂ anomaly observations (NASA OCO-2) with crop yield records, wind modeling, sensitivity testing on non-ag land and the result is broadly repeated: +1ppm CO₂ -> +~0.5-1% yield. CO₂ levels went from ~310-410ppm in that period.
If true, it seems at odds with the normal account that gave nitrogen fertilization the lead role for increasing crop yields this past century and lifting most of the world's population out of regular food insecurity, and the growth of our population into billions. Seems more likely both were necessary.
It also makes sense, as C3 plants (e.g. soy, wheat and most plants besides hot/dry grasses) respond positively to CO₂ partial-pressures up to 800-1000ppm. C4 low-CO₂ adapted corn also fertilizes, just to a lesser extent. The authors also note the negative effects of heat stress, but this wouldn't have been as significant in the extrapolation period.
From Epoch Times: “The paper’s first author, Taylor, said: 'We reiterate that climate change will have a large negative impact on agriculture in aggregate, especially in places exposed to extreme heat. And higher CO2 may even lower food nutrition. But the countervailing fertilization effect should also be taken into account.'”
The overall phenomenon is well documented, so I'll give just one highly credible account: "The greening [over the last 35 years] represents an increase in leaves on plants and trees equivalent in area to two times the continental United States… Results showed that carbon dioxide fertilization explains 70 percent of the greening effect" - NASA Apr 2016, https://climate.nasa.gov/news/2436/co2-is-making-earth-green...
I'm not wanting to rehash the climate debate, nor am I implying how much it offsets negatives and would grant all of your points otherwise. I do think it's wise to do whatever we can to move towards cleaner carbon burning to remove toxic and particulate emissions, and limiting CO2 emissions as well, but also within fair economic tradeoffs. I also support developing nuclear, fission and fusion.
A roll down the line will rotate the circle once. Coming back the same. But rolling around one corner will add half a circumference, and another half for rolling around the other. So you get 2 * 0.5 extra circumferences, and so + 1 C. Somehow that helps with the other polynomials for me too. Super cool.
https://www.theinfinitycomputer.com/wp-content/uploads/2020/...
To the sibling comment about arbitrariness, we could use a hybrid where we trade off some bits from IEEE FP to introduce far reaches and also some precision there.. so like, keep 32 bits or 54 bits for IEEE compatibility, then switch to "extended" ranges for e.g. BB numbers, higher alephs, etc..
There was this one system for calculation with infinities that avoided the Hilbert Hotel problem.. can't find it but was called smth like Infinioid or some other play on the name. Would be neat to bolt those on too :)
Edit: "grossone" is the calculus for infinities.. love this work! https://www.theinfinitycomputer.com/
This was well reported for major digs like WTC/WFC, which created Northwest Battery Park City.
It's also seen in local landmarks like the Ear Inn on West Spring (one of the oldest bars in the city) which has an old painting inside showing the shoreline being not far outside the building, whereas it is currently a full block or two
For more: https://en.wikipedia.org/wiki/Lower_Manhattan_expansion
Good names in there :)
Looks like it used to be brute force, as you say, but has since incorporated an NN after ita loss to AlphaZero
"Early attempts to open the scrolls unfortunately destroy many of them. A few are painstakingly unrolled by an Italian monk over several decades, and they are found to contain philosophical texts written in Greek. More than six hundred remain unopened and unreadable.
What's more, excavations were never completed, and many historians believe that thousands more scrolls remain underground.
Imagine the secrets of Roman and Greek philosophy, science, literature, mathematics, poetry, and politics, which are locked away in these lumps of ash, waiting to be read!"
YT vid says Journey Time: 3 hr 33 min
Google maps says this 3h30m by car MCO to MIA.
Security overhead on the train looks medium, with TSA-style scanners but otherwise normal walk-up access when train arrives.
1h flight (tho add 1-2h for airport overhead)
Price: $79 (Smart), $149 (Premium)
Maybe the specific claim was that wavelength at a resolution good enough to resolve individual stars, or a star per pixel often enough