It could be that Microsoft is leveraging them to bring him back. This board may seem mercurial at the moment, but we really, truly, and honestly still do not have the big picture yet.
1,158 karma · joined December 12, 2022
great for students. better for researchers. best for unhinged scientists who work at the microscopic level.
it's built to be experimented with. many small hacks take 20 seconds or less from ideation to next completed run. very few take over 5 minutes to implement in the right conditions. using the power of science, version 0.7.0 will be coming soon with one less layer (what?!?!)
------------------------------------------------------------
Rest of the bio:
okay so what i really need tbh is someone to take https://github.com/tysam-code/hlb-gpt and make a plot of a trained network using the top n% of the token mass with banded colors representing the lyapunov exponents of the full following sequences (up to about 128 characters or so) mapped onto a poincare disk
bonus if it is clickable so you can advance one token and look at the subgraph for that plot instead
ill handwrite you a very nice thank you note if you do
if you're confused, the above is a plot that would let us explore visually the output space of language models. the lyapunov exponents represent how much information is added, and are calculated by brute force calculating all of the possible sequences of the network to estimate the entropy added per token over each path. this lets us know how chaotic each token branch is. but to save on exploding compute, only calculating the top X% of tokens (by probability mass) keeps the branching a bit lower, and also bonus, we only really see the tokens we sorta care about (maybe this would shoot us in the foot but who knows). because the longer sequences are precalculated, just clicking in a direction would require only calculating one more round of tokens, and you could do a nice animation from wedge to full circle so you're back to where you were on the first step.
super cool and useful i think. and you know you want a nicely written thank you letter. ill even spill peanut butter on the inside if that's in your taste and helps motivate you some. now it's definitely worth it.
And 'anywhere' includes a lot of very scary places.
NAC is quite fantastic. Not just for energy, but for autism too, in which glutamate dysfunction is also implicated! Some people do just as well with sulforaphane as well, which I really need to do.
Jarrow has a great NAC extended release brand, I've been spotty with it recently but it feels really good when I take it. Feels like my autism symptoms go down half a notch (i.e. the opposite direction of aspartame, lol), my energy goes up, and I just sorta feel more 'real', for lack of a better word.
Very great medication/molecule, all around, it does so much more than that, too (for example, NAC is just glutathione without the glutamate component IIRC, and that's how it binds the glutamate out of your bloodstream. 2 huge birds, one spicy sulfurey stone). Let's hope it stays open access, for all of the benefits it has.
Thank you so very much for the recommendation, this is a great one! <3 :')))) <3 :D
So much mitochondrial implication in any post-viral fatigue syndrome, it seems like somewhere in the stack is oftentimes disrupted, thankfully it can be managed though!
Night and day for me. On the Appalachian Trail, I had to count steps and watch my heartrate on the climbs to keep from triggering PEMS (you don't want that out in the semi-wilderness). I had extra food and would take days off when I did trigger PEMS/go mildly hypoxic/lactic acidosis, whatever. Not fun, but boy was it a pristine place to take break days, in the perfectly gorgeous wilds (surrounded by interesting and talkative, friendly people. heaven).
In any case, on the AT it took so much discipline. I switched to keto after getting norovirus because why not on the momentum of not eating for several days, and some of the ridge runners helped me out as well (they are true trail angels I think). The keto really softened my PEMS a lot, I think basically by preventing the lactic acidosis spiral because...yay, no glycogen in the energy cycle anymore. That was my motivation/thought at least, and it did really pay off, it seems (for whatever reason!).
Pyruvate was my second best shot, it would really reverse a lot of problems to a manageable degree so I could do more basic stuff and focus on rest days, etc.
Eliminating free glutamates was _huge_ for me. Maybe not for everyone, but for me, absolutely. This is yeast extract, miso, soy sauce, hydrolyzed corn protein, anything that is umami basically is glutamate or glutamate-related in some form or fashion. For whatever reason, blood glutamate scavengers seem to be well-implicated in reducing symptoms of me/cfs quite well (and fatigue in general? i think?), this is not quite as well-known as I think it should be, but check the research out on that for yourself!
I'd have energy crashes after having, say, beef jerky (which already has glutamate, just competing with the other amino acids during breakdown), which often has tons of soy sauce or the like. Everything seemingly has MSG (or the equivalent), though they can legally lie if they put it in via a different source, apparently.
(Don't get me wrong, I love MSG. But it can make the lives of me and other people with fatigue stuff absolute freaking hell).
All that to say, niacinamide sorta waltzed up and smashed the delicate glass wall that I'd been balancing things around. I'm very grateful for it, it sorta feels like easy mode (minus all of the symptoms lol) compared to the other stuff.
Please feel free to ping me on twitter if you ever want to! <3 :'))))) Happy to chat further about this, and much love! Thank you so very much for your encouraging words, ME/CFS, long covid, post viral stuff is a hellhole of a syndrome as it basically feels like it sucks your soul out of you physically (experientially, at least, lol), and leaves you dead-alive. There is hope, and much CFS does not last forever! And even if it is there, I can assure you that there is a way to enjoy and love life, it just becomes much more of an inward and spiritual (if necessary regardless of the requiring circumstances) journey.
Sorry, that was a lot, I, er, maybe am recalibrating to my energy levels. Also, I'm very passionate about this particular topic! <3 :')))) :'))))) :'))))) :')))) <3
I can have some vegetables (yay salads! I love certain kinds of salads, have to be picky about dressings and toppings though), but at the very minimum meeting caloric needs is the core goal for me.
As far as infection goes, I've had many antibiotics over the years of different classes. I believe it's fungal (mayyyybe with a parasitic co-infection of some kind?) as that anti-parasitics/antifungals are the classes that seem to introduce the strongest response in me.
Also, I cultured some of the skin from my foot and it grew yeast, in great abundance! (not the most reliable test, but it works okay...ish, lol).
Additionally, I had the same systematic spike in pain in my leg when taking niacinamide originally as I did when surface-treating the fungal foot infection, which cleared up pretty rapidly underneath it.
Things like thujone/juglone/etc seem to be quite helpful too, though, so I do worry about a potential opportunistic parasitic component to it as well, as I seem to respond in some way to drugs in that class (even the ones potentially without apparent antifungal activity). Parasitic infections actually are quite common in the US compared to our expectations -- just look at pinworms or toxoplasmosis (even though it is quite low here in the US compared to the rest of the world)!
I think something that I've slowly been finding is hard about epidemiology is that immunocompromised individuals can be subject to a whole host of odd infections & etc in comparison to a standard, healthier population.
I have reactions to nearly every food group out there, even things traditionally safe for many (minus the ultra safe things like rice, lamb, etc).
It's definitely not a digestion/absorption issue, in fact I have a reasonable belief that I have barrier permeability issues as things like taking GABA will result in respiratory depression + temporary oxygen starvation for a few seconds - half a minute or so (yay) followed by that emergency contraction of blood vessels that the body does when trying to get oxygen to organs. That's one good indicator of systemic barrier dysfunction, for example, as that is certainly not supposed to happen in healthy people since GABA is not supposed to cross the intestinal barrier (or the BBB for that matter) in healthy individuals (though I wonder if one tiny benefit is making GABA slightly more effective for me for anxiety over, say, the average person due to potentially-increased BBB permeability). Additionally, things like P-glycoprotein inhibitors (like piperine, etc) cause me to react much more strongly to food/environmental things.
Apples would contain, say, the most pyruvate and malic acid (though I react to apples pretty badly, sadly, I'll get arthritis-like symptoms (which makes typing and manipulating objects difficult, for example), though I take an OTC 5-LOX inhibitor nowadays that is actually surprisingly quite effective at preventing things like that from happening in accidental food exposures. Still not enough to have apples straight though, lol).
So it's things like that. Most of this is things like looking at the Krebs cycle, finding upstream things to targets that I want that seem to perform well (like oxaloacetate, which is sold at an outrageous price), and then putting them through the empirical test pipeline to see if they hold up.
Things like niacinamide for example I'm taking 1.5-2. grams a day which would be hard to get from food, pyruvate is also hard to get from food as well. I love the idea of food as medicine, but unfortunately it's mostly calories in (and thankfully I don't seem to react to multivitamins, thank God).
I've tried several thousands of dollars worth of supplements over the last several years trying to find combinations that work. This is most certainly harder than most of the machine learning problems that I work on, as the loss signal is not all that clear, really (unfortunately)!
I am glad to have found something that seems to have a positive impact, however!
Lyme, babesiosis, rocky mountain spotted fever, etc, all negative. I also did a urine Lyme DNA test a while back but looking back on it, apparently that was not an extremely reliable test (though the western blot has a decent amount of false negatives as well).
I believe it's a chronic yeast infection that's slowly traveled up under the skin of my left leg for the last 7 years or so. It started out as athletes foot in college, and then when I got past the point of being too overwhelmed to treat it, it had set in pretty well.
I wondered if the two were correlated, but over the past few years I've had increasing leg pain in that area, and the same pain in my toe where the infection was. Of course, the nature of this infection is that it seems to be resistant to many OTC classes of antifungals, so it did not budge. The most effective agent was carvacrol/thymol, which is extremely broad range (even against MRSA), but it macerated the skin to a point of strong pain, bleeding, etc, all that jazz. Even when it cleared up on the outside, there was a deep white patch under the skin that you could see under the (healthy-seeming) skin, which, of course, is its own unique class of horror.
About a year ago or so, I finally broke and decided to get rid of the skin infection no matter what. I basically mixed isopropyl alcohol and table vinegar and put it in the skin after removing the top layer of dead skin, which as it was an open wound was extremely painful. I used the reasoning that I could temporarily damage the nerves with overexcitation via the isopropyl alcohol stimulation, and that reasoning panned out after an excruciating several days, after which my nerves were damaged enough to no longer hurt under the raw vinegar and alcohol combination.
That at least cleared up the surface infection until I was on the Appalachian Trail this year (which, yes, I did my two months on it with chronic fatigue + post exertional malaise (!!!!) !), where, more than a year later, the infection inexplicably came back again.
I found niacinamide in my search for treatment-resistant infections, and it cleared up the surface infection within an astounding 2-3 days. I started taking it orally, and became extremely sick, it felt like I was being poisoned and doing a mini-chemotherapy of sorts, so I slowed down and began taking silymarin and NAC to help preserve liver and kidney function (which seems to still remain okay so far).
The process of treating it has been painful, as the initial days caused a huge flare in leg pain in my left leg corresponding with swelling of the lymph nodes almost exclusively on the left side of my body, oddly enough (though it seems to have balanced out, a bit). I was in the emergency room on my birthday last year due to the leg pain, and they couldn't figure it out so they offered a potential umbrella diagnosis of complex regional pain syndrome.
It's hard to keep up with dosing as I do have a bit of an aversion to feeling sick -- it quite literally feels like I'm being poisoned! -- but, my CMP from even this week came back okay, and I'm getting a feeling of underlying 'rightness' despite the transient negative symptoms (including having to pee every 2-3 hours...not fun).
I've had maybe two dozen doctors, both in and out of the ER (6 visits in the last year or so), or so give me that look of not quite being able to know what to do. I applied to the internal medicine program at the Mayo Clinic a few months ago and got a form letter turning me down. It's not due to incompetence, it's just that there is a long tail of extremely complicated conditions that categorically fit an umbrella of symptoms, but are extremely hard to actually track down (and the semi-binarization of medical specialties makes cross-disciplinary diagnosis extremely difficult for these vague kinds of conditions).
I'm sitting here now and can feel the leg pain when I focus on it, for example. But I think treating it is worth it -- especially as one has to work to live in many circumstances! That said, this does not underpin how grateful I am to at least have one avenue -- whether it's temporary or not -- where I can have energy.
This also doesn't cover things like the mast cell sensitivity (diagnosed as not MCAS by _two_ separate immunologists, though they both fall in the tryptase-test-philosophy camp). I react to everything, including All Free and Clear detergent, for example, and with food and such sometimes it's just eating tons of meat (it's pure muscle protein and fat and doesn't have things I can react to like lots of plants do) and multivitamins to cover the nutritional gaps.
I wish it were as simple as just having Lyme disease, as I think that would be much, much easier for me personally. But, I suppose to keep living we have to accept the limitations that we have (and unfortunately this is not the only journey of limitation I'm on -- I also have autism to a mild-moderate degree! That would be enough challenge for a normal lifetime as it is. D'''':), and I had a number of years where I really struggled (like, really really struggled). But all of that said, I think although it's taken a few years, I've come around the bend in starting that process of eventually accepting the challenges I've been given, and I am on currently on that particular upward walk of learning to enjoy the life that I do have access to in the meantime. It's not perfect, and I have a lot of big bumps in the road, for sure, but the slope for me is pointed up, and that's what I wished for for years. And I'm enjoying life a lot more than I have in the past, on average! :')))) <3 <3 <3 <3 :'))))
Hope that answers your question.
Title: "I created the ultimate Chess Ai (it can cheat)"
N-acetyl Cysteine and other blood glutamate scavengers (BGS) like malic acid and pyruvate are indispensable in these scenarios. They don't solve the issue but dampen it a bit.
Additionally, a ketogenic diet helped me a lot.
Most of all, high dose niacinamide holds it in remission at times, though I have a theory it's caused by a well-set-in, chronic infection as the reduction in symptoms with niacinamide correlates with the symptoms of fighting off an infection (very swollen lymph nodes, histamine release, sometimes nausea &etc, headaches, some other clear indicators, etc). I've been on it for about 7-8 weeks or so and we're still going!
That said, having energy is a gift that is hard to quantify. Chronic fatigue takes away your ability to think about anything, so you have to have discipline to not think about anything sometimes...which also takes mental energy. It's a bit of a living hell, for suresies.
Here's hoping I get to stay in remission. <3 :')))))
The version with the Weaver introduction is quite good as well, there are other versions of similar papers covering the topic from different angles, I find it to be well-worth the read.
For more information, please see https://people.math.harvard.edu/~ctm/home/text/others/shanno...
I mean, yes. One distills information from a training set into a compressed representation, and the other generates a compressed representation that yields (more or less) fixed state space attractors. It's just inducing a bias over the state space of the network, nothing incredibly special, though I'd consider the initial stage of training to be the most important, as it is responsible for all of the ingest of all of the embedded information the network will be using during autoregressive inference (especially w.r.t. the context of doing so during longer generations).
So the notion of 'in context learning' is one I find to be a bit of an illusion, of course, as no actual learning is being done, just the induction of biases, which appears to give rise to a transiently-'better trained' network.
You could see this as a bit straightforward perhaps, but I feel it needs to be said.
The entropy gotta get slung around somehow.
Best to simply scale log-likelihood-based training, next-token-based training trivially contains a requirement for learning all of the subproblems that predict said, next token, and hardcoding something to get warm human fuzzies would be creating a biased estimator (and move us back towards the 90s a bit).
Models already very constantly do context-dependent token utilization, it's an autoregressive feature based on the entire stream of incoming tokens. Humans have a bias to focus on the 'last token used', this is not what language models look at.
It seems like you're referring to an associative Hebbian/Hopfield-like lookup, which the current 'dumb' optimizers already do. Better yet, the learning rates are normalized by the diagonal of the empirical Fisher so that the learning w.r.t. to some estimated expected information is more constant, for said associative lookup operation.
Additionally, the training loop (which you call 'dumb') is just...a teacher-forced version of inference, which you call 'smart'?
It's better to simply minimize the log-likelihood in a scalable way. Hand-engineered solutions rarely survive compared to strong-scaling ones.
"A number of investigators in the first half of the last century reported prolonged EPOC durations and that the EPOC was a major component of the thermic effect of activity. It was therefore thought that the EPOC was a major contributor to total daily energy expenditure and hence the maintenance of body mass. Investigations conducted over the last two or three decades have improved the experimental protocols used in the pioneering studies and therefore have more accurately characterized the EPOC. Evidence has accumulated to suggest an exponential relationship between exercise intensity and the magnitude of the EPOC for specific exercise durations.
...
Notwithstanding the aforementioned, the earlier research optimism regarding an important role for the EPOC in weight loss is generally unfounded. This is further reinforced by acknowledging that the exercise stimuli required to promote a prolonged EPOC are unlikely to be tolerated by non-athletic individuals. The role of exercise in the maintenance of body mass is therefore predominantly mediated via the cumulative effect of the energy expenditure during the actual exercise."
Good re-testability, and they're part of what they use for a formal diagnosis.
oddly enough, autistic people can have an extremely heightened sense of empathy and moral justice compared to the average population. this seems to me to personally have some very distinct upsides and downsides.
Bluntness != correctness.
We have the normal emotions and human flaws, they're just expressed differently.