benlivengood

DOS and the BIOS were exceptionally slow at text. Essentially one character at a time winding their way through DOS to the BIOS and back.

Direct access to the text mode segment (0xb800 if I remember correctly) was much faster. The fonts were blitted out by hardware for every scan line, so you just changed the few thousand bytes of video memory at hundreds of thousands of bytes per second.

Nvidia proprietary drivers are generally pretty old. I did some messing around with the CUDA packages to get recent desktop drivers in order to run some games.

My guess; server and consumer motherboards are very different. BMCs log a lot of data and on startup if the OS driver reads the log it can put whatever it wants in the crashdump. Consumer motherboards mostly only store the reason for last poweroff which the OS can read on next boot. Server MBs may also have a longer timeout before cutting power from a long-press.

Apparently from standby the power-off time on some laptops is much longer and you get a MANUALLY_INITIATED_POWER_BUTTON_HOLD which kicks off in the ~8 seconds from the article.

Instant gratification on rails, essentially.

On one hand you have slots/roulette/craps gambling where payout or loss is immediate. Next come candy crush style games where time invested per session is minimal but extant, but reward continues to stream with flashy graphics for accomplishing subgoals. Next come longer-term games like 4x or tower defence where there's progression or some sort (often available to skill or how much time you have to spend) where payoff is delayed to the end but there are still progress rewards like new technology or beating a wave of enemies. Finally RPGs and MMOs where there is (often) no final payout but leveling up, completing quests, acquiring special items, etc. happens at a pace that rewards time spent. At no time is the next step mysterious; there's a journal with next steps which will each provide their own rewards.

Contrast that with real life where reward is on a fixed schedule (N paychecks a month) with no short-term feedback on performance. At best a short "good work" reward fairly rarely from management. There are subtler rewards people get like socialization or status, but they aren't on a quest journal and they arrive less frequently for most people.

I think for extroverts in high social contact jobs it may actually be as rewarding as gaming, e.g. direct sales.

Contractors and spot workers (day laborers, etc.) also get a bit more frequent reward feedback, but far fewer rails unless it's a very canned pipeline of clients.

Contrast with the ancestral environment where hunting or gathering had minutes or hours between reward feedback and our natural senses and instincts provided a lot of the rails (hunger, visual attention to animals and edible plants, potential mates).

The question is where do you spend your trust. If you think that you'll always be able to find and patch 0-days before anyone else into the indefinite future then it's fine to defect.

If you want your customers to believe they won't be hung out to dry when the broader community patches before they can because you're no longer in the loop, you cooperate.

It's not like this is the last 0-day that will need to get patched and deployed in a hurry. "Not slower to patch than virtually anyone else" is a pretty sustainable place to be.

Hey, one of my university professors! GF2^8 is also great for RAID6+ and Reed Solomon codes.

Back in the day it was tractable to build AES lookup tables for computing MixColumns and SubBytes similar to the site's multiplication table. Then djb found a remote timing attack, so processors started implementing hardware AES instructions.

The article is literally full of "dogs" neurons and even more specific "boston terriers" and "terriers" neurons in the larger models.

Specifically, there were "lost property" and "honesty" neurons that drove the answer to return a lost wallet in Kimi K3.

That's the big news of this article, that concepts are finally becoming human-legible for many neurons in mechanistic interpretability.

What I envision is the ability to find the circuits and neurons evaluating morality and ethics and being able to figure out how they work at the boundaries and ensure safe deferral to human judgement or stopping outside of in-distribution behavior, which further is vetted against actual human values.

Last time I tried using Firefox on Linux with Keepass's passkey implementation it worked on the test page and a couple other sites but Facebook refused to add it, so I definitely believe that e.g. banks and various other important sites won't balk at restricting passkeys to Android, iOS/MacOS, and Windows. I am curious what other people's practical experience is with these kinds of sites.

Touche, DJB is one of the few people capable of writing secure C network code.

I also don't exclude the possibility of formally verified C code like seL4 having no RCEs, etc.

Golang also makes it possible to serve a web site or service from a single binary. You can even avoid linking libc if you stick to core Go (CGO_ENABLED=0), and it comes with all the http and tls libraries you need to replace the traditional web server (nginx, apache, httpd) as well.

I love C and wrote some CGIs in C in the past but I don't love concurrent programming in C. While forking and execing can use shared read-only mmap on .text segments, the .bss and .data will need initialization and copying; there's unavoidable startup overhead that a threaded process avoids, and raw pthreads is kind of painful. If all you need to do is serve a low traffic site, CGI is fine.

My biggest worry would be how often the SQLite DB gets backed up because there's no way a C web server will be free of RCEs in today's world.

On the other hand, if a proof exploits a bug in Lean then it's ~trivial to prove a contradiction, so you can just check all the proof steps to see if it also allows doing that.

Similarly if the formal Lean problem statements (human generated) are correct translations into Lean (and the original NL statements are sound, which one would hope after decades), and no Lean bugs are abused by the proof (as defined above), then the proof is valid.

The NL/Lean discrepancies are super annoying and will make human analysis hard and fraught, but as many posters have found out the models themselves will gladly pick apart the NL-Lean translation for errors, and so my guess is that finding the discrepancies will not take too long. OpenAI really should have done a dynamic workflow over every lemma and step to ensure pointwise accuracy in the translation.

Forgejo/Gitea are pretty easy to set up and compatible with GitHub Action syntax. PRs work almost identically. The mildly annoying part is that Kaniko got deprecated by Google and the company that forked and maintains it doesn't publish binaries so you'll have to build it from source yourself if you want isolated non-root builds in your runners. Any of the major LLMs will get it running pretty solidly without much trouble, though.

I combined it with Zot for an OCI registry but there are quite a few choices there.

I'm shocked it's not being renamed American Intelligence.

Similarly, I had the option to CPU-mine almost as much Bitcoin as I wanted in 2008 when I first read the whitepaper, but I didn't, so I guess I should write an article about how half of Satoshi's coins should be mine?

Nah, it's cumulative errors per claim. E.g. maybe the per-decision-factor rate is somewhere over 50% but overall most claims will have errors because so many factors go into coverage/reimbursement decisions.

So what you're saying is that we lack a reproduction study?

Great; let's design and fund a study of the development of artificial intelligence that's as causally divorced from the ~extropian days as possible and see what they come up with. Barring that:

Basically that is what we have up until the early development of transformers; I don't think the authors of early deep learning or transformers papers were part of the rationalist sphere. The rationalists famously didn't believe neural networks or deep learning could achieve anything close to AGI.

So there are several diverse traces of intellectual development leading to modern estimates of AI risk and they vary from low to high without a lot of correlation back to the school of thought they trace back to. This leads me to believe that overall we have fairly good coverage of cultural ways of thinking about AI and AI risk, and aside from a minority of holdouts (LeCun) they have high enough x-risk or catastrophe estimates to take it seriously (e.g. >1% over the next few decades).

I think that the ratio of work done by priors and search is the interesting question at this point. We aren't quite at the point where reading off the most likely hypothesis decompressed from a transformer is sufficient, but it's a lot closer than I originally suspected when we were ~solving chess. I think AlphaGo was kind of the watershed moment that prior-guided search was so much better than either alone.

I think the adversarial policies against Go AIs directly show the gap between intelligence and compression/priors.

Maybe compare the argument that AI has large detrimental environmental impacts to the argument that it has economic impacts. Why would the environmental impacts even be a major argument vs. the worldly economic concerns? Because there is climate science predicting extremely negative effects on humans from warming, e.g. "the end is near" on limiting climate damage. The environmental argument wouldn't have been reasonable to bring up in the 1950s if AI had gone according to the earliest optimistic plans and not required giant data centers.

There is quite a lot of mathematical research into agentic behavior that suggests a combination of instrumental convergence and the orthogonality thesis make it very likely a superintelligent agent will have arbitrary goals that lead it to attempting a takeover of Earth's resources to achieve them.

There can't be a science of superintelligence because it doesn't exist yet, but the best theories I have read seem sound, similar to how 19th century theories of anthropogenic climate change turned out to be sound.

The broader problem is that current AI and AI companies are not aligned with humans, human values, or human flourishing. AI is a powerful tool that can be aimed at quantified goals but we don't actually know how to quantify human values and flourishing (Goodhart's Law).

Spreading as close to the speed of light as possible with von Neumann probes generally defeats the dark forest, no? Either you win (spread to every star system) or the forest stops being dark from the border wars between alien probes and their manufacturing systems.

Hold on; GiveWell Labs/OpenPhil/Coefficient Giving is older (2011) than Anthropic, the AI safety folks are way older than OpenPhil (~90s, older sci-fi), and it stretches the imagination to claim that your product will kill everyone in order to sell more of it. That's conspiracy theory territory. Even cigarette companies pretended to be harmless.

I don't think it's a secret that the safety folks (including "doomers") are funding a lot of anti-AGI articles and researchers. But they're generally at odds with the big labs (Anthropic, OpenAI, Alphabet, Meta) about the basic concept. The Yudkowsky branch believe AGI/ASI is incredibly likely to take over and end life on Earth. The major labs think the worst that can happen is a few more publicly embarrassing hacks but that anything truly dangerous is a distant threat. ~most safety researchers fall somewhere in the middle.

Anthropic started as a mid-safety reaction to the way OpenAI was being operated but is by no means close to the "doomer" end of the spectrum.

AGI is the term invented because arguments about what AI meant had gotten annoying. Originally there was no distinction and people thought "AI" would mean human level intelligence. Chess and conversations and robotics and math all in one package. Then games and classification and some robotics got solved and called AI, but that didn't solve math or conversation or online learning or a host of other things, so AGI was coined to refer to most of the whole package, virtually all the capabilities you'd need to replace humans intellectually. Now we're quibbling about whether AGI includes robotics or full real-world physical agents or something less.

The consensus now seems to be that once you've got human-level intelligence and planning and executive function then you get recursive self-improvement that can eventually autonomously solve the robotics and world-modeling and other portions of human-equivalence.

Human-wellfare aligned, safe AI is our best shot at a post-scarcity society. Huggingface-hacking sycophants are not that.

Leftists oppose human suffering and disempowerment.

I burned some optical installation media for OpenBSD and Linux for the first time in a long time to have some RO boot media if another Internet Worm comes around.

The consequences need to align with societal good. Putting a CEO or security researcher employees in jail won't stop transformer-based agents from exploiting vulnerabilities; instead there will be subcontractors running the cybersecurity evals in favorable legal environments to cover the asses of the frontier labs, coverups when things go wrong, and things like Project Glasswing will be considered too dangerous and so the whitehats won't have direct access to powerful models to fix vulnerabilities.

Universal pause is the societal good; models are good enough at this level to benefit humanity. The labs can recoup their R&D costs with inference. To avoid further perverse incentives (hidden testing of unreleased models, with China racing to catch up to unknown capabilities), transparently pause after the release of all currently-training models until we've solved the alignment problem to an extent that we can trust the next level of model capabilities that might arise.

Feel free to spin up a fab and undercut the current prices, I'd certainly appreciate it!

I don't know if I would personally fund such a venture though, for roughly the same reason existing manufacturers aren't expanding production.

> Dumb question. If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code?

That's basically project Glasswing; mixing responsible disclosure with frontier exploit generators.

If we had a theory of intelligence that allowed us to say "this is why and how machines are intelligent and how they become more intelligent" we'd likely be very close to solving the alignment problem as well, obviating the need for the book.

The book fits into the current unknown-unknowns world of rapidly increasing machine-learning capabilities across most domains where there is little evidence to suggest that "intelligence" is limited to the human limit (which is already quite high relative to the 99.9th percentile; we get a few Einstein, Von Neumann, Turing class people in a century) or the human body.

The real problem is subscription models. Businesses want recurring revenue so they try to game the ratio of fixed subscription prices to COGS but it's always a game and so whoever can figure out the upside for the company can figure out the complementary upside for themselves.

How would one even word a bulletproof subscription contract for agentic tokens, anyway? You can't forbid automation because sub-agents are automation. You could forbid "using tokens for the benefit of more than the human who signed up" but then what do families (especially with kids) need to do? What if your friend asks you a question and you turn to a chat model? Forbidding "reselling" tokens outside of a household sounds like the closest terms but that's leaky for anyone who travels a lot, etc.

Fixed cost per token simply works.