sometimelurker

> Where in nature has anything ever grown exponentially in perpetuity?

you could make the argument for human total GDP, which looks like https://ourworldindata.org/grapher/global-gdp-over-the-long-..., but then again, you could say we just haven't reached the sigmoid in it. I personally doubt we will

LLM + scale = more intelligent, this can be proven more than empirically, https://arxiv.org/abs/2409.15318 shows that neural nets can fit a number of independent AND-gate operations in their weights,

- if you have a system that is large enough to store, lets say 10^12 AND gates (all frontier llms can do this) - and this system can produce outputs based on previous things it has outputed

its turning complete, and RLVR on it is optimization over the space of algorithms. If an algorithm exists to do a task, and the task can be verifiably done, this finds the algorithm to solve the task most often.

it is obvious that this scales, from much-worse-than-human to slightly-worse-than-human, therefore it 100% can exceed humans.

> defeat any nations that don't discard humans

AGI doesn't do away with nuclear MAD, it just messes with economics and makes many people temporarily jobless. Temporarily because in a literal sense RLVR needs verification to train off of, and a lot of jobs cant be easily checked if theyre done. this includes AI safety people, preschool teachers, psychologists, and probably a lot more, including most of their bosses

> Ensuring Chinese compliance would be incredible hard to enforce or to check

consider SALT and other treaties from USSR vs USA cold war. they checked

> But every exponential is at a certain point a sigmoid.

strongly, no. its just hard to distinguish them. for example, radioactive decay. cmon

> Governments exerting this much control will only end in war

elaborate. regulation -> war, how?

we should never make ASI, what I am saying is strongly sane. ASI = not good. AGI, ~human brainpower, can be made safe enough.

> Are there examples of where we have collective decided not to pursue knowledge? Successfully?

human GMO, some bioweopns, I'm sure theres a long list of awful stuff no one wants to exist.

source needed. I'm not saying ur wrong, but source needed

all easily varifyable tasks can now be solved with money. this is worth paying attention to. math proofs are verifyable -> math proofs are easy now. you can think of other such tasks: cybersecurity, AI R&D/RSI, killing people, 3d-printing helpful tools, maxxing-out human health, manipulation, self-driving cars, anything that can be checked

all jobs in the future will be those can not be easily verifiably done. if you need a team of people to decide if you have been productive, and those people cant be automated, you're in luck.

people use different models. I use small local ones for instance

Can they add hackernews to their study analysis? And do a measure over time?

why is this better than eg matplotlib?

strawman fallacy, I can debate anti-subscription arguments much better. I don't think subscriptions have to be bad, but this is unreasonable

reminds me of https://www.lesswrong.com/posts/qEbqPitYhWHthwFNu/bun-s-migr.... remember anthropic owns bun

I like the quote at the end

"we cannot choose to become idiots"

I like stuff that takes a statistic that we view one way and explains how to see it another way

in this sense, related https://danluu.com/p95-skill/

linux is gpl, not dead

'X-risk/S-risk' is a superset of 'hierarchical world with zero social mobility' and 'ASI taking over the planet is ridiculous'. x-risk=bad, smart enough power-seeking llms automatically lead to x-risk

Probably has something to do with smart technical people being attracted to free software, and then OSS, and then copyleft. the idea of writing code under gpl feels pretty left to me.

coordination problems. everyone (mostpeople) wants action to be taken, but it requires coordination

> CO2 has been rising

it rises faster now because we have more co2-emitters today than we did 20 years ago. source: economy has grown. idc if this sounds rude, but honestly you shoulve guessed this

In pretty much every single HN post ~~on this topic~~, there are a number of commenters that are probably bots. Anyways it makes sense from a theoretical standpoint that LLMs should be able to find flaws in code better/faster than humans eventually, and its reasonable to think that time has come

I like the metric of tok/joule a lot. it really brings to mind a lot of really nice ideas about energy and work and ideas and thought and efficiency

one option Ive liked for entering flow state for this is running a small 3b param model and using it as an alternative to stackoverflow or websearch while I code.

I don't often do this, mostly bc I don't care too much about most of the code I write and just want it to work, and then I skip over it and see what's missing and tell the LLM to fix it

llama.cpp + pi.dev for me, using small Gemma and qwen llms

providers do not want power-seeking LLMs. no one does. this (bad personality) is incentivized during training, especially RL, and is something they would rather not have. tell me, do you think training a power-seeking ASI is a good idea?

ok, that is evil because this would inflict pain on people to no overall benefit. this is equivalent to saying "prisons should torture the worst criminals", which would be insane

> Working there is a choice

look into non-competition clauses, there's a real chance its not. also people who work there are not somehow vastly less moral than those that work at Google ect

it would be so easy to rent GPUs in europe and use unbannable encryption (tor,i2p if needed) to get inference from them. that remaining as an option makes banning local AI pointless. if all else fails I'm sure china will be able to sell to NK some GPUs and we can get inference from them lmao

I'm gunna say something crazy sounding that might be insensitive but is really obviously true to anyone that thinks about it for a few seconds. Meta throwing away a ton of really good talent is probably better for the world by a large degree. It is better if less very smart people work for them. We do not need more tracking algo's and unsafe uninterpretable AI (search 'meta coconut latent reasoning paper') made by a company that's fine with using the information of when teen girls delete selfies to sell them ads for beauty products. We don't.

I think googles been putting more of their money into super long context stuff, not powerful coding stuff. they've got 10million token windows for some internal stuff (Gemini 1.5 Pro Test), more research says meta is also in on the long context game (Llama 4 Scout)