HNHacker News
TopNewBestAskShowJobs

aezart

273 karma · joined July 29, 2021

submissionscomments
aezart··on The Reddit blackout has left Google barren and full of holes
Not all pages are cached. If a cached copy exists, it'll be in the 3-dots menu.
aezart··on The Curse of Recursion: Training on Generated Data Makes Models Forget
That's true but I think you're overestimating how careful people will be in their curation. There are tons of LLM-powered bots on twitter and reddit already, and no one is bothering to delete the output.

Also, the curation process counts as "real human input" itself, so I don't think it contradicts my point.

aezart··on The Curse of Recursion: Training on Generated Data Makes Models Forget
It's not the most useful outcome for the end user, but it's the perfect outcome from the perspective of the learning algorithm. All these things care about is minimizing their loss function, where loss is deviation from the training set.
aezart··on The Curse of Recursion: Training on Generated Data Makes Models Forget
The goal of an LLM, before RLHF, is to accurately predict what token comes next. It cannot do better than that. The perfect outcome is text identical to the training set.

Let's say your LLM can generate text with the same quality as the input 98% of the time, and the other 1% of the time, it's wrong. Each round of recursive training amplifies that error. 96% accuracy after the next round. 67% after 20 rounds.

There's no way for it to get better without more real human input.

aezart··on Johnny Decimal
I genuinely can't imagine this working at all for any sort of software development project.
aezart··on Modern language models refute Chomsky’s approach to language
Not for a lack of trying.

If you give it a small enough training set or a big enough neural network, it will directly memorize the whole thing. You have to intentionally make its brain too small to do that in order to force it to find patterns in the data instead.

aezart··on WHO advises against use of artificial sweeteners
The last time I tried Coke Zero, it tasted like Diet RC. I didn't like it at all.

Diet Coke has huge differences in flavor between sources and packaging options. It's universally bad out of a can or bottle, and the stuff from soda fountains varies by restaurant.

aezart··on “Fractureiser” malware in popular Minecraft mods and modpacks
That's quite possible, I'm not super into the modding scene. I guess my default assumption is that mods should be running through some sort of scripting interface that doesn't have access to things like the filesystem.
aezart··on “Fractureiser” malware in popular Minecraft mods and modpacks
I foresee this being a big problem for the Godot game engine as it gains popularity. There's absolutely no sandboxing or "safe mode" or anything when loading extra resource packs, and any resource or scene file can have an embedded script which executes upon loading.
aezart··on Moderation strike
Rapid degradation, like recording multiple generations of VHS tapes. The LLMs make the internet dumber, the LLMs get dumber by learning from it. Rinse and repeat.
aezart··on New warning messages might as well be fatal errors
Yeah, a third output would be nice to have. For example, grep could use stdout to print the actual matches it finds, stderr for any files it was unable to open, and a third stream to show the progress of the search.
aezart··on When the Neighbors Don’t Like Your 10-Foot-Tall Transformer Statues
I think they're ugly as hell and in poor taste but I also think people should have the right to do what they want with their homes as long as it's not dangerous.

Although since they're on a sidewalk it's a problem if they're blocking public spaces.

aezart··on US banks lost $472B in deposits in the Q1 2023
"US Bank deposits down $427B in Q1 2023" would make the most sense.

I also read "lost" as "lost track of", as in "we received these deposits but then don't know where they went".

aezart··on HP printers should have EPEAT ecolabels revoked, trade group demands
I've had bad luck with third-party toner cartridges on my Brother printer. Eventually it just stop detecting them at all.
aezart··on 12-Year-Old to Graduate from College with Five Degrees
Typically it takes 2 years instead of the 4 a bachelor's degree would take
aezart··on Legend of Zelda game sells 10M copies in three days
Yeah, from the trailers and footage I've seen on Twitter this doesn't seem like the game for me. I want a dungeon crawler, not a physics sandbox. Also I want Link to be left handed again.
aezart··on A weed is swallowing the Sonoran Desert
Oh no, another one? We already have a buffelgrass problem.
aezart··on Come back, c2.com, we still need you
A time waster's not always a bad thing; I spent lots of time browsing it back when I was a student intern because it seemed on topic" enough for me to not feel guilty.
aezart··on Delimiters won’t save you from prompt injection
It's all based on attention, right? If the model winds up giving a low attention value to the delimiters, they won't help at all.
aezart··on Pain points of teaching computer science
I didn't get credit for the first few assignments in my first CS class because the automated grading tool couldn't find my code. Finally I figured out that the tool only worked if the code was in the default Java package, which I wasn't using because Eclipse told me I shouldn't.
aezart··on Please don’t upload my code on GitHub
The funniest outcome of all this would be a ruling that the entire copilot model is GPL.
aezart··on An Update on the Lock Icon
I wound up just getting a signed cert for my home webserver. It wasn't too painful.
aezart··on Let's make sure GitHub doesn't become the only option
That's what github enterprise is for
aezart··on ChatGPT has no inner monologue or meta-cognition
One thing I'm not clear about is how the LLM decides on the correct tokenization to use.

Looking at GPT2's token vocabulary, it has the word "the" as token number 1169, but it also has "t", "h", and "e" separately as tokens 83, 71, and 68. When the tokenizer is reading the prompt, does it greedily create the longest tokens possible from the start of the input, or is there a more sophisticated process going on?

I guess in theory it could try multiple tokenizations and see which one results in the overall best response.

aezart··on MLC-LLM: GPT/Llama on consumer-class GPUs and phones
I can already run a language model on my GPU using minillm or text-generation-webui, and on my CPU using llama.cpp. What makes MLC-LLM better?
aezart··on How to spend money on your friends without it looking like bribery
Yeah, you have to exactly match electric output to demand or things get very bad. Until we have a few more breakthroughs in battery technology, we're going to be stuck with fossil fuels to make up the gap for variable demand. Nuclear can't be ramped up and down quickly, and solar and wind just do whatever they want.

As I understand it, a typical strategy for a power company is:

- nuclear or similar for the base load that's always required

- solar and wind as available

- weather forecasting to determine how much solar and wind will be available

- demand forecasting to determine how much power will be required at any given time

- natural gas to meet predicted solar and wind shortfalls

- RICE engines and battery storage to handle unexpected fluctuations

- contracts to buy and sell power with other nearby power companies

aezart··on Commercial lunar lander presumed lost after moon landing attempt
The point is that the American units are now defined relative to metric units.

As recently as the 1800s, the yard was defined as:

"the Straight Line or Distance between the Centres of the Two Points in the Gold Studs of the Straight Brass Rod now in the Custody of the Clerk of the House of Commons whereon the Words and Figures "Standard Yard 1760" are engraved"

... That standard yard was destroyed in a fire in 1834, and a new standard yard was made using several copies of the original standard yard as reference.

But now, it's just defined as an exact fraction of a meter. First as 36/39.370113 m, now as exactly 0.9144 meters.

aezart··on FastGPT: A fast, full search, AI answering engine
LLaMA claims to be GPT-4 when I ask it what it is, so I don't necessarily trust that response.
aezart··on Scaling Transformer to 1M tokens and beyond with RMT
From my limited understanding of how LLMs work, I believe this behavior is enabled by embeddings. The model maps its ~50,000 token vocabulary into a lower-dimensional vector space. Each dimension in the vector adds some sort of meaning (or at least association) to the word.

I saw an example in a Numberphile video where they were able to take the vector of the word "prince", subtract the vector of the word "man", add the vector of the word "woman", and the resulting vector was closest to the word "princess". So in theory there could be a "gender" dimension and a "position of authority" dimension in that vector (or the model might be making other, stranger connections between words that we don't understand).

I think the same thing is happening in your example. The model identifies and produces output that keeps it in the general region in the vector space for the "Shakespearean" and "Bowiean" dimensions while still satisfying other requirements.

aezart··on The New XOR Problem
True but not very useful. That's like writing an "isPrime" function with a switch statement for every possible input number, rather than coming up with a general algorithm. It can't give you any answer you don't already know.
← PreviousPage 4 of 5Next →