HNHacker News
TopNewBestAskShowJobs

anonymoushn

6,269 karma · joined December 9, 2010

bluntsmoker@ltcm.lol
submissionscomments
anonymoushn··on Why my p(doom) has risen, dramatically
please post about my pet issue instead of whatever the comment section is nominally about.
anonymoushn··on Strategies for Fast Lexers
Well, when it's this fast already, there may not be much point in vectorizing it.

For integer parsing, this is a pretty cheap operation, so I wonder if it is worthwhile to try to avoid doing it many times. For double parsing, it is expensive if you require the bottom couple bits to be correct, so the approach in the blog post should create savings.

anonymoushn··on Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
Isn't this incorrect? If the user doesn't specify what to do with almost all of the special tokens, you still must detect them so you can raise an error.
anonymoushn··on Show HN: Learn LLMs LeetCode Style
Accounts with high karma have a "flag" link in the header of comments and submissions for indicating that it may not be the right kind of content for the site. Human moderators actively monitor the site, so your comment about this user's behavior will be seen by one of them soon.
anonymoushn··on Zig's New Async I/O
Yes, I understand that the designers prefer the Allocator situation and that Reader and Writer being anytype was downstream of the difficulty of using async readers and writers otherwise. So the intention was always to go with the design that I do not prefer. One reason I do not prefer it is that the Reader and Writer interfaces were already staggeringly inefficient, despite the lack or virtualization. We have avoided the issue by reimplementing a bunch of their API in some specific readers and writers and modifying the stdlib Reader and Writer to dispatch to these methods if they are present.

To be honest, I just do not have much faith in the commitment to optimality, when it seems like the team has not spent time doing things like profiling a program that spends a lot of time formatting integers as decimal syrings, and noticing that the vast majority of that formatting runtime is UTF-8 validation. I am happy to continue using the language, because it makes it easy enough to fix these issues oneself.

The only aspect that may not be recoverable by the end user is the "am I async/is this async" reflection issue, though a core team member has clarified in this comment section that the code in the article is a sketch and the design of stackless coroutines is far from done, so we may yet get this.

Some other philosophical point is, like, lua's coroutine.create/resume/yield/clone are control flow primitives for use within a single thread of execution. It's fine to ship an async runtime, which embodies the view they they are not control flow primitives for use within a single thread of execution, for doing I/O. But focusing the primitives for creating and switching between execution contexts too narrowly on the async runtime use case is liable to he harmful to other use cases for these operations. Ideally, we would be able to write things like a prominent SNES emulator that uses stack switching to ensure the simulation of different components proceeds in an order known to be more correct than other orders, and we would be able to do it using native language features, which would compile down to something a bit cheaper than dumping all of our registers onto the stack. Ideally when we do this we would not be asked by the language to consider what it would mean to "cancel" the execution context managing one of the components, in the same way that we do not need to consider what it means to cancel an arbitrary struct, or the function which is calling the function currently executing.

anonymoushn··on Zig's New Async I/O
I think this design is a regression from the previous design, in which you could use compile time introspection to check whether things are actually async (calling convention) or not.

Additionally, I don't necessarily want to delegate the management of the memory backing the futures to an Io, or pass around a blob of syscalls and an associated runtime, which accesses everything via a vtable. I would prefer to have these things be compile time generic only.

anonymoushn··on OpenAI delays launch of open-weight model
The closed weights models from OpenAI already do these things though
anonymoushn··on The messy reality of SIMD (vector) functions
For now, at work, it's just some parts with AVX-512, some parts with AVX-512 that we can't really use, so we should use AVX2, and some parts with NEON and SVE. So the implementations for SSE basically are a courtesy to outside users of the libraries, and there are no RVV implementations.

If we were already depending on highway or eve, I would think it's great to ship the generic SIMD version instead of the SSE version, which probably compiles down to the same thing on the relevant targets. This way, if future maintainers need to make changes and don't want to deal with the several implementations I have left behind, the presence of the generic implementation would allow them to delete them rather than making the same changes a bunch of times.

anonymoushn··on The messy reality of SIMD (vector) functions
It's really not. As an example, for string processing tasks (including codecs which various server software spends a significant percentage of its runtime on), NEON includes a deinterleaving load into 4 registers and byte-wise shuffles that accept 2, 3, or 4 registers worth of lookup table. These primitives are quite different from those available on AVX2 or AVX-512, and the fact that they are available and cheap to use means you end up with somewhat different algorithms for the two types of targets. Even the practice of using the toys available in AVX2 well for this sort of task is somewhat obscure. Folks who have worked on codec-type stuff but primarily used AVX-512 often have trouble figuring out how to do most of the same things in similar instruction counts if masked versions of the instructions are not available.
anonymoushn··on The bitter lesson is coming for tokenization
Sure. Another view is that an LLM is an immutable function from document-prefixes to next-token distributions.
anonymoushn··on The bitter lesson is coming for tokenization
I see. You're right. I was either badly mistaken or only thinking about small k.
anonymoushn··on The bitter lesson is coming for tokenization
It's about the immutability of the network at runtime. But I really don't think this is a big deal. General-purpose computers are immutable after they are manufactured, but can exhibit a variety of useful behaviors when supplied with different data. Human intelligence also doesn't rely on designing and manufacturing revised layouts for the nervous system (within a single human's lifetime, for use by that single human) to adapt to different settings. Is the level of mutability used by humans substantially more expressive than the limits of in-context learning? what about the limits of more unusual in-context learning techniques that are register-like, or that perform steps of gradient descent during inference? I don't know of a good argument that all of these techniques used in ML are fundamentally not expressive enough.
anonymoushn··on The bitter lesson is coming for tokenization
I think within the framework of "almost-orthogonal axes" you can still create a vector that has the desired mix of projections onto any combination of these axes?
anonymoushn··on The bitter lesson is coming for tokenization
There are various papers about this, maybe most prominently Byte-Latent Transformer.
anonymoushn··on The bitter lesson is coming for tokenization
I mean, this depends on your sampler. With temp=1 and sampling from the raw output distribution, setting aside numerics issues, these models output nonzero probability of every token at each position
anonymoushn··on The Bitter Lesson is coming for Tokenization
yes, in typical architectures for models dealing with text, the output is a token from the same vocabulary as the input.
anonymoushn··on Jemalloc Postmortem
Well, to be blunt, the company does not care about this, so it does not get done.
anonymoushn··on Jemalloc Postmortem
The big recent change is that jemalloc no longer has any of its previous long-term maintainers. But it is receiving more attention from Facebook than it has in a long time, and I am somewhat optimistic that after some recent drama where some of that attention was aimed in a counterproductive direction that the company can aim the rest of it in directions that Qi and Jason would agree with, and that are well aligned with the needs of external users.
anonymoushn··on Why Use Structured Errors in Rust Applications?
I think that like actually using Rust and encountering situations where one must define a new error enum that exists only to wrap the values returned by two different fallible functions belonging to different libraries all the time was sufficient to convince me that the situation in Zig is preferable. It's also generally good that users are unable to attach strings to their errors when using the built-in error handling, in the sense that things that are usually bad ideas ought to be a bit more difficult, so that users do not end up doing them all the time.
anonymoushn··on Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
In SuperBPE, a fixed number of tokens are learned, and then the constraints of pretokenization are removed entirely, and then the remainder of the target vocab size is learned.

In Boundless BPE, no schedule must be chosen, because there is not any point at which the constraints of pretokenization are removed entirely. Instead, at any point in the learning process, merges between adjacent pretokens are permitted if the pretokens are each represented by a single token. There are some additional details about how the authors incorporate Picky BPE, which I will not try to repeat because I would probably get them wrong.

anonymoushn··on Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
I think that the ideal number of initial n-grams would be large enough that adding additional initial n-grams has no effect on the output, because I expect to not be very good at tuning two different knobs.
anonymoushn··on Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
No, the segmentation algorithm you have implemented has runtime O(N*L^2). In particular, if you want to do a hash table lookup using a string key, that takes time proportional to the length of the string, not constant time.
anonymoushn··on Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
I actually wonder if he could just claim a win by calculating validation set BPB for both equally-sized vocabs instead of targeting the same level of perplexity as in the speedrun finish line lol
anonymoushn··on Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
Cool, I can have a look. By the way, I've filed a PR against the pathpiece repository correcting an error in the docs.

I have read this paper before, and now that I have looked more closely I am a bit disappointed that the precise method of construction of the large initial vocabularies used for vocabulary learners that begin with large initial vocabularies in the experiment is not specified. I also notice that the same error I have proposed correcting in the docs is present in the paper in section 3.1.

Additional edit: Alright, I see that A.2.1 specifies that the "n-gram" initial vocab condition means the most frequent 2^18 n-grams for n in 1..L. This seems like not very many, and not weighting n-grams by "potential CTC reduction" (=length-1) for inclusion in this initial vocab seems to create a bias against long n-grams. I'm also sort of wary of these greedy learners, because many candidates for inclusion in the vocab are interdependent with other candidates (the marginal loss or gain of a candidate may be near 0 because of the presence of some other token which may not actually be optimal to include, or there may be groups of included tokens which cause other better groups to have near 0 marginal loss or gain), so I would really want to avoid being too strongly path-dependent in the learner. If I may hand wave a great deal, I think this sort of issue also arises if your objective function is not CTC.

anonymoushn··on Deepseek R1-0528
If you try to reproduce various open datasets like fineweb by scraping the pages again, you can't, because a lot of the pages no longer exist. That's why you would prefer to store them instead of losing the content forever.

It's not "all of the text", it's like less than 100 trillion tokens, which means less than 400TB assuming you don't bother to run the token streams through a general purpose compression algorithm before storing them.

anonymoushn··on Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
Well, he says he achieved some downstream win on the same size, but it didn't translate into a win in perplexity, so he tried to do something else. Like I said, it's unfortunate.
anonymoushn··on Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
https://x.com/alexjc/status/1881410039639863622

I agree that it would be more useful to compare vocabs of identical size.

anonymoushn··on Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
You can use modded-nanogpt for testing this sort of change. If you don't want to do anything really weird, you can just train whatever tokenizer you like on the training set, then retokenize the input, then run the existing training code. One person did this earlier this year using a Tokenmonster vocab and got better downstream performance in less train time.
anonymoushn··on Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
They have not.
anonymoushn··on Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
If the library has two vocabulary learners, only one of which does the described thing, then isn't it unambiguous which implementation within the library the question refers to? And wouldn't it be ambiguous to instead say "how does Unigram do it" without referring to any particular implementation?

Anyway, the paper says "Frequent substrings can be enumerated in O(T) time and O(20T) space with the Enhanced Suffix Array algorithm (Nong et al., 2009)", which is hilariously underspecified, at least in part because a suffix array algorithm isn't a top-k algorithm.

← PreviousPage 3 of 34Next →