If labs provided the corpus and source code for training their tokenizers, it would be a lot easier to produce results about tokenizers. As it is, they provide neither, so it is impossible to compare different algorithms running on the same data if you also want to include the vocabs that are commonly used.
providing a large list of bitrotted URLs and titles of books which the user should OCR themselves before attempting to reproduce the model doesn't seem very useful.
I evaluated Windsurf at a friend's recommendation around half a year ago and found that it could not produce any useful behaviors on files above a thousand lines or so. I understand this is mostly a property of the model, but certainly also a property of the approach used by the editor of just tossing the entire file in, yeah? I haven't tried any of these products since then, but it might be worth another shot because Gemini might be able to handle these files.
The behavior of "let" with for loops where the variable is declared more times than it is initialized, despite the source code having one declaration that is also the only initialization, is not very explicit.
generic simd abstractions are of quite limited use. I'm not sure what's objectionable about the thing Rust has shipped (in nightly) for this, which is more or less the same as the stuff Zig has shipped for this (in a pre-1.0 compiler version).
there are no loads in the above unsafe block, in practice loadu is just as fast as load, and even if you manually use the aligned load or store, you get a crash. it's silly to say that crashes are unsafe.
This sort of measure is a decent match for BPB though. BPB=-log(document_probability)/document_length_bytes and perplexity=e^(BPB*document_length_bytes/document_length_tokens). We already train models by minimizing perplexity, and model outputs are already those that are high probability. Though like with EBMs, figuring out outputs with even higher probability would require an expensive search step.
I always ask in these threads, what sort of data do you have that cannot be divided into several integers, such that sorting by those integers according to some priority implements your comparator correctly, and also your comparator is not so expensive as to be impractical? on one occasion someone replied "sorting unicode strings in collation order" but it doesn't seem like recomputing the collation inside of each comparison is practical, and the cache behavior of comparison sorts if you precompute all of them (spending several times the memory occupied by the input) will be quite bad.
structs containing strings and doubles for example are well suited to radix sort.
When people go to Nextdoor to drum up opposition to a development on the basis that they have enjoyed driving by that disused refinery for a decade, maybe we shouldn't care whether they think "anything other than a disused refinery on that particular lot" is a nuisance.
Why do we want people to buy houses? It incurs transaction costs and makes it difficult for people to relocate to be near their jobs and such. It seems like we should optimize solely for the cost of meeting people's needs - either by paying rent or by owning - compared to the sale value of their labor.
I have tried to explain the sense of "good investment" that is meant by policymakers and voters and the cliche above. You have tried to explain that there is another sensible definition available. I don't disagree, but I think it is beside the point.
Even better, if you click through to the linked source he doesn't say "low-level" at all, or make any claim that is at all like the claim he is cited as making!
You can use https://github.com/simd-everywhere/simde if you like. In general portable SIMD libraries are of limited utility because having different primitives available on different architectures often means that you should approach problems differently. That is to say, in many cases using any portable SIMD API to solve your problem means leaving 200% speedups on the table on at least one of your top 3 targets.
The thing that is present in Zig and not yet stable in Rust does not include any dynamic shuffles, so these end up requiring intrinsics or asm for all sorts of things. It's a significant weakness compared to e.g. highway, eve, or simde.
I suspect that it is not worth using AVX2 vector gathers on any CPU. But certainly you could end up with the best implementation varying between microarchitectures for other reasons.
I'm primarily writing "general-purpose" code (especially parsers and formatters) rather than code that does the same math operation on a big array, so it's usually not reasonable to even use the same approach to the problem with different vector extensions :(
I have no idea how to get the compiler to generate wider-than-16 pshufb in the general case, for example, and for the 16-wide case, writing the actual definition of pshufb prevents you from getting pshufb while writing a version with UB gets you pshufb.
Zig ships LLVM's internal generic SIMD stuff, which is fairly common for newish systems languages. If you want dynamic shuffles or even moderately exotic things like maddubs or aesenc then you need to use LLVM intrinsics for specific instructions or asm.
the old commits are still in your reflog, and if they belong to a branch that isn't only on your machine, you can nag a coworker if you don't know how to use that
Often people arrive on a team that has committed a grave sin in the past and all future team members are forced to use workflows involving submodules as penance.