6,269 karma · joined December 9, 2010
For integer parsing, this is a pretty cheap operation, so I wonder if it is worthwhile to try to avoid doing it many times. For double parsing, it is expensive if you require the bottom couple bits to be correct, so the approach in the blog post should create savings.
To be honest, I just do not have much faith in the commitment to optimality, when it seems like the team has not spent time doing things like profiling a program that spends a lot of time formatting integers as decimal syrings, and noticing that the vast majority of that formatting runtime is UTF-8 validation. I am happy to continue using the language, because it makes it easy enough to fix these issues oneself.
The only aspect that may not be recoverable by the end user is the "am I async/is this async" reflection issue, though a core team member has clarified in this comment section that the code in the article is a sketch and the design of stackless coroutines is far from done, so we may yet get this.
Some other philosophical point is, like, lua's coroutine.create/resume/yield/clone are control flow primitives for use within a single thread of execution. It's fine to ship an async runtime, which embodies the view they they are not control flow primitives for use within a single thread of execution, for doing I/O. But focusing the primitives for creating and switching between execution contexts too narrowly on the async runtime use case is liable to he harmful to other use cases for these operations. Ideally, we would be able to write things like a prominent SNES emulator that uses stack switching to ensure the simulation of different components proceeds in an order known to be more correct than other orders, and we would be able to do it using native language features, which would compile down to something a bit cheaper than dumping all of our registers onto the stack. Ideally when we do this we would not be asked by the language to consider what it would mean to "cancel" the execution context managing one of the components, in the same way that we do not need to consider what it means to cancel an arbitrary struct, or the function which is calling the function currently executing.
Additionally, I don't necessarily want to delegate the management of the memory backing the futures to an Io, or pass around a blob of syscalls and an associated runtime, which accesses everything via a vtable. I would prefer to have these things be compile time generic only.
If we were already depending on highway or eve, I would think it's great to ship the generic SIMD version instead of the SSE version, which probably compiles down to the same thing on the relevant targets. This way, if future maintainers need to make changes and don't want to deal with the several implementations I have left behind, the presence of the generic implementation would allow them to delete them rather than making the same changes a bunch of times.
In Boundless BPE, no schedule must be chosen, because there is not any point at which the constraints of pretokenization are removed entirely. Instead, at any point in the learning process, merges between adjacent pretokens are permitted if the pretokens are each represented by a single token. There are some additional details about how the authors incorporate Picky BPE, which I will not try to repeat because I would probably get them wrong.
I have read this paper before, and now that I have looked more closely I am a bit disappointed that the precise method of construction of the large initial vocabularies used for vocabulary learners that begin with large initial vocabularies in the experiment is not specified. I also notice that the same error I have proposed correcting in the docs is present in the paper in section 3.1.
Additional edit: Alright, I see that A.2.1 specifies that the "n-gram" initial vocab condition means the most frequent 2^18 n-grams for n in 1..L. This seems like not very many, and not weighting n-grams by "potential CTC reduction" (=length-1) for inclusion in this initial vocab seems to create a bias against long n-grams. I'm also sort of wary of these greedy learners, because many candidates for inclusion in the vocab are interdependent with other candidates (the marginal loss or gain of a candidate may be near 0 because of the presence of some other token which may not actually be optimal to include, or there may be groups of included tokens which cause other better groups to have near 0 marginal loss or gain), so I would really want to avoid being too strongly path-dependent in the learner. If I may hand wave a great deal, I think this sort of issue also arises if your objective function is not CTC.
It's not "all of the text", it's like less than 100 trillion tokens, which means less than 400TB assuming you don't bother to run the token streams through a general purpose compression algorithm before storing them.
I agree that it would be more useful to compare vocabs of identical size.
Anyway, the paper says "Frequent substrings can be enumerated in O(T) time and O(20T) space with the Enhanced Suffix Array algorithm (Nong et al., 2009)", which is hilariously underspecified, at least in part because a suffix array algorithm isn't a top-k algorithm.