celeritascelery

This makes me wonder if I should be writing on substack. I currently use a static site generator and host with netlify[1]. This is great because it is simple and I have complete control over the site. But at the same time I can’t see when people visit my site and there is no comment section. I rely or external forums like HN, lobsters, and Reddit to have a discussion. And I have no way of knowing when my content gets shared because I can’t see the traffic.

But my purpose of writing is the same as the authors, to find like minded people and have a discussion with them. It’s kind of cure for intellectual loneliness.

[1] https://coredumped.dev/

Some of my favorite technical podcasts

Oxide and friends: https://oxide.computer/podcasts/oxide-and-friends

On the metal: https://oxide.computer/podcasts/on-the-metal

Handmade network: https://handmade.network/podcast

Signals & threads: https://signals-threads.simplecast.com/

This seems like it has some really cool ideas! This reminds me a little of eshell, which has some of the same features around allowing you to define shell functions in lisp. However it does run into issues moving between the lisp (sexp) and shell (string) boundaries.

There is an even better method to board a plane

https://m.youtube.com/watch?v=oAHbLRjF0vo

> Wouldn’t hybrids be more efficient if the gas/diesel was just generating electricity for an electric drivetrain? They do this for logging and mining trucks.

They might be more “efficient” from a maintenance point of view, but they will have worse gas mileage then gas/electric. Anytime you have to transform the energy to another form you loose some of power. So ICE > drivetrain will be more efficient then ICE > generator > drivetrain.

Using UTF-8 only has better engineering trade offs if you represent text as UTF-8. If you don’t (like JavaScript) then it’s just extra complexity. It’s no better than UTF-16. Only advantage it has is it’s more common among newer languages.

Code points is the only thing that makes sense for a multi-language protocol. It’s unambiguous and every Unicode client can talk in code points, even if they use some exotic encoding.

It is actually not very novel. It is just a Fenwick tree[1]. And a Fenwick tree is basically a rope where the leafs don't store any text.

basically every leaf holds some metrics (say bytes and line endings) for some chunk of text. By just looking at the leaf we don't know which chunk it points to. But since the leafs are all in order, we can calculate their byte position and line ending by adding them up. If we add all the leaf we should get the total number of bytes and lines in the text. In order to avoid O(n) cost of summing every leaf, we store them in a tree, where each parent holds the sums of it's children. So when we want to find the byte position[2] of a particular line ending we can just walk down the tree and see if the line ending we are searching for is greater then the left child or not. If it is, we add the left child's sum to our running total and go to the right. Otherwise we go to the left. By the time we get to the leaf, we have summed all the chunks before it in O(logn) time.

Insertion/deletion are similar, except when we update the leaf, we also go and update each parent on the way back up.

[1] https://www.baeldung.com/cs/fenwick-tree

[2] https://github.com/CeleritasCelery/rune/blob/be243ea2bd385ec...

It’s not quite that simple, but folks are working on it.

https://github.com/rust-lang/regex/issues/425#issuecomment-1...

https://github.com/helix-editor/helix/pull/211#issuecomment-...

> you just have to ask for 16 GiB of RAM, as per the proper definitions.

If you walked in an asked for 16 "gibibytes" of RAM they would just look at you funny. Or think you were non-native speaker.

I work in the hardware software boundary, and I have never once heard someone use the term kibibyte, mebibyte, or gibibyte in real life. Occasionally I will see KiB, MiB, and GiB in an academic paper, but that's about it. When I am talking with people and they say kilobyte, they always mean 1024. Trying to pretend otherwise would just cause unneeded confusion. That's what it meant went the term was coined, that what it meant when IEC tried to change the definition 25 years ago, and (unless you are selling hard drives) that's what it means to most people today.

I agree that it is unfortunate that they reused the terms from SI prefixes, since they are not SI prefixes. And if I had a time machine I would tell them to use different prefixes. But they didn't and those are the terms the world has adopted. It's also unfortunate that hard drive manufactures played fast and loose with the definition to pretend like their drives were bigger then they really were. Words are democracy, and they mean what people say they mean, not what the IEC says they should mean.

> As for JEDEC, standards bodies are made of people and people are fallible.

Exactly, so why is the IEC redefinition somehow more valid?

This does seem like a really neat trick I have not heard of before. I am still confused about moving the gap in the line-tracking buffer. It seems to me the only way you could make look-up O(1) is if you updated all the line indexes you passed over when you moved the gap. Because they are pointing to absolute positions (right?). So moving the line-tracking gap would technically be O(n), but you couldn't use memmove, and instead would need to iterate over one and add or subtract the gap size. Am I misunderstanding?

> There’s nothing stopping you from using it in an interactive environment. What else does a text editor do that would make JumpropeBuf not appropriate in an interactive editor?

I wouldn't say it's "not appropriate" just that it is not applicable. JumpropeBuf is really design (as I see it) for handling large numbers edits to the same location. Which is exactly what we have in the tracing benchmarks. But in an "realworld" "interactive" environment usually after every edit there are follow up operation like rendering to the screen, sending change set to language servers or linters, running syntax highlighting etc. So you want those edits to applied immediately, not buffered.

I understand this is probably different in the CRDT case where you could have swarms of edits coming from other clients. In that case I can see buffering being useful. And as you pointed out, all of these data structures are more then fast enough at the scale of the tracing benchmarks.

That being said, it's interesting that piece tables are kind of a similar idea, expect they never actually apply the edits. They just keep them buffered and parse the buffer when they need to get the current state of the text. I wonder if JumpRopeBuf could become something similar. Basically a piece table the occasionally merges the edits to avoid fragmentation.

hmm. What happens when you move the gap from the front to the end of the text (or vice versa)? wouldn't that require you to update all the line boundaries? And wouldn't finding the position of the Nth line be O(n).

> Also I expect if that was a real problem you could probably implement a system where you have more than one gap. Does that exist?

That is a really cool idea! I actually tried implementing that. However I gave up because the code became quite a bit more complex, but it would totally be possible! That being said, multiple gaps would only help the "move gap" latency, they wouldn't help with the resizing latency. The later is both less predictable and much higher then moving the gap. Also when you need to coalesce the text for searching you would loose all your gaps.

Thanks for all the work in bootstrapping this part of the ecosystem! I opened an issue[1] on the memory issue for jumprope. It seems to really come down to the large size of skiplist nodes relative to the text.

I did some testing with JumpRopeBuf, but ultimately did not include it because I was comparing things from an "interactive editor" perspective where edits are applied immediately instead of a collaborative/CRDT use case where edits are async. But it did perform very well as you said! I feel like JumpRopeBuf feels similar to a piece table, where edits are stored separately and then joined for reading.

[1] https://github.com/josephg/jumprope-rs/issues/5

depends on the context. The rule of thumb I have seen is that anything under 100ms is perceived as instantaneous by a user. So for interactive editing those latencies are acceptable. Though is should be noted those latencies are only for 1 GB file, so they will get worse as the edited file gets larger. But if you are building something like a CRDT where you can have edits coming in from many sources then those will start to compound, and will destroy your responsiveness. Also if you are performing edits while doing something like scrolling you will start to drop frames. Jumprope and Xi-rope were specifically designed for the CRDT use case. As Raph points out, the latencies are what will really bite you. And latency is actually why I picked a gap buffer; the ropes have too high of latency with regex searches. If that was ever fixed I would be tempted to switch to ropes.

I should have added a section on that.

JumpRope combines a gap buffer with a skip list. Crop combines a gap buffer with a rope. ropey is a "traditional" rope that doesn't use a gap buffer under the hood.

All the containers store the metrics (line endings, unicode codepoints, UTF-16 codepoints, etc) in a tree. The tree stores the value of all it's children summed up. So if if you do an insert at the start you only need to update a single branch. When calculating the line count you have to walk down the tree summing all the children as you go. But this means updating and lookup are both logn instead of having to update N nodes.

These benchmarks include that time.

I used to use YouTube Red and had a playlist of BGM that i would work to on YouTube. Then one day about 20% of the video’s disappeared due to DMCA takedowns. That was the day canceled my subscription and switched to getting everything I care about with yt-dlp.

> I still don't even understand why stackless coroutines are necessarily self-referential, but I am quite curious!

Because when stackless coroutines run they don’t have access to the stack that existed when they were created. everything that used to be on the stack needs to get packaged up in a struct (this is what `async fn` does). However now everything that used to point to something else on the stack (which rust understands and is fine with) now points to something else within the “impl Future” struct. Hence you have self referential structs.

> Oldest spoken language in the world? All equally old.

That doesn’t seem possible. Surely some spoken languages are older then others.

Why couldn’t the Rust compiler also assume that dissimilar types don’t alias?

"basically free" is very different then actually "free". Adding a ton of extra branches comes at a cost. Even though it is cheap it still takes an instructions that could be used for something else. You fill your branch prediction target cache must faster, ejecting branches you actually care about. It also makes it harder for the compiler to move code around and harder for the prefetcher to break data dependencies. This all adds up to non-trival overhead. You can tell your C compiler to always check arithmetic, but most don't because of the cost.

> You can do any rust optimization yourself in C++ (ie. aliasing assumptions)

I don’t think that is entirely true. C++ doesn’t have any aliasing requirements around pointers, so if the compiler sees two pointers it has to assume they might alias (unless the block is so simple it can determine aliasing itself, which is usually not the case), but in Rust mutable references are guaranteed to not alias.

This was part of the reason it took so long to land the “noalias” LLVM attribute in Rust. That optimization was rarely used in C/C++ land so it had not been battle tested. Rust found a host of LLVM bugs because it enables the optimization everywhere.

This looks very cool! Thank you for sharing. I find GC's in rust particularly fascinating.

So if I understand correctly, every time a Gc drops you add it a hashmap and then periodically, you run through all Gc's in that map and trace all their children to see if they are part of a cycle? I still don't understand how this would work without knowing the rootset. Just because something is part of a cycle doesn't tell you if it is inaccessible. There must be something here that I am missing.

So then why would these ones cause tremendous damage, I’m given how small they are?

I will share one more story. Some of the leaders were saying they couldn't hit the validation milestone because of all the things that had to completed before they could sign-off. Jim Keller's response was:

"This is how were going to validate this chip:

1. go to the beach

2. open your laptop

3. run the chip in emulation through boot

4. tape-in! (send to the fab)

5. take the rest of the year off at the beach "

His point was that we need a simple focus on proving the chip works and then shipping it. If any validation milestone or objective is not directly connected to that, it's not worth doing. We don't need all this extraneous validation and multiple layers of rechecking. Just emulate the thing and then your done.

In his interview with Jordan Petersen he talked about how some people spend all their time "admiring the problem"; Trying to understand and categorize it. Keller didn't want to spend any more time talking about the problem then needed. He was focused on finding good solutions (and it had to be a good solution), and then executing them.

I was at Intel when Jim Keller came. He was one of the most inspiring technical leaders I have ever encountered. He obviously had very strong engineering knowledge as well. When he came in, he took some time to really understanding our process and challenges. Then he created 5 areas that Intel needed to improve. He said “we are going to fix these, or burn down the place trying”. And these weren’t vague goals like “execute more effectively” or “be customer obsessed”, but really concrete and measurable things like “reduce empty silicon area by 50%”. He had a task force of principle engineers on each task, and every meeting he would spend almost the entire time going into detail of the progress and challenges on them. Anytime someone would complain and say the goal couldn’t be done, his response was “AMD did it, why can’t you?”. He was always trying to simplify things and cut through complexity. His vision was that if we hit those 5 targets, we would win the market. As an engineer, his clear vision and strong technical understanding made him the best leader I have ever been under.

it’s a pity he left Intel so quickly. After he was gone, his 5 goals basically fell by the wayside. I don’t think people like being told they’re doing it wrong. I still try to listen to any podcast or interview with him, because I think he has many deep insights.

Do you sit on cushion or mat? Or just on the floor directly. And are you sitting cross legged or some other method?

> I wanted to express the fact that everything that gets allocated (call stack, frames, classes, and objects) is alive and valid until the "root" VM is, thus I used 'a more or less everywhere.

That's not being expressed in the type system. The lifetime 'a is unbounded (meaning you can make it anything you want, including 'static) so anything that shares 'a can outlive the vm without rust complaining. it would be no different then if you removed 'a completely. If you wanted to ensure anything couldn't outlive the vm you could tie the lifetime to a reference to the vm, but then the vm can't hold those values (it would be a self-referential lifetime).

EDIT: Looks like this is not an issue because the invariant lifetime 'a is not used for the mutable reference of vm or call_stack. So it's not the invariance that is the problem, but rather how Rust reasons about the lifetime of mutable references, which this avoids.

In that case I don't understand what the point of 'a is on VM and CallStack. You can create[1][2] those with any unbounded lifetime (including 'static[3]), which means it is not constraining anything. What is the lifetime 'a doing here? Why not remove it?

[1] https://github.com/andreabergia/rjvm/blob/be9c54066c64a82879...

[2] https://github.com/andreabergia/rjvm/blob/be9c54066c64a82879...

[3] https://github.com/andreabergia/rjvm/blob/be9c54066c64a82879...