avinassh

I posted in another comment: There is a smoll demo in the post. In it, I corrupt an old frame that is not needed by the database at all. Now, one approach would be to continue the recovery and then present both states: one where the WAL is dropped, and another showing whatever we have recovered. If I had such an option, I would almost always pick the latter.

> The WAL was corrupted, the actual data is lost. There's no parity. You're suggesting that sqlite should somehow recreate the data from nothing.

Not all frames in the WAL are important. Sure, recovery may be impossible in some cases, but not all checksum failures are impossible to recover from.

> 2. You can (and probably will) submit a change that would solve it.

SQLite is not open to contribution - https://www.sqlite.org/copyright.html

> 1. You don't understand why it was done like this.

sure, I would like to understand it. That's why the post!

> Aight, I'll bite: continue or stop... and do what? As others have pointed out, the only safe option to get back to a consistent state is to roll back to a safe point.

Attempt to recover! Again, not all checksum errors are impossible to recover. I hold the view that even if there is a 1% chance of recovery, we should attempt it. This may be done by SQLite, an external tool, or even manually. Since WAL corruption issues are silent, we cannot do that now.

There is a smoll demo in the post. In it, I corrupt an old frame that is not needed by the database at all. Now, one approach would be to continue the recovery and then present both states: one where the WAL is dropped, and another showing whatever we have recovered. If I had such an option, I would almost always pick the latter.

> How would this work differently?

I would like it to raise an error and then provide an option to continue or stop. Since continuing is the default, we need a way to opt in to stopping on checksum failure.

Not all checksum errors are impossible to recover from. Also, as the post mentions, only some non important pages could be corrupt too.

My main complaint is that it doesn't give developers an option.

There are also CoW B Trees not entirely similar, but kinda same.

I don't get this scheme at all. The protocol violates durability, because once the client receives success from server, it should be durable. However, completion record is async, it is possible that it never completes and server crashes.

During recovery, since the server applies only the operations which have both records, you will not recover a record which was successful to the client.

    * * Wait for intent and completion to be flushed to disk * *

if you wait for both to complete, then how it can be faster than doing a single IO?

what does the name mean

How about Limbo, the SQLite rewrite in rust?

https://github.com/tursodatabase/limbo

this one - https://www.jetbrains.com/junie/

That is awesome! I am looking forward to the release.

I like this site, I have been trying. One suggestion to you OP, if there is a way you can make lessons and if one can practice through one by one, it would be great!

you could use SQLite, store all of them

I signed up the site mentioned, most features are free and I like it. The biggest plus I see here compared to monkey or typeracer is that this site shows the hands and fingers visually. That makes it easier to follow and learn

I would like to give a try, but I am getting this for India:

My name is Aline, and I'm the founder of interviewing.io. Thanks so much for your interest. It looks like you’re not in a country where we’re open for business yet, so we can’t create your account, but we’ll add you to our waitlist.

> Indexing files is faster: A 38-minute video now indexes in ~3 minutes instead of 10-14 minutes

can you explain how indexing is done and how rust helped in this case?

I want to ask developers here who keep and maintain notebooks: How exactly do you take notes? What things do you write down? And how does it help you, or how has it made you a better developer?

OP's post also references a similar post [0], but I am curious to hear from the people here.

[0] - https://hamatti.org/posts/how-i-take-work-notes-as-a-develop...

Do you use any iOS 18 specific APIs? I am on an old device using iOS 17, so I cannot upgrade to 18. I'd love if you can consider making this compatible with 17

This looks great! Is it compatible with lazyvim's installations? Because I could not get it working and I suspect it is conflicting with some other plugin

I am not sure if it helpful to you, but I did RC remotely and my experience was great

> I remember the first post by the Neon team here on HN. I think I commented at the time that I thought it was a great idea.

Same! I remember it too. I found it quite fascinating. Separation of storage and compute was something new to me, and I was asking them about Pageserver [0]. I also asked for career advice on how to get into database development [1].

Two years later, I ended up working on very similar disaggregated storage at Turso database.

Congrats to the Neon team!

[0] - https://news.ycombinator.com/item?id=31756671

[1] - https://news.ycombinator.com/item?id=31756510

this lwn article supports the argument that many cloud providers contributed back to Redis: https://lwn.net/Articles/966631

> It is also hard to reconcile the claims that cloud providers do not contribute with the actual commits to the Redis repository. A quick examination of the commits since the 7.0.0 release using gitdm shows 967 commits over that time period:

    Top changeset contributions by employer
    (Unknown)         331     34.2%
    Tencent           240     24.8%
    Redis             189     19.5%
    Alibaba            65     6.7%
    Huawei             50     5.2%
    Amazon.com         50     5.2%
    Bytedance          19     2.0%
    NetEase            13     1.3%

> Binbin Zhu, of Tencent, is responsible for nearly 25% of the commits to the project. Some of the contributors without a readily identifiable employer surely are Redis employees, but it's clear that the company has not been working alone.

Garnet fascinates me. Their benchmarks even claim that it is better than Redis and also Dragonfly. Are there any papers or write ups explaining what makes Garnet fast? (I do know its based on FASTER)

> This surely could be recreated with off-the-shelf components, no question.

I would love to know how does one go about doing this with off-the-shelf components. Can you share more, please

I am not the target audience for this, but I really appreciate you for keeping this in the FAQ and explaining the inner workings:

> Do I even need VRAM Pro to do this?

I wish you best.

> pgdog represents maybe just the VTGate portion of Vitess

That’s today. The project is developing fast, so I am sure more things will be added :)

I am long on this project and excited about it.

I am wondering, why it took so long for something like this to happen for Postgres, where as Vitess existed for MySQL more than a decade. I am not an expert, but I hear Citus is not really a fair comparison with Vitess.

For "memory", I wonder how it would be if you use vector search in SQLite and pass that info to reduce context size. The ValTown SQLite should have support for vectors API - https://docs.turso.tech/features/ai-and-embeddings#vectors

Go also had one more requirement:

Many hash table designs, including Abseil’s Swiss Tables, forbid modifying the map during iteration. The Go language spec explicitly allows modifications during iteration, with the following semantics:

- If an entry is deleted before it is reached, it will not be produced.

- If an entry is updated before it is reached, the updated value will be produced.

- If a new entry is added, it may or may not be produced.

which I don't think was needed for valkey