ayende

Unless I missed something, China just got the biggest wheat export in the world in its pocket (Russia). And the fifth is Ukraine.

That means that food security issue is far less of a problem.

Transaction merging allows you to handle that nicely. By handling concurrent writes and merging them into a single write to the disk.

4% - 6% overall query time improvement? That is absolutely an actual problem that is very attractive to solve.

You'll see database people jump through far bigger hoops to get much smaller improvements.

The linker can usually merge identical functions, and the compiler does this too. You'll only get different copies if they are relevant, and usually the branch elimination is more than enough to compensate.

Another aspect about this is that the cache will hold the _hot_ stuff. In most systems, even if you have three copies of a function, it will likely only have one of those that is hot.

Generally, really bad idea. Assume a 512gb RAM and most of that being in use. With a log file for durability.

It sounds fine, but...

What is the cost of restarts? You need to read through the log, apply it, etc.

That can take a LOT of time, especially on cloud disks

That reminds me very strongly how Zig does it

A better term might be state of the art, but that has its own issues. In this case, I mean, recent

Can you explain what you mean by generics modules?

Spaces are heavily used where I am right now as a replacement for radio with plenty of really interesting topic. Basically, "everyone can open a station" scenario.

Off the top of my head, there is a big trial going on, and people are discussing details and commentary on that (with competing spaces for the defense and persecution sides). Stuff that you don't have the time to convey using traditional media.

It has gotten big enough to have politicians show up and do special shows. Thing about 2+ hours of interacting directly with the public.

Something like this:

    struct cmp
    {
       hash: *fn(_:*void) u64,
       eq  : *fn(_:*void, _:*void) bool,
       free: *fn(_:*void) void 
    }

With maybe some default options should be enough for at least the basics. Note that the indirection overhead would be significant, but without generics or something similar, you don't really have a choice.

Linked list has a O(N) search time, which is atrocious.

A really common optimization is to replace a linear scan on a list with a hash table. In most languages, that is a trivial step to do, which means that it gets done.

There isn't any overhead.

With hare, you just blew your complexity budget on this thing.

I think that there is a problem here with regards to what you are trying to do.

You are conflating the _implementation_ of a hash table with the _existence_ of a hash table.

You list a few problems:

Computing the hash for arbitrary structure & check equality of the hash is something that you can do by adding a func in the creation of the map.

That will likely add overhead (since you can't inline it, and most of them are trivial), but at least you'll have something.

I'm not sure how memory allocations for a hash table is any different than the usual. Memory ownership semantics are likely relevant, so you'll likely need to add a way provide a free() routine here, as well.

A common example may be a `map<string, file>` - where you need to deallocate the string and `close()` the file.

This gets interesting when you do a set over an already existing value, by the way. And certainly a challenge you'll need to deal with.

My issue isn't so much with the complexity of the solution. Design choices such as not having generics will have impact on performance. But having _a_ solution is still a baseline requirement in my eyes. The original post said that you can just roll your own, except that you really can't. Certainly not over & over again.

With regards to the hash table I criticized, that is exactly the point. You were able to "get away" with implementing a pretty bare bone thing, but you called it out as something that should be viable in general.

Hashtables are in _common_ use. They aren't something that you'll use once in a blue moon. Go's decision to provide dedicated syntax for map wasn't an accident.

And for certain scenarios, you may need to use a different implementation to get the best results. But for the vast majority of scenarios, you just want _something_. And whatever you have in the box should suffice.

That is because a generic data structure has a lot of work done on it to optimize it for a wide range of scenarios. Conversely, a one off implementation is likely to be far worse.

There are also security considerations to take into account. You are not likely to properly implement things like protections against has poisonings attacks, etc.

And "easy to fix" is sure, but you have to remember to _do_ that, each and every time you write this code. Because your approach is to give that ownership to the user.

And your language design means that you _cannot_ actually solve that properly. That is not a good place to be.

C was designed in the 60s, and it is very much a legacy of that timeframe. Today, there is literally no system beyond hello world you can build that won't require hash tables galore.

The web is full of those (response and request headers, query string parameters), data formats (JSON is basically a hashtable plus some fancy pieces) , caches (that are just hash tables), etc.

The issue is _generic_ data structures, not external code. If I have to re-create a hash table for each scenario, I'll not invest the proper time to do so.

So if I need a hash table to map inodes to strings and another to map strings to IP addresses, I'll need to create that anew each time.

That means either writing a lot of code twice, or relying on manual code generation (which sucks for many reasons).

I don't think I ever heard such an appropriate turn of phrase as "woke clippy", bravo!

Marking a page pointer as modified / accessed for example

Just to note, they did ask, early 2000s iirc

NATO primary reason to exist is to counteract Russia

A really annoying aspect of keyrings in Linux is that if you are writing a generic Linux service (runs in the background, should support all distributions), there is no real option without 3rd party components.

On Windows, there is the Data Protection API, which allows you to encrypt a secret tied to a user's login or a machine's key. That is managed by the OS and allows you to lean on the built-in facilities.

Less secure is exactly that. If `Mr Shady` is in the mandated list (by law), and the browsers are required to threat their certificates as trusted, what do you think is the impact on the whole ecosystem?

When building databases, we care about durability, so database authors are usually well aware that you _have_ to use `F_FULLSYNC` for safety. The fact that `F_FULLSYNC` isn't safe means that you cannot write a transactional database on Mac, it is also a surprise to me.

Note that the man page for `F_FULLSYNC` itself doesn't mention that it is not reliable: https://developer.apple.com/library/archive/documentation/Sy...

Having a separate syscall is annoying, but workable. Having a scenario where we call flush and cannot ensure that this is the case is BAD. Note that handling flush failures is expected, but all databases require that flushing successfully will make the data durable.

Without that, there are no way to ensure durable writes and you might get data loss or data corruption.

Except that the response would be that any product with that IP will be banned from EU/USA. That is a huge hit to take.

As an aside, the code is also wrong and will fail in some rare cases because it assumes that the `read` syscall will only return partial information if you are at the end of the file. This is false, however, and will lead to data loss if happen.

Full post with details here: https://ayende.com/blog/196289-C/dont-assume-the-result-of-r...

Defer, comptime, far better cross file model, yo name just a few things

That isn't the problem. This is the issue:

   let x = str.parse()?;

   someFunc(x);

The function called here is dependent on what `someFunc` will accept. That means that I can change what values will be accepted here by changing a completely different piece of code.

For the record, I find Rust's ability to change the type of the variable _after_ it was declared to be absolutely bonkers for my ability to understand what is going on in the code.

I understand what it is doing, but the fact that changing the return type of a method will change what previous code will compile to is something that I find extremely non obvious and surprising.

Inferring what the type is from the current code, sure. That saves me typing, but from what happens elsewhere looks like time travel.

That actually doesn't matter This is orthogonal to mmap

The key from my perspective is that I CAN design my access patterns to match what you'll optimized.

Another aspect to remember is that mmap being even possible for databases as the primary mechanism is quite new.

Go 15 years ago and you are in 32 bit land. That rule out mmap as your approach.

At this point, I might as well skip the OS and go direct IO.

As for differ OS behavior, I generally find that they all roughly optimize for the same thing.

I need best perf on Linux and Windows. Other systems I can get away with just being pretty good

Yes, isn't that wonderful?

You get to take advantage of literally decades of experience

What is more, if you can match the profile of the optimization, you can benefit even more

Almost no one is going to have a lot of map calls

Uou map the file once, then fault it in

That is an interesting statement, when discussing what you want to do

In this case, there is the issue of who is the you on question

For the database in isolation, maybe not ideal

For a system whre db and app run on the same machine? The OS can make sure you are on friendly terms and not fighting

Same for trying to SSH to a bust server and the OS can balance things out

I'm the author (well, one of) RavenDB

You are correct to an extent, but there are a few things yo noted.

* you can design your system so the access pattern that the OS is optimized for matches your needs

* you can use madvise() to give some useful hints

* the amount of complexity you don't have to deal with is staggering