15k inserts/second is basically nothing. This is just a batch load of INSERT statements.
Now do it while simultaneously serving 15k selects/second.
Now try it with 15k counts/second. You'll see where this breaks down fast.
Now do it while simultaneously serving 15k selects/second.
Now try it with 15k counts/second. You'll see where this breaks down fast.
How many CPUs and cores are available? Are we talking SSDs?
How much data is being inserted?
What are the keys and index?
15,000 per second seems like plenty much in some contexts.
CREATE TABLE IF NOT EXISTS users (
id BLOB PRIMARY KEY NOT NULL,
created_at TEXT NOT NULL,
username TEXT NOT NULL
);
let user = User {
id: uuid::Uuid::new_v4(),
created_at: chrono::Utc::now(),
username: String::from("Hello"),
};
I really like SQLite. But this blog isn't any kind of great performance indicator for it.
insert into users (id, created_at, username) values ("<id>", "<created_at>", "Hello"), ...15,000 times.
Disk I/O has always been the bottleneck for writes.
That said, it is increasingly rare for disk I/O to be a bottleneck for inserts. When your storage will happily sustain 10+ GB/s of write throughput, you tend to run out of things like network and memory bandwidth first.
For example, if the whole world wanted to create an account, and an account consisted of a UUID and a date, it'd take 5.4 days using this metric as a minimum viable benchmark.
It's all about context, perception. 15,000 seems like a large number, and 1 second seems like a small amount of time. On the other hand, 5.4 days seems like quite a large number. Likewise at small scales, 15,000/second operated at a fidelity only able to capture 0.0005% of a typical processor's clock.
Like a picture in a frame.
This is not to say that the ability to casually write 15,000 things a second to a database is not impressive. Nor that a beowulf cluster of it wouldn't be.
I made a prototype message queue in Rust that processed about 7 million messages/second. That was running from RAM.
I also had a prototype that ran from disk and it very rapidly maxed out the random write performance of the SSD. Can't remember the number but it was in the order of 15K to 30K messages a second, bottlenecked by the SSD.
If the sqlite test is writing to disk it is likely also bottlenecked at the disk.
Can you share the source code?