2,085 karma · joined November 23, 2016
```
[log] SQLite: 100000 embeddings inserted in 2635 ms
[log] SQLite: 100000 embeddings retrieved in 561 ms
```
Isar is still ~2x faster, but there's room for optimization.
Thanks a bunch for sharing this, I will update the post.
[1]: https://github.com/ente-io/edge-db-benchmarks/commit/51ec496...
I have nothing against SQLite, but I unfortunately haven't yet understood how to write and read Lists more efficiently. Once I do, I'll definitely post a follow up :)
Removing the offsets has unfortunately not sped things up, since it's the step post reading the entries (deserialization) that is the bottleneck[1].
Since SQLite does not support Lists, Protobufs came across as the best way to serialize the data at hand.
If there's more native way to solve this problem within SQLite, please let me know!
I was playing around with the offsets incorrectly, while attempting to reduce the memory footprint on a low-end Android device.
I've updated the code[1] to read all embeddings in one shot, and I'm now measuring the time to read and the time to deserialize separately.
On an iPhone simulator running on an Apple M3 Pro, this is what it says:
```
[log] SqliteDB fetch all took: 1572 ms
[log] SqliteDB deserialization took: 4268 ms
...
[log] Isar: 100000 embeddings retrieved in 616 ms
```
[1]: https://github.com/ente-io/edge-db-benchmarks/commit/e02c3d3...
The code we used for running the benchmarks is available here[1].
If you have suggestions on how we could speed up the deserialization during reads, please do share. We'd be happy to go back to SQLite.
But as an end consumer, the overall throughput while using SQLite as the DB engine was not sufficient to serve our use case of reading 100k embeddings.
[1]: https://github.com/ente-io/edge-db-benchmarks/blob/43273607d...
We are executing the entire write in a single batch query.
From what we observed, and have documented, it is the serialization that is causing the bottleneck, not the write itself.
Also, more than writes, it is the latency associated with deserializing the read data into models that pushed us to look at options outside SQLite.
[1]: https://github.com/ente-io/edge-db-benchmarks/blob/43273607d...
[1]: https://github.com/ente-io/edge-db-benchmarks/blob/43273607d...
Edit: Updated the permalink
Here you go: https://github.com/ente-io/edge-db-benchmarks/blob/43273607d...
Within reads the most expensive part was the re-construction of models from the rows returned by SQLite.
While deserialization is not the responsibility of the database, the overall throughput with SQLite was too low for it to serve our usecase.
Snippets are available @ https://github.com/ente-io/edge-db-benchmarks/blob/main/lib/...
Context: I'm working on an e2ee alternative to Google Photos[1] where we have to cluster embeddings (for face recognition) and run similarity searches (for semantic search[2]) on device.
[1]: https://ente.io
The closest I can see is the VSS extension[1] for Sqlite.
Ente was first a piece of hardware, then a self-host-able project, but we had a hard time monetizing both, which lead to the E2EE pivot.
TIL about TagSpaces, thanks!
Our server can be open-sourced, but we're unsure of the value E2EE will provide, with services like Photoprism[1] and Immich[2] already doing a good job of serving customers who prefer to self host. In this context E2EE might become a constraint, rather than a feature.
If your primary device is Android, please check out Ente[1].
We are an E2EE alternative to Google Photos. We had launched on HN[2] a while ago, and have been working towards feature parity. We aren't "there" yet, but hope to soon be.
If you've any feedback, please share it with vishnu@ente.io, I'd be grateful!
[1]: https://ente.io
Sync is optional, and it's for those who value convenience (both for multi-device access and backups).
Our architecture[1] is similar to that of password managers, and we've been externally audited[2].
Also, Auth has helped spread word about Photos, and we're very pleased with this side effect.
[1]: https://ente.io
We've been building Auth for almost a year now. It started off as a pet project to cure our own itch. At the point, we couldn't find a cross platform open source app that offered e2ee backups. Once we had an MVP, we released the project to our community[1], who responded favorably, and with their help the project has matured to where it is now.
If you've any feedback or questions do let us know!
All the best!
[1]: https://listmonk.app
Fun fact, the author (Frank Denis) is an incredibly talented photographer[1].
Thanks for pointing out the workaround with GPS, will look into it!
Currently on iOS, our servers send silent pushes to your device every hour. This wakes up the app and gives us 30 seconds to execute a background sync (provided you've previously not killed the app). This is not sufficient to backup videos, but it works pretty well for incremental backup of photos (I personally use an iPhone).