HNHacker News
TopNewBestAskShowJobs

vishnumohandas

2,085 karma · joined November 23, 2016

v [at] vishnu.tech
submissionscomments
vishnumohandas··on SQLite vs. ObjectBox vs. Isar
Yes, this[1] did the job, thank you!

[1]: https://news.ycombinator.com/item?id=39291411

vishnumohandas··on SQLite vs. ObjectBox vs. Isar
This[1] has improved the situation considerably!

```

[log] SQLite: 100000 embeddings inserted in 2635 ms

[log] SQLite: 100000 embeddings retrieved in 561 ms

```

Isar is still ~2x faster, but there's room for optimization.

Thanks a bunch for sharing this, I will update the post.

[1]: https://github.com/ente-io/edge-db-benchmarks/commit/51ec496...

vishnumohandas··on SQLite vs. ObjectBox vs. Isar
Sorry if I sounded defensive!

I have nothing against SQLite, but I unfortunately haven't yet understood how to write and read Lists more efficiently. Once I do, I'll definitely post a follow up :)

vishnumohandas··on SQLite vs. ObjectBox vs. Isar
Hey yes, thank you for pointing it out!

Removing the offsets has unfortunately not sped things up, since it's the step post reading the entries (deserialization) that is the bottleneck[1].

Since SQLite does not support Lists, Protobufs came across as the best way to serialize the data at hand.

If there's more native way to solve this problem within SQLite, please let me know!

[1]: https://news.ycombinator.com/item?id=39290784

vishnumohandas··on SQLite vs. ObjectBox vs. Isar
Hey, since SQLite does not yet support lists, if there's a better way to serialize so that reads can be sped up, do let me know, it would help!
vishnumohandas··on SQLite vs. ObjectBox vs. Isar
Nice catch, thank you!

I was playing around with the offsets incorrectly, while attempting to reduce the memory footprint on a low-end Android device.

I've updated the code[1] to read all embeddings in one shot, and I'm now measuring the time to read and the time to deserialize separately.

On an iPhone simulator running on an Apple M3 Pro, this is what it says:

```

[log] SqliteDB fetch all took: 1572 ms

[log] SqliteDB deserialization took: 4268 ms

...

[log] Isar: 100000 embeddings retrieved in 616 ms

```

[1]: https://github.com/ente-io/edge-db-benchmarks/commit/e02c3d3...

vishnumohandas··on SQLite vs. ObjectBox vs. Isar
Hello, OP here!

The code we used for running the benchmarks is available here[1].

If you have suggestions on how we could speed up the deserialization during reads, please do share. We'd be happy to go back to SQLite.

[1]: https://github.com/ente-io/edge-db-benchmarks

vishnumohandas··on SQLite vs. ObjectBox vs. Isar
Hey, we're already using sqlite_async, and yes it does a nice job of abstracting away the complexity introduced by isolates.
vishnumohandas··on SQLite vs. ObjectBox vs. Isar
I agree that deserialization is not the responsibility of the database.

But as an end consumer, the overall throughput while using SQLite as the DB engine was not sufficient to serve our use case of reading 100k embeddings.

vishnumohandas··on SQLite vs. ObjectBox vs. Isar
The code was always calling the latter[1], I had pasted the incorrect link.

[1]: https://github.com/ente-io/edge-db-benchmarks/blob/43273607d...

vishnumohandas··on SQLite vs. ObjectBox vs. Isar
My bad, I linked to the wrong portion of the code. Here's the updated link[1].

We are executing the entire write in a single batch query.

From what we observed, and have documented, it is the serialization that is causing the bottleneck, not the write itself.

Also, more than writes, it is the latency associated with deserializing the read data into models that pushed us to look at options outside SQLite.

[1]: https://github.com/ente-io/edge-db-benchmarks/blob/43273607d...

vishnumohandas··on SQLite vs. ObjectBox vs. Isar
Hey, the writes were batched[1]. What is expensive here is the serialization.

[1]: https://github.com/ente-io/edge-db-benchmarks/blob/43273607d...

Edit: Updated the permalink

vishnumohandas··on SQLite vs. ObjectBox vs. Isar
Sorry, we should have linked to the DB Schema.

Here you go: https://github.com/ente-io/edge-db-benchmarks/blob/43273607d...

vishnumohandas··on SQLite vs. ObjectBox vs. Isar
More than writes, reads is what we struggled with.

Within reads the most expensive part was the re-construction of models from the rows returned by SQLite.

While deserialization is not the responsibility of the database, the overall throughput with SQLite was too low for it to serve our usecase.

vishnumohandas··on SQLite vs. ObjectBox vs. Isar
Hello! OP here.

Snippets are available @ https://github.com/ente-io/edge-db-benchmarks/blob/main/lib/...

vishnumohandas··on Are we at peak vector database?
From a cursory glance, usearch[1] seems more portable.

[1]: https://github.com/unum-cloud/usearch

vishnumohandas··on Are we at peak vector database?
Running machine learning on device.

Context: I'm working on an e2ee alternative to Google Photos[1] where we have to cluster embeddings (for face recognition) and run similarity searches (for semantic search[2]) on device.

[1]: https://ente.io

[2]: https://openai.com/research/clip

vishnumohandas··on Are we at peak vector database?
By "edge", I was talking about mobile / IOT devices.

The closest I can see is the VSS extension[1] for Sqlite.

[1]: https://github.com/asg017/sqlite-vss

vishnumohandas··on Are we at peak vector database?
There are none that run on the edge, yet. Few more miles to go before we "peak".
vishnumohandas··on Escaping surveillance capitalism, at scale
Thank you!

Ente was first a piece of hardware, then a self-host-able project, but we had a hard time monetizing both, which lead to the E2EE pivot.

TIL about TagSpaces, thanks!

Our server can be open-sourced, but we're unsure of the value E2EE will provide, with services like Photoprism[1] and Immich[2] already doing a good job of serving customers who prefer to self host. In this context E2EE might become a constraint, rather than a feature.

[1]: https://github.com/photoprism/photoprism

[2]: https://github.com/immich-app/immich

vishnumohandas··on Escaping surveillance capitalism, at scale
> there is no better alternative

If your primary device is Android, please check out Ente[1].

We are an E2EE alternative to Google Photos. We had launched on HN[2] a while ago, and have been working towards feature parity. We aren't "there" yet, but hope to soon be.

If you've any feedback, please share it with vishnu@ente.io, I'd be grateful!

[1]: https://ente.io

[2]: https://news.ycombinator.com/item?id=28347439

vishnumohandas··on Show HN: Ente Auth – open-source 2FA app with end-to-end encrypted backups
Yes, it changes the security model from "something you have" to "something you know".

Sync is optional, and it's for those who value convenience (both for multi-device access and backups).

vishnumohandas··on Ente-io-auth: open-source 2FA app for Android, iOS and web
Hey, thanks for the feedback, this is already on our roadmap: https://github.com/ente-io/auth/issues/222
vishnumohandas··on Ente-io-auth: open-source 2FA app for Android, iOS and web
Hey, all of your tokens are encrypted with a key, that only you have access to. So in the case of a security breach, the attacker would only be able to access encrypted data.

Our architecture[1] is similar to that of password managers, and we've been externally audited[2].

[1]: https://ente.io/architecture

[2]: https://ente.io/blog/cryptography-audit/

vishnumohandas··on Ente-io-auth: open-source 2FA app for Android, iOS and web
Our primary source of income is our cloud for photos[1]. Storage and bandwidth consumed by 2FA secrets is minuscule in comparison.

Also, Auth has helped spread word about Photos, and we're very pleased with this side effect.

[1]: https://ente.io

vishnumohandas··on Ente-io-auth: open-source 2FA app for Android, iOS and web
Hey, one of the devs behind the project here. We were planning to do a Show HN sometime later, but thanks to OP for sharing this (and saving us from perpetual procrastination).

We've been building Auth for almost a year now. It started off as a pet project to cure our own itch. At the point, we couldn't find a cross platform open source app that offered e2ee backups. Once we had an MVP, we released the project to our community[1], who responded favorably, and with their help the project has matured to where it is now.

If you've any feedback or questions do let us know!

[1]: https://ente.io/community

vishnumohandas··on EU tells Apple to open everything up to its rivals
Looking forward to a healthier background execution framework so apps other than iCloud / Apple Photos can offer a reliable sync experience.
vishnumohandas··on Show HN: Dittofeed – 1-Click deploy, self-host Mailchimp alternative
Hey, looks very cool! If you could share a comparison against Listmonk[1], that'd be useful.

All the best!

[1]: https://listmonk.app

vishnumohandas··on Libsodium: A modern, portable, easy to use crypto library
This is arguably the safest library you can use if you are adopting crypto. The documentation is a delight to read and the abstractions make it difficult to make mistakes.

Fun fact, the author (Frank Denis) is an incredibly talented photographer[1].

[1]: https://prettysimpleimages.format.com/

vishnumohandas··on Self-hosted photo and video backups directly from your mobile phone
Hey, one of the devs building ente.io here.

Thanks for pointing out the workaround with GPS, will look into it!

Currently on iOS, our servers send silent pushes to your device every hour. This wakes up the app and gives us 30 seconds to execute a background sync (provided you've previously not killed the app). This is not sufficient to backup videos, but it works pretty well for incremental backup of photos (I personally use an iPhone).

← PreviousPage 5 of 12Next →