The Architecture of Datomic | Hacker News Reader

The Architecture of Datomic

infoq.com

89 pointschillax19 comments

Datomic is intriguing and powerful but I'd be reluctant to use a proprietary database developed by such a small company. I think Riak has a better model for free/paid usage. There is nothing like Datomic but there are so many excellent open-source databases these days.

I'm the dev for an open source event-based database called Sky (http://skydb.io). Similar data model but it's aimed at a different use case than Datomic -- it's primarily used for analytics. The event data model is nice for avoiding conflicts but it's an up road battle to try and replace traditional relational databases. I find that storing historical events is primarily useful for either analytics or auditing. It makes more sense to move that off your primary data store if possible.

I do find that making databases smarter about the data they hold can really provide some great optimizations. Sky can crunch through about 60MM events per second on a single core because it is built for event data models. I'm sure Datomic can probably get some good optimizations too. I'll be curious to see some benchmarks.

Here's a demo app of Sky crunching all the events in the GitHub Archive (starts around 3m:20s):

http://skydb.io/blog/github-archive-visualizer.html

I'm working on multi-core distribution right now and I'll be doing multi-node after that. Check out the web site or GitHub (https://github.com/skydb/sky) for more information. It's MIT licensed so feel free to hack away.

You really need a "how it works" section. Few people are willing to read the source code to see if you have interesting or useful ideas.

That's a good point. I'm cleaning up and stabilizing the code right now and part of my v0.2.0 release is going to be documentation.

I did a presentation recently about some of the ideas behind the database:

https://speakerdeck.com/benbjohnson/behavioral-databases

The intro on the GitHub Archive Visualizer video explains some more of it too:

https://vimeo.com/51629936

I'll add that stuff to the README though.

I agree completely; we are looking at tech choices for a new project and Datomic is an excellent fit for the way we want our systems to work, but we're just not comfortable with the closed/paid model, especially for such a young product. For now we are looking at ways to use an accumulation based system (vs update-in-place) in PostgreSQL. If anyone has any experience with this I'd love to hear about it!

There is another database which can be used in the 'accumulation-based' space: http://geteventstore.com

It has the same main drawbacks as Datomic - it's young and multi-node deployment comes with commercial plan only, however it's really fast and easy to work with once you embrace the event-sourcing programming model.

Take a look at event sourcing. This is a good introduction. http://martinfowler.com/eaaDev/EventSourcing.html

How long do they plan on keeping it proprietary? Is there a timeline?

I doubt very much that there is a timeline. It's a product and the last I heard was staying close sourced. Having said that, I would not be surprised if some of its libraries are open sourced. I know of at least one that is in the pipeline.

Datalog for clojure's data structures would take Clojure to a new productivity level. It would also greatly simplify a lot of the code out there today, further driving software to be more data-oriented. Data is the ultimate API. Please make it be Datalog.

Datomic Free gives you that capability right now, for free, no strings attached, and no need to use a Datomic db. Not open source, but free as in gratis, a la Creative Commons.

We may trim down the deps a bit for that use case.

A trimmed down version is great news. Datalog makes Clojure so much better! Ideally it would be part of Clojure itself, but I'll take what I can get :)

I would also love to see a trimmed down version just to use datalog.

The datomic-free jar provides Datalog. I use it myself https://clojars.org/com.datomic/datomic-free.

I'm working on a port of BloomL for clojure which will include a semi-naive datalog interpreter (it will already support naive, monotonic datalog by tonight). It will likely never be as well optimised as the Datomic libs but it is LGPL.

Source:

https://github.com/jamii/droplet

Background:

http://www.bloom-lang.net/

http://boom.cs.berkeley.edu/

http://db.cs.berkeley.edu/papers/UCB-lattice-tr.pdf

If you say the datalog query aspect of Datomic, I would be tremendously (maybe embarrassingly) happy.

In related news Riak was in fact added as supported storage today: http://blog.datomic.com/2012/11/riak-and-couchbase-support.h...

Related: presentation on how datomic was written in clojure http://www.infoq.com/presentations/Datomic

Sorry for going off-topic, but is there any explanation why InfoQ does not have an entry on Wikipedia?

Is it just because no one has bothered to do it yet, or is there some other reason why this quite big, and frequently cited, site is not on Wikipedia?

Reply on news.ycombinator.com