Show HN: A fast append-only data store for Java 8 | Hacker News Reader

Show HN: A fast append-only data store for Java 8

probablyfine.co.uk

74 pointsprobablyfine31 comments

Open articleView on HN

An example of good idiomatic Java 8 source code.

Example:

https://github.com/mrwilson/java-dirty/blob/master/src/main/...

If only enterprise didn't move so slow =/. I wish I could migrate to 8.

Thanks. Curious... are the parens meaningful in this method body?

    private int getPartition(int globalInsertionPosition) {
        return (globalInsertionPosition) / this.sizePerPartition;
    }

No, that's likely a leftover from refactoring. Good spot.

Haha, no worries! I've been checked out of Java for a while and I was a little bit worried that this was semantically meaningful, which would be a pretty confusing addition to the language :)

What are the Java 8 features being used here, besides the obvious lambda expressions?

(genuine question, I haven't stayed current with Java)

Streams are also being used.

Optional is also new to Java 8 although syntactically very similar that Guava.

How would you compare and contrast this with http://openhft.net/products/chronicle-queue/? (which I knows does a bunch more but seems like it could be used for this purpose too).

This is not comparable to chronicle, this is one of the mechanisms by which chronicle works (memory mapped files).

http://stackoverflow.com/a/9268065

This reminds me of the log abstraction:

http://engineering.linkedin.com/distributed-systems/log-what...

> frequently find myself wanting a light append-only data store for pet projects

As someone who is mystified by that statement: why?

Could be used as an event store. Rather than storing a list of mutable objects as in a normal database, you store the changes themselves as a list of facts. This makes things like replication a lot easier and can be optimized to be much faster than a traditional database.

There's Apache BookKeeper to fill some of those needs.

I've never seen BookKeeper in production. Where would you recommend its use over something like Kafka?

Great, but that's not a product. If you're on board with the CQRS/ES thing, actually building your own should not be difficult. (Ok, so I'm assuming non-distributed DB here, but with a bit of imagination one could also do infinitely(!) scalable CQRS/ES within Bounded Contexts, given the fresh crop of easily understandable/implementable concensus algorithms.)

Auditing becomes easy. The history of all changes is automatically available.

If you know your data has no duplicate (eg, you want to sort and store a list of cities), you know you don't need to check the already-written objects; you only need to append objects.

I've used it as the backing for a time-series database and to store deltas for a state-machine - in both situations I wanted something that understood "POJOs" and didn't require a lot of setup.

Even in the case of multiple non-unique entries, if the stored objects are not meant to be mutable, the API exposes .reverse() to find the most recent.

Really nice, I am just needing this ;) Super that it is MIT licensed as well. I was halfway on building something on top of XZ SeekableInput Streams but this is probably a lot saner to use ;)

Great. I've been wanting something like this for a while. I've got a few questions:

1. How large files are supported?

2. It says that it will only persist primitive fields on objects - does that exclude persisting a byte array?

3. What are the failure semantics? For example, what happens if my computer's power supply fails in the middle of a write? How will the program behave the next time it is started?

Good questions, but wouldn't it be fun to find out the answers using empirical tests? You might even be able to add tests and submit a PR, and be immortalized in the annals of github.

You don't get things done very fast, do you?

> You don't get things done very fast, do you?

Please don't make rude comments on Hacker News.

It is a legitimate response to a problem that plagues the hacker community. Focus on the task at hand. Trying to understand the intricacies of a system can be educational but the best lessons learned are from completing your current project.

I'm surrounded by intelligent people that can't get anything done.

Ok, but the problem with your previous comment wasn't your view, but the rudeness. It's possible to express your view without rudeness; you just did. That's what we're going for here.

Hi, glad you like it.

1. Under the hood it partitions the log into sections to avoid Java's MappedByteBuffer size-limit. I've not tried to deliberately push the file-size limit yet, but it's something to add to the benchmarks when I publish them (soon!)

2. It doesn't support primitive arrays as of 1.4, but this is something that I have considered in future versions. Pull requests are definitely welcome.

3. I've not tested these failure cases yet.

+ For chronicle

Have look at my profile. Also there is KVEngine from H2.

I tried googling for H2 KVEngine but didn't come up with anything, did you mean http://www.h2database.com/html/mvstore.html perhaps?

yes

This actually seems pretty useful. I'll be sure to check it out for my next Java project.

Reply on news.ycombinator.com