NeDB, a pure Javascript database for Node.js and Node Webkit | Hacker News Reader

NeDB, a pure Javascript database for Node.js and Node Webkit

github.com

83 pointslouischatriot46 comments

As someone that has a JS background and is looking to get started with node, this is a welcome addition. I like the idea of being able to throw a require statement in and get up and running. Syntax for inserts and queries look straightforward too. Thanks!

Thanks! The syntax is the same as MongoDB's, so that developers used to MongoDB (most Node.js devs are) can get started on NeDB right away.

The trend continues where all things eventually get implemented in Javascript. Waiting for JS.js

With the rise of very fast javascript VMs, lots of things can now be done in the same language, javascript. That doesn't meand everything should of course, but a Node Webkit app is definitely a case where I want a JS DB instead of a native one. Less powerful of course but less of a pain for my app's users.

It's a very cool idea, I wish I had an excuse to use it.

https://github.com/mozilla/narcissus, https://github.com/jterrace/js.js, etc.

Narcissus is an excellent name for a meta-circular evaluator

Indeed. In my mind much of the value of javascript is for rapid protoyping and for allowing those who enjoy JS (I do not [1]), to build medium performance tier applications.

What strikes me as odd is people wanting to write things that traditionally are written in C/C++ and more recently also in Go/Rust/Java/D in JS. The reason these arguably slower, in development speed, languages are used for things like DBs is that they are write once re-use thousands of times, and this use case warrants being slightly lower level and less rapid in development and gaining performance and I would argue maintainability.

[1] Note: In case you are wondering why I dislike JS: 1. JS's extensive use of spaghetti cough I mean callbacks 2. Higher effort needed to use compute resources for a single task in a parallel fashion compared to other langs 3. I dislike dynamically typed languages (I belive they are harder to maintain) 4. Performance: http://benchmarksgame.alioth.debian.org/u64/benchmark.php?te...

> What strikes me as odd is people wanting to write things that traditionally written in C/C++

I should note I know people who make this choice. They can write these things in C, and in the past, they have. But they are choosing Node.js. So it's not just a "what you know" affect.

The goal in this particular case is to have all your code in a single language for easy distribution. If you write and distribute a Node.js webapp or Node Webkit app (which I guess you don't since you don't like JS!), needing your users to also install whichever DB you use is a pain for them (esp. for desktop apps with Node Webkit, users may not be developers). That's the reason I built this.

First off, louischatriot kudos to you for building this. Building embedded DB's is fun and worthwhile. The fact I'm not a JS fan doesn't diminish that value to me.

While I understand that packaging motivation; it seems to me to be less relevant due to msi/rpm/deb packages. Those technologies make it pretty easy to package an application built with heterogeneous components.

Thanks :)

It's true that it's easy to package such applications for unix systems, but it's much harder on Windows. What makes Node Webkit great is the fact that it's cross-platform with very little pain, and in that case having the DB in the JS code makes packaging a breeze.

Great idea and project. Questions: a) Is the database loosely coupled to query language implementation? b) In my biased opinion RethinkDB has a much better query language. Is there a way to implement RethinkDB's query semantics and how much work is involved? c) Underscore.js has a ton of collection manipulation functions. How easy is it to use it from within NeDB? d) How easy/difficult is it to run NeDB as a lightweight RESTful server, in its own Node process?

Thanks! To answer your questions:

a) and b) The database is loosely coupled with the query language implementation, so it could be possible to change the query semantics without breaking it. That said I have other priorities for now (some adaptation for Node Webkit, porting it to the browser).

c) NeDB queries return vanilla objects or array of objects , on which you can use all the underscore functions you want.

d) Depends what you want to do. If you want to write a REPL I think it shouldn't be too difficult, and it may be a good idea to do one. In the meantime, since the API is the same as MongoDB's, you can learn NeDB using MongoDB's REPL :)

Thank you. Can you elaborate your statement "you can learn NeDB using MongoDB's REPL"? Specifically steps to use MongoDB REPL connected to NeDB.

I was referring to the fact that the syntax is the same so if you don't know the NeDB syntax you can play with the MongoDB REPL. Appart from this there is no connection between the two.

Perfect timing. I spun up a project fairly recently using node-webkit and wanted this, exactly. Getting SQLite integrated wasn't particularly painful, but a document storage model is just what I had hoped for. Thanks for the contribution; looking forward to giving it a try!

Thanks! I am finishing a small helper that will enable you to specify the location of the datafiles relative to the directory where Node webkit stores data. You should find it helpful.

It might be interesting to persist snapshots to S3. It might be usable for small projects on Heroku/etc with ephemeral storage. It could be patched here (?) [1], but clearly there'd be a lot of issues with the long network call, how often to do it, how durable it really is, etc.

1: https://github.com/louischatriot/nedb/blob/b1abd34384cefea16...

Reminds me of a beefed-up version of this similar project: https://github.com/chjj/node-tiny

node-webkit seems to have a number of options for local storage: https://github.com/rogerwang/node-webkit/wiki/Save-persisten...

Amongst those are a native SQL engine (sqlite) and a native NoSQL engine (levelDB, through the IndexedDB API)

It'd be useful to know what benefits NeDB brings over other existing storage mechanisms.

I should stress this more in the README ... The key difference is that NeDB is a "real" (meaning with an actually powerful query and update language) database that is written in pure javascript. That's not the case for the other engines that need you (your users actually) to install a third-party DB or compile it. That makes prototyping, distribution and packaging much easier, and that was the main motivation (appart from fun) for this project.

From what I can make out, this won't work under cluster, right? You won't get new data written from one worker in another. It also has no locking so you can easily corrupt the database if you access it from multiple programs at once (or with cluster).

It's not supposed to work with cluster and using it with multiple programs that update/remove/insert docs will cause unexpected behaviour, although no data corruption since the datafile is append-only.

But if you need multiple programs to use your DB at once you need a more powerful solution such as MongoDB. The goal of NeDB is to bring DB features to desktop apps or low concurrency webapps, not replace Mongo and the like.

No I get that, I'm not saying it should support those things, just saying you should probably document it clearly. People don't like to lose data.

And append-only doesn't make you immune to corruption. There's still a possibility of two processes writing at the same time corrupting what gets written to the file.

Perhaps you should provide an option to have the db itself running as a child process (via cluster), and CRUD statements are actually proxied to that process (via a Unix pipe or socket) instead of updating the data directly in the process' state. This would work well as an extension.

From reading the description if you need this functionality you should move to something else as the functionality you mentioned is out of it's scope.

Just saying, I think it should be documented in the README.md as SQLite would be fine in those scenarios, which it tries to compare to.

I see why you would think that, this part of the README is not well formulated and I'll change it. What I mean is that is similar to SQLite since it is an embedded database, but the key difference is that it's pure JS so you don't need to compile or install anything, which makes it very handy for Node Webkit, low concurrency Node.js webapps an rapid prototyping.

I totally get the use-case. As the author of Perl's DBD::SQLite I know exactly how high the demand for this sort of thing is. I think it's a great project and wish you luck with it. (and if you ever decide to lock the files, I'm also the author of node's fs-ext library which provides flock() support).

What does this have to do with node-webkit? My understanding is that nodejs is a subset of node-webkit, so by saying "this is for nodejs", you imply "this is for node-webkit".

That's true, but I want to stress Node Webkit because NW apps are probably one of the best usecases. Also, there is a bit of helper code specific to Node Webkit, so that you don't need to worry about where the data is stored.

node.js used to see really slow GC cycles (>1s) at over ~5/600MB of heap usage, and bomb out fatally at 1GB. Is this still the case?

If it is, you should probably put a larger warning about dataset size than the line you currently have :)

I think it's better now, it handles as least 1.4GB in the heap. That said if you have a database with more than, say, 300MB of data, I think you need a more large-scale database.

As an aside, I'm considering enabling large datasets for NeDB as some users asked, but that would be a lot of work so I'll wait for more requests.

Nice welcome addition to the node webkit landscape.

Agree. This just make writing simple apps that much easier. Great idea!

How do you pronounce "NeDB"?

I may need to choose another name, you're not the first one to ask this :) I pronounce it letter-by-letter as it is an acronym for "Node Embedded DataBase"

That makes sense, but then maybe capitalize the E. I think that would make more sense orthographically anyway to signify that each letter represents a word.

At first I was trying to think of words that start with "ne-" before I clicked through and realized "E" stood for something on its own :-)

My instinct was to pronounce it "knee dee bee", so I think there's definitely an ambiguity there. On the other hand, that didn't stop people from naming GIF, SQL, or nginx, so I don't see why it should stop you.

Indeed. "enn eee dee bee" (or "any dee bee") was mine.

Awesome! I've been looking for a project like this for a small app and have been exploring things like EJDB and UnqLite, among others. I'll definitely be checking this out. Thanks.

Great to hear :)

How is this different from mongodb? Aside from being written in javascript and having a non-existent community around it?

It's different in that it's embedded in your application (like SQLite). It does not require a separate process or server running Mongodb. There are certainly appropriate use cases for this module, and since it has a Mongodb-like interface it can be swapped out for Mongo at any time.

Yep. I also made a small script that automatically transfers all your data from a NeDB to a MongoDB: https://github.com/louischatriot/nedb-to-mongodb

You can use that once you've outgrown NeDB.

Reply on news.ycombinator.com