nateps

Also, here is a video from a very informal talk I gave on similar concepts last year at our office: https://www.youtube.com/watch?v=iTC5i63eOzc

DerbyJS (http://derbyjs.com/) is built specifically for these principles. I'm working on big updates to the documentation right now, but I can say that we have a great solution for Server HTML + Client DOM rendering, realtime data updates, and immediate optimistic updates on user interaction in the client.

By building on ShareJS, DerbyJS uses Operational Transformation for collaborative editing of data in realtime, which means that users can see data updated immediately in their browser even if other users can edit the same thing at the same time. This is the same approached used by Google Docs.

Hey! I'm Nate, one of the founders of Lever and our CTO. Sourcing is a word that companies and recruiters use to mean going out and finding candidates that aren't looking for a job (passive), but that have a really high relevance to what the company is looking for.

As hiring has gotten increasingly competitive, companies have been finding that the best candidates usually come from a personal connection. The best candidates usually come via a referral from someone's personal network or via sourcing efforts, where the company goes out and finds people that are a really strong fit based on their interests and experience.

Lever has a number of features that specifically make sourcing easier: https://lever.co/features.html. The best is our Chrome Extension for easily creating candidates in Lever from a web profile. We also have really great 2-way email sync that uses IMAP to sync Gmail threads in realtime, so that your company can follow along on what is happening with candidates and make sure that you don't drop the ball.

Good point. Thanks for mentioning. Made the logo at the top link to https://lever.co, which probably makes more sense.

Email integration is usually done by manually BCCing or sending emails from an email address generated by the system. BCCing is difficult to do consistently, and replies don't get captured automatically. Sending emails from a special address has much lower deliverability and more importantly is awkward for the person receiving it.

We send emails as you over Google OAuth and Gmail SMTP. That way, the email literally comes from you and shows up in your sent folder. The advantage to doing this in a shared system like Lever is that everyone else on your team can see the full history among multiple people.

When you send another email in Gmail or the recipient replies, we use IMAP to listen on threads associated with Lever. That way, we capture emails much more reliably and they don't have to come from strange random email address. It is really cool stuff, and it happens just as fast as Gmail can show an email, because we are acting just like a full email client.

Congrats! Glad to see you ended up with such excellent investors, and will be looking forward to continued improvements in Firebase! Looking forward to more progress making all web apps realtime.

While we've only written a MongoDB plugin so far, we have designed specifically to be work with any similar datastore. Contributions of other database plugins are welcome. The Mongo implementation is here for reference: https://github.com/share/livedb-mongo

This module is intended specially for spiders to be able to see something of what the rendered page would look like. That provides an SEO stop gap, but it does not have any effect on regular users. Server-side rendering the first request greatly decreases the first page load time, which is typically extremely slow for complex client-only web apps.

OT is done at the document level. Documents are JSON objects of arbitrary complexity. Any mutation of the JSON object can be composed against any other mutation of the JSON object.

Firepad doesn't actually do OT of JSON. In Firepad, a string OT algorithm is used to merge text edits, and the metadata describing rich text is separately managed as span objects, which get modified in accordance with text changes.

Firebase provides conflict resolution of JSON structures using their Transaction primitive, which allows them to create a Software Transactional Memory used to update the journal that powers the text OT in Firepad. ShareJS inside of Derby uses a similar approach to allow multiple servers to access the journal, but it is done in Redis with Lua scripting.

Definitely try playing the collaborative story game demo explained in the post: http://stories.derbyjs.com/

http://sharejs.org/ is in use for a number of production sites, and we've had users report that it is quite reliable. How would you like to see it further generalized?

We are still actively developing Derby and Racer. We are currently working on a re-write of Racer's internals on top of ShareJS that will make Derby useable for a production app. See: http://blog.derbyjs.com/2013/03/26/getting-derby-ready-for-p...

Our initial goal in open sourcing the framework was to get some ideas out there and see what worked well and what didn't. We've learned a lot from this process, and we now have a much better idea of how to build a great system.

We are excited to see all of the innovation coming from Meteor, Firebase, and other new realtime systems. While there are similarities, our teams have different perspectives on what the future of web development looks like, and we believe there is still a great deal to be gained from further development of each of these platforms.

Currently, our changes to Racer will keep a similar API but make Derby apps much more stable and scalable to large production deployments.

To clarify, I agree that RDBMS's are fantastic, and you pointed out many of the benefits. We are not religious about SQL vs NoSQL.

For our particular use cases, we think that document stores are a better fit, but I am not arguing that document stores are universally easier or better.

Actually, we are big fans of modular libraries. We know that a lot of the code in Derby and Racer is too monolithic, and we intend to restructure as the projects mature. For example, we already refactored out Derby's simple HTML parser (https://github.com/codeparty/html) and our wrapper around Express for routing on the client and server (https://github.com/codeparty/tracks). These are now indendent npm modules that were originally part of the Derby codebase. Also, as you mentioned, the Racer data-syncing and model library can be used separately from Derby.

Derby is an npm module, and developers are in control of their own standard Node.js/Express server files. We use Browserify (https://github.com/substack/node-browserify) to bundle up scripts, so you can use any standard npm module on both the server and the client without modification.

We have a long way to go in terms of customizing Derby; however, it is already very flexible in some ways. For example, Derby apps can be rendered and sent to the client from Express routes or via Derby app routes. There's also a method to simply get a blob of rendered HTML or render a static page using the same templates. Derby includes model-view bindings, but it's simple to bind some things and not others or nothing at all.

Much of Derby and Racer is very different from how web apps are written today, and we are focused on iterating quickly to start. As it matures, Derby will become more of a collection of independent modules. That way, most developers could use Derby out of the box, but others could fork the project and customize or use features of the framework independently. It's a ways off, but that's the long term goal.

Thanks, Zach!

This is correct. We have been developing and testing auth with our own app, and we'll publish a simple example using it soon.

Thanks for the support!

We understand that authentication and authorization are critical, and we have been ironing our issues with them using the app that we are developing as a first test. Pretty much all of the required code is already in master, and an example is forthcoming.

In the future we would like to make it easier to do relational queries, but we plan to continue to use document stores instead of RDBMS's. Document stores are more easy to map to the way that data is represented in JavaScript, and they make it easier to reason about versioned data. Currently we support MongoDB, and we intend to support other similar stores like Redis, Riak, Postgres HSTORE, etc.

We just added a lot of functionality to our query system, but it is very much a work in progress. Making it easier to create queries that represent relationships is definitely on our task list.

Personally, I agree that sticking with npm is the best solution. We have managed to do this with Derby (http://derbyjs.com/) while still providing easy app creation, automatic packaging, realtime data syncing, and reactive model-view bindings.

It's awesome to see that Meteor is now permissively licensed, but I would prefer to see them embrace npm as well.

I can't speak for Meteor as much, but Derby's bindings support pretty much everything that Knockout and Ember can do. The big difference is how you connect the bindings to model data. Knockout has a generic API that you can pass pretty much any JSON data to. This is flexible, but you end up having to write a lot of glue code to hook it up to whatever type of model you use. Ember bindings only work with Ember models. Derby automatically knows to look in its model to bind data.

Knockout is pretty minimal and fast, Ember has not performed very well in update intensive benchmarks. From our testing so far, Derby is somewhere in the middle if you bind everything, but it has granular control of bindings so you can get it to perform about as fast as Knockout.

This is just one small part of synchronizing clients, and Derby takes care of realtime data syncing as well. Synching data with ember requires a lot of manual server implementation. In addition, Ember and Knockout don't have any way of supporting server-side rendering, while Derby automatically renders everything in both the client and on the server.

You are right that you can't partially implement OT. The way that we handle it is by keeping OT operations on defined paths so that they are separately namespaced from other kinds of updates.

The reason we use a document based datastore is that it is easier to think about conflict resolution, and it maps more easily into how JavaScript represents data in the browser. It is common for large-scale eventually consistent systems to use document based stores instead of RDBMS.

We also are big fans of testing, and we will have a better answer to how testing can be done. We currently use Mocha, which is a great test runner for the server and the browser. All of our frameworks are still quickly iterating and adding important features like this.

Comparing PHP and Rails to Node.js is not quite right, as PHP and Rails are application frameworks and Node.js is primarily a platform for creating application servers. There are a number of more analogous frameworks being written on top of Node.js, but none of them have reached a similar level of maturity and adoption yet.

I believe that the author is arguing that Node.js could reach a similar level of popularity as Rails if there were a framework that provided some of the ease of use for common use cases, such as creating typical CRUD apps. However, he also states that in order to displace Rails, such a framework would also need to offer some distinct advantages, such as better support for realtime.

Backbone, Ember, Knockout, etc. are client-side frameworks. They make it easier to structure complex client code, but they don't include a simple solution for creating a server or synching data among clients and servers.

It's still under active development, but I have been working on a full-stack framework on top of Node.js that is more like what the author was envisioning: http://derbyjs.com

Derby provides server and client rendering, and it automatically synchronizes data in realtime. The idea is to make writing realtime apps so simple that any app can take advantage of advanced realtime features, fast server-rendered page loads, and instant client-rendered page updates.

This is the approach that we took with the Derby framework (http://derbyjs.com/): we use a template engine that parses Mustache-like templates and infers bindings at the same time. You don't have to write any additional bindings declarations, and everything updates in realtime among different windows and users as well.

We plan to support reactive functions, more complex queries, and authorization, but such features are not implemented yet.

Hey, I am one of the authors of Derby. Knockout and Backbone are pretty different, and they both have good aspects. Knockout's strength is its powerful view-model bindings, and Backbone is targeted at providing an MVC structure. However, neither of these libraries claim to be a complete solution. If you want a more full-featured client framework similar to these, SproutCore is one of the better ones right now.

The reason we started working on Derby, is that none of these frameworks offer a full-stack solution. Backbone and SproutCore will provide better structure and some automation of client-server data syncing. However, they still require manually creating appropriate server routes that handle AJAX requests and communicate with a database. They may provide client-side routing, but these routes are totally different from the server routes, so if you want to use the same URLs, you have to duplicate routing code. SproutCore uses Handlebars, so you might be able to render the same templates on the server for faster page rendering, and non-JavaScript browser or search engine crawler accessibility. However, this will require writing a bunch of custom server code to pull out the right data and render the templates. In addition, none of these frameworks provide tools to deal with data conflicts. Conflicts are inevitable in a multiple users realtime application, especially when supporting offline.

Derby's focus is to unify as much code between server and client and to provide automatic realtime syncing among all models and views. This approach requires less code, delivers faster load times, provides instant client-side rendering from full pages to individual data bindings, and supports offline by default. It's a work in progress, but let us know what you think about it so far.