A New Approach to Databases — Simulating 10Ks of Spaceships on my laptop
blog.paralleluniverse.co
blog.paralleluniverse.co
I think the ideal way to demonstrate this kind of claims while eliminating the effect of other factors such as memory performance is to use a beefy server with many cores (let's say N >= 16), and adjust your benchmark to use 1, 2, 3, ..., N cores, then plot the line to see if it's really linear.
Two dots make a line in geometry, but you need more dots to project the trend.
Time to spawn an EC2 Cluster Compute Eight Extra instance!
Also, there should be multiple phases like, first everybody shoots, then everybody is hit, then blast force is applied. Otherwise there might be an unfair advantage for some spaceships, which can destroy their opponent, before it had the chance to return fire.
Naive implementations would have a single global lock on the resource, resulting in lots of contention. They appear to be saying that by converting to callbacks, they are able to determine which transactions will interfere with each other and limit the contention to just those portions. Deadlock would be a serious possibility here, same as in regular databases.
The size of the data is largely unimportant for this problem, it's the contention that's the issue.
If the spaceships didn't have to interact through the global state then you could have sharding. You can change the problem by adding "planetary systems", and saying that ships in one system can't interact with another. Instant shard.
Also, SpaceBase guarantees no deadlocks.
:) :) :) Isn't that a constraint on the application? :) :) :)
I think I'm going to have to look at the demo and see how a shot which triggers an explosion then does the query to perform the push of the surrounding ships.
I would expect something like:
Ship A:
Query(origin+direction+range, callback Shot())
Shot()
if Exploding
Delete
Query(origin, range, callback Boom())
Boom()
Update location
An AOE which causes cascades of death would be cool to see too.
Nifty idea.
I once wrote a ship builder and simulator for EVE and man there's a lot of variables to take into account: the ship, the pilot, modules, implants, boosters, etc.
Still, like you said, very interesting technology and hopefully helpful to CCP!
I'm no expert on parallel programming, but isn't this sloppy thinking? Like the implication is that we must have an exponential increase in cores in order to accommodate a linear number of users...
More generally is anything about this approach really novel? Databases can already run transactions and queries asynchronously. Any language that supports concurrent programming can take advantage of this, even if the database isn't issuing a callback. And ruling out concurrent languages that are capable of implementing this approach, but include some language features the authors don't like seems like a side issue.
Not just trying to snark... if someone can explain what I'm missing I'm open to it!
As the article says, you can just ignore those extra cores. But, if you want to do cooler stuff with your data you'd better use them, and if so, sharding is a dead-end.
> More generally is anything about this approach really novel?
I said it's a repurposing of an old idea, that of the old RDMSs where the application was the database. The idea is not about concurrent transactions but about uniting the application and the database and letting the database run and parallelize the application's logic, and be the engine of scaling. I think most people see the database simply as a data storage component, and try to make it fast and scalable enough. We say, let the database run the application.
> And ruling out concurrent languages that are capable of implementing this approach...
I haven't ruled out any language. In fact, the demo is in Java, which certainly doesn't qualify. Any language can do this. I only said that in order to go forward, in order for the common programmer to take advantage of modern hardware, we must use languages built specifically for modern hardware. I've named two that I think fit the bill, but if other languages end up becoming the language for the new age - it's all the same.
Since I found some things to nitpick I want to say I definitely like the approach you are coming at this from -- in terms of how you apparently view sharding, NoSQL, database as a processing engine, etc.
But I still think you are presenting it as something novel (just look at the title) when what you are really doing is promoting a long-established but sorely neglected philosophy... a good philosophy, which needs to be promoted. The novel aspect seems to be supporting callbacks in your Spacebase product, which isn't particularly essential to this philosophy.
Thanks for the article.
Database engines already possess sophisticated scheduling logic for query processing. We are constantly taking advantage of that, in different ways, when we delegate to the database.
And even though we are supposed to have all these business layers and whatnot, I think centralizing logic in a DB is a pretty common practice.
So I have been asking myself, is there a significant functional difference between the db doing a callback to the program, versus other methods of interacting with that built-in scheduling engine?
My supposition is that in the absence of such a capability, one could always get mostly (or fully) equivalent behavior by some combination of
(A) having the app fire queries (and consume their results) asynchronously
(B) giving queries some other means of initiating a response, e.g. writing to a table that acts as a message queue.
To me it seems like such a transformation wouldn't even change the balance of code between app vs DB all that much, it would just change the calling mechanism, and maybe you need these couple of extra modules to manage your query submissions and asynchronously dequeue stuff.
So I admit this qualifies as more than syntactic sugar, but not sure goes much beyond that. Of course, many prized language features don't go beyond that at all; and calling/concurrency idioms are important.
I like the idea of centralizing logic in the DB and I can see how your approach allows this in a much more elegant and straightforward manner. So I think I get it. Whether it's a totally new architecture, I dunno...
I come out of this thinking if there was a way to improve your article it would perhaps be to avoid the claim that this is allowing us to take advantage of database to "schedule and run" whatever business logic is implemented in the db via the highly optimized query processing facilities. All programs effectively do that in their use of the database, which is why the scheduling capabilities exist in advance of your approach.
The core thread of what you are doing/advocating, as I understand it, is something like this. (a) Centralize business logic in the DB. (b) Stop being ashamed of it. (c) Support calling conventions that make it easier.
If you made the argument that way, I would have seen it as a slam dunk. And I would expect that to be a controversial argument on HN.
Interested in your thoughts.
(BTW, I bet one could do these sorts of callbacks using the relatively recent capability of SQL Server to run Stored Procedures written in .NET.)
Our ultimate goal isn't to make a new programming model for its own aesthetic (or non-aesthetic) sake, but in order to take advantage of multiple cores -- namely, for the sake of performance. So the database does not "control" the application -- it actually parallelizes it.
And as for SQL server, yes, I guess it's a similar concept but a different kind of database. I advocate that the database should be part of the application; they should be sharing the same heap in the same process, and only then should the applications let the database "drive". That is why I also don't quite agree with your wording of "centralizing logic in the DB" because the DB and the application are one; certainly the DB is not more (or less) centralized than the app. Once you get that, there is no point in being ashamed of it. Just like you let your web-app container parallelize your presentation layer, you let the database parallelize your business logic. In both cases the middleware shares the same process as the app, and the leap of faith required isn't big at all.
It does make me think about the fact that almost every major DB today runs out of process and that the market seems to have selected this pretty definitively. It seems like only recently some products are running in-process for performance reasons. But if I understand the trade offs here, only a single application can run in-process with the db. Also the db is not protected from crashing when the app crashes. But this is the 1990's explanation... any thoughts on how that applies to your situation?
True. That's why I've tried to show the performance to be gained by letting the DB become integrated with the app. This is a result of the DB having just the right kind of knowledge about the relationships among domain objects to allow it to parallelize business logic.
> But if I understand the trade offs here, only a single application can run in-process with the db. Also the db is not protected from crashing when the app crashes.
Well, if the application and DB are one, so what? Note that we're only talking about OLTP DBs here. I don't see much of a reason for analytics DBs to be unified with the app. How often do you need more than one app working with the same OLTP data?
However, if you absolutely must, there is no reason why the in-process DB shouldn't store its data in shared memory, thus allowing other DB instances to access the information. There are ways to ensure that if one app crashes the memory is left in a consistent state.
If you are running a modern web startup, it's probably not an issue. But in a typical business... often IMO. And it was one of the original use cases that drove the adoption of databases in the first place.
Would be interested in seeing you blog about the last sentence of your last reply.
Thanks for engaging my comments.
Having moved one codebase OUT of T-SQL stored functions and onto Django's ORM, I can say that this article does not have me convinced the way forward is to move the code back INTO the DB. If my data model is going to be dictating my development toolset, it better be giving me some amazing performance gains and be as good as the toolset I've come to enjoy in a post-ruby-on-rails-world.
What am I missing?
And yes, the performance gains are amazing.
This makes it seem like it's either too high priced and you know it or the pricing scheme is too complicated and should be simplified.
It's just that as a young startup, it's very important for us to learn about our customers. At this stage, it might be beneficial for us to offer you a fantastic deal if we think we there is much to gain by the partnership. Also, learning about the problems our customers face is more important to us than sheer sales volume at this stage. That's why we want you to talk to us. But we're sorry for the inconvenience.
It's okay to qualify the number. "Ten million dollars, but order now and..."
If you'll note, your customer just said the same thing I did. Listen to him.
Also, the very same code would work if actions were triggered by a network request rather than by a main loop. We don't assume anything about how the spaceships are coordinated. If you coordinate them to run in lockstep, you could get better performance. But, again, we've tried to show the viability of the naive implementation, where nothing is assumed.
How exactly does clojure qualify while go and haskell do not?
As for Go, it allows sharing of mutable state, and so does not prevent races and does not make it clear when modifications to said mutable state are visible. It supports concurrency well but is not built around it.
http://hackage.haskell.org/packages/archive/base/4.6.0.1/doc...
http://hackage.haskell.org/packages/archive/stm/2.4.2/doc/ht...
[1] http://redmonk.com/sogrady/2012/09/12/language-rankings-9-12...
Seems to be more widely used than clojure.
>Also, I don't know whether it supports mutable concurrent constructs
Yeah, Control.Concurrent.STM has vars, queues, channels, etc.
In this case, at least, industry seems to be doing it right. C++ is reserved for the industries that really need it (gaming, embedded) and enterprise work focuses more on programmer productivity instead.
At the end of the day, each one of your VMs are running on hardware, the point about Clojure being JVM-friendly is more about ecosystem and interop then some efficient execution concerns. You could say Haskell is interoperable with everything and nothing, and be right on both counts.
If you had only written that, you wouldn't have been downvoted. Almost everything else you've written here is wrong.
The qualification is admittedly subjective but Haskell is not as practical for use in many use cases, whereas Erlang fits a niche very well, and the others (scala, clojure) are more general purpose. My assertion is based on what I know all my peers in the industry to be working on, and the sizes of the projects they are working on. If you have more data than this empirical data set, I'm all ears.
Edit, I see you reference tiobe and langpop.
This is a horrible* way of assessing industry usage, just fyi. Niche languages for example, may see relatively small general usage, but may be the de facto answer to a particularly type of problem. Many people use Erlang to solve their concurrency problems. Few use haskell (although that part is improving and I love haskell a lot).
Except that is entirely nonsense. Your misconceptions about the languages in question do not imply a particular amount of usage from other people.
>the others (scala, clojure) are more general purpose
I can't even imagine how one could make such an absurd statement. All that demonstrates is that you have not actually used haskell as you claim, or you only used it 20 years ago.
> My assertion is based on what I know all my peers in the industry to be working on
Which is ridiculous, just like claiming haskell is more widely used than C by only looking at the financial industry.
>This is a horrible* way of assessing industry usage,
It is a poor way, but it is infinitely superior to the "cause I said so" method you are employing. Erlang and haskell rate close together in all of them, so it certainly makes no sense to pretend one is more widely used than the other. Clojure rates miles behind both, and it is equally absurd to pretend it is so far behind in every measure but is actually ahead based on nothing.
I love Haskell and use it nearly every day to do mathematics and a number of side projects. Please step off your high horse and throwing all this ad hominem around. You're giving haskellers a bad name everywhere.
The author is right about there being few languages that do this, though.