BTW, if you have any feature request for django-mediagenerator please let us know.
1,122 karma · joined January 23, 2010
https://www.ensody.com
https://github.com/wkornewald
https://github.com/ensody
https://bitbucket.org/wkornewald
BTW, if you have any feature request for django-mediagenerator please let us know.
With django-dbindexer you can even use QuerySet features which are not natively supported by App Engine. At some point django-dbindexer will even support JOINs via automatic denormalization (or via in-memory filtering where desired) and maybe also simple aggregates and scalable counters (via QuerySet.update()). This will simplify complex DB code a lot (compared to App Engine's model API) and by reusing Django's ORM as the abstraction for common NoSQL patterns you can not only move between NoSQL DBs, but also between SQL and NoSQL (at least to a certain extent; the goal is to provide a better high-level NoSQL API and only implement scalable features, not build a SQL emulation layer that wouldn't scale).
2. The slow startup problem doesn't really matter, anymore, because App Engine provides reserved instances and warmup requests. It only matters to those who don't want to pay a few $ for reserved instances (but even without reserved instances there are ways around that ;).
Also, it often drops after lunch, but this also depends on when we have lunch. It's significantly worse if we have lunch around 12:00 than around 13:30-14:00. Our research group loves to go at 11:30 (!!!) which destroys the whole day for me, so a few of us have just stopped eating together with them.
Update: I've added a poll: http://news.ycombinator.com/item?id=2052007
Now we only need SSL support and here we come, blasting everyone away who wastes time on maintaining his own servers. :P Well, at least until the next feature is needed, but not available, so let's hope this won't be the case. ;)
There are several reasons why assets should be configured in the settings. The most important ones: (1) This is the only way to support sandboxes like App Engine. (2) This makes advanced features like HTML5 manifest support possible, at all (you have to know all generated assets in advance). (3) Assets are pre-generated, so when deploying a new version your users don't have to wait for yuicompressor and other stuff (e.g. upload to S3) to finish.
The only advantage of the template-based approach is that it's more "familiar". However, that approach has too many limitations and disadvantages. That's why we went with settings-based asset definitions.
I originally wanted use Copperhead and got in contact with the developer a year ago, but it was too early even for "private" beta testing, so I never got access to their code. Also, my compiler is specialized on image processing, so probably Copperhead wouldn't have worked, anyway. I'm only jealous of Copperhead's type inferencer. :) But then again, I have to get finished with my thesis and a type inferencer wouldn't help with that goal. ;)
As you've said, scalability is one valid reason for using a NoSQL solution. Sometimes you also have special needs for multi-datacenter replication or whatever. Sometimes a NoSQL DB can fit your problem better than a SQL DB. However, what does it look like from a development standpoint?
If you have an offline-capable app it might be easier to implement it with CouchDB.
Also, CouchDB's MapReduce views could allow you to run queries that wouldn't be possible with SQL. Note that MongoDB also has MapReduce support, but it rebuilds the whole index instead of only the parts that changed, so on a large DB this might take too long and thus be impractical. Anyway, I guess you're talking about NoSQL solutions in general.
In some cases you actually want to store very flexible schemaless data in your DB. SQL would be a huge pain, here.
It might also be easier to switch from an XML DB to e.g. MongoDB.
Migrations can sometimes also be easier with a schemaless DB. This could be important when experimenting or developing iteratively.
Anyway, most of these are all rather special use-cases and that's probably the important development-side strength of NoSQL: Having solutions for special problems.
Some people here say that working with NoSQL is more natural than with SQL. If you're comparing hand-written SQL code with hand-written code for the low-level MongoDB API then that's indeed true. However, in practice you should better use an ORM because that makes you more effective. When comparing, for example, the Django ORM with MongoEngine you'll see that the APIs look similar. It's just that SQL gives you infinitely more powerful queries and an ORM makes writing even complex queries very easy. A NoSQL DB sometimes forces you to jump through hoops to implement something that would be a two-liner in SQL+ORM. So, from this point of view NoSQL has nothing to offer for a large number of problems. Of course, some projects only use very simple queries which map nicely to NoSQL. In these cases it doesn't make any real difference whether you use SQL or NoSQL, but that still is not a real plus for NoSQL.
One other reason that once spoke for NoSQL: Google App Engine and Amazon SimpleDB were among the first solutions which offered scalable, replicated, managed hosting with an acceptably cheap pay-as-you-go model. Today we have Amazon RDS, MS SQL Azure, and soon App Engine will have SQL hosting, too, and there are several other cheap cloud hosts with SQL support. So, this reason isn't valid, anymore. It might still be part of the hype, though.
Yes, NoSQL is indeed over-hyped. Too many newbies use it for the wrong reasons (esp. on App Engine where they use it because they can save $3/month). Only if you need to scale or if your project has special needs you might find rescue in the NoSQL world.
Finally, since many of us are working on a web startup: As long as it's not clear if you'll become successful and thus have to scale you might be tempted to use SQL and develop faster in the early phase of your project. However, when looking at a complete web project the development/code overhead caused by NoSQL (vs SQL) is often not very large (well, depending on your particular project), so you might want to go with NoSQL right from the start and not have scalability woes later on. Anyway, you need to make an informed decision for your particular problem. Don't just pick NoSQL because it's hyped and also don't just pick SQL because you're wary of hype. :)
BTW, if you want to go with NoSQL and still have more complex queries like with SQL you should join our open-source project django-dbindexer. The goal is to automate denormalization and index generation such that you can use e.g. simple JOINs and aggregates with Django's ORM instead of emulating them with hand-written code. This should be useful to a lot of web startups and make it easier to use NoSQL instead of SQL right from the start and not worry about scalability. See here:
It's possible that with the upcoming pre-warming requests this bug will disappear in practice because your whole project gets a chance to import and initialize all modules before the first request is received. If these pre-warming requests don't have a 30s timeout your instance will always have enough time to finish the initialization in a non-broken state.
Anyway, it's your own mistake if you stay with the Django helper. The bug can be worked around. It's fixed in Django-nonrel. I still see people who try to use the helper for mission-critical projects. Don't do that. The helper is buggy. It uses monkey-patches all over the place. This makes it extremely vulnerable to bugs caused by DeadlineExceededError. Just use Django-nonrel, even if you only want to use App Engine's models.