Things I Wish I Had Known About Django Development Before Starting My Company
medium.com
medium.com
13. You almost never need to write bash scripts. Use django management commands to write scripts that interact with your application. Use fabric and a fabfile to do deploys. Use something like chef or puppet to do machine configuration. Your bin/ folder will turn into an unmaintainable mess very quickly.
14. Use south. Yes, it's obtuse, but it gets the job done.
15. Use a pre_save hook on all of your models to do full_clean for validation before anything goes to the database. This will save you from cleaning up your data later.
from django.db.models.signals import pre_save
def validate_model(sender, **kwargs):
if 'raw' in kwargs and not kwargs['raw']:
kwargs['instance'].full_clean()
pre_save.connect(validate_model, dispatch_uid='validate_models')
16. If you're writing a javascript-heavy application, consider all of your options for static asset management before you get too deep. I've used django-compressor, django-pipeline, webassets, and django-gears. None of them are perfect solutions (there still isn't a sprockets-esque one-catch-all solution for django like in the rails world), so consider the pros and cons before you make a choice.edited for formatting
I had this problem with django-compressor as well (but more with configuring 3rd party asset compilation). I usually recommend people use webassets via django-assets. It's easy to configure and very feature-ful.
This is definitely a little frustrating. I'd kill for a solution that handles minification, versioning & can upload to S3/etc. You can pass things through boto to get to s3, but the rest requires a bit of fiddling.
I've seen a few solutions that come close, but then they have extremely odd versioning systems. I should just fork something so I can add an ISO8601 timestamp to my CSS & JS; which I like because having granular dates helps when debugging front-end issues.
Would you code for one? I, too, am sorry that we don't have a thing like asset pipeline. I'm pretty sure it would be welcome...
I had to solve issues with custom domains, HTTPS, gzip, separate media/static files buckets, and storages/compressor not playing nicely together in general.
I don't mean to insult the asset pipeline, it provides a lot. But it definitely doesn't save you hours. Out of the box the asset pipeline is great for all the things that come for free but if you are doing a lot of development in the framework you probably return to the asset pipeline documentation on a regular basis. And I consider time spent in documentation a negative compared to time finding your own solution if the API is not intuitive and you find yourself consistently returning to the docs about similar problems. And let me tell you, plenty of my fellow Rails devs have said to me they also regularly return to the asset pipeline docs.
class ValidatingModel(Model):
def save(self, *args, **kwargs):
self.full_clean()
super(ValidatingModel, self).save(*args, **kwargs)
Then inherit from that everywhere.Signals have a not-insignificant overhead. They are most useful for tying together code that comes from different places. (If you want to do full validations on 3rd-party app models, then you might this signal approach, but it might also be a bug in the app that the hook isn't just there for you.)
Is it just me, or is it a little absurd to have a wrapper for a wrapper to a Python module?
I've had to resort to
$ vimdiff <(pip freeze) requirements/prod.txt
You can't just do "pip freeze > requirements.txt" like you see on every tutorial. I wish there was a better way to do this.And I would also like to only display without dependencies. Say I "pip install X", which depends on Y and Z as well. Then later I want to delete X, so I just "pip uninstall X", but I will still have Y and Z.
It's worth noting that I'm a pretty big NodeJS fan and am even using grunt via node for building the client bits in my latest .Net project at work. It's pretty sweet, though some addons are broken since the new grunt version.
In the top level of my project I have a makefile so make assets runs grunt and collects static.
My grunt also runs jslint so it will halt if there are errors.
basically though, I've used requirejs for the js both for single page apps and for general js like plugins and galleries.
compass/sass for the css which does minification and concatenation already, and also ensures there is no illegal css
js is processed by requirejs into STATIC_DIR/r/
css is compiled into myproject/static/myproject/css and then uses django's staticfiles system to collect and deploy
grunt is what calls requirejs. it could also concat and minimize css but compass has already done it. it also has watch and does live reload so that my browser will reload the css and even the js when either of them are edited.
on my pages I use a tag:
{% vcss "nestseekers/css/front.css" %}
which renders a <link css tag with a ?v=HASH appended
and
{% r_url 'nestseekers/js/libs/dist/html5shiv.js' %}
for the js
We use brunch.io a node.js build tool. It handles concatenation and minification of CoffeeScript, Stylus and Handlebar templates for us but it can work with almost any front end technology out there. A cool thing is that we can use require within out CoffeeScript files and that allows us to better organize code into modules.
Brunch even provides file watching capability so when we save a stylus or CoffeeScript file it will update the browser without a reload.
In our setup we compile everything to our django static folder so when we collectstatic, files get uploaded to s3.
One thing that was a bit a tricky and something that we have to make better is cache expiration of files. For now we are appending static files with a few characters from the current commit hash (app.css?e2232 something like that), while this is pretty effective we've found some cache systems that ignore it. The next step is to rename files before uploading to s3 (app-e2232.css).
I agree that this might be a bit too complicated for most apps but in our case (getblimp.com) we have a pretty heavy JS app and we really need to take advantage of all the "better" front end tools available.
I will be willing to contribute to FOSS project to solve this issues once and for all on the django/python side. BTW I think that file watching is better than compiling on every reload during development. In our experience (many js files) request on the development side took seconds and became very frustrating.
Am I missing something? What's wrong with:
return HttpResponse(json.dumps(data), mimetype='application/json')
Wrap it up in a convenience function and you're done.The JSONResponse class suggested automatically implements JSONP, which is extremely dangerous. Consider a view on /accounts/info which returns some information about the currently logged in user. A malicious site could embed
<script src="http://example.com/accounts/info?callback=someFunction">
and access the account information of any user logged into your site. JSONP is a technique to bypass the same-origin policy in appropriate cases; don't just blindly apply it everywhere or you're giving up the protection of the policy.The way to deal with is __complex__ as a method on the object and recurse through asking the complex method to return nested simpler python types.
http://docs.python.org/2/library/functions.html#complex --and-- http://docs.python.org/2/library/cmath.html
This is strange advice; while you can use gunicorn as a front-facing webserver, the gunicorn docs strongly recommend against doing so. In a typical deployment scenario, then, gunicorn and Apache would occupy different levels of your stack, with one running your WSGI app and the other exposing it to the world. The advice ought probably to be "Use gunicorn+nginx instead of Apache+mod_wsgi," and indeed, lots of people do make that recommendation about Django deployment.
(I work at Phusion, and am not a Django guy, just curious)
3. use nginx to reverse-proxy WSGI to Gunicorn over a unix socket
6. don't have per-environment config in your app, use the same config in all environments and configure each host with aliases/proxies to consume the appropriate resources (use local_settings.py for local development on your workstation & exclude it from your package)
7. use native operating system service management (systemd, upstart, launchd)
10. use munin if your app can withstand a CPU spike every 5 minutes, use ganglia/graphite/collectd otherwise
I'm working on a large app at the moment where all hostnames are the same regardless of environment, and the only way to switch envs is via a proxy. It's a massive pain.
Keep environments mostly similar, but always make sure you can configure locations of the app itself and its various external resources separately on staging, test and development platforms.
One of our projects works like this, it sucks.
For per-host app location, we set X-HTTP-Script-Name in the nginx location block and extracted it in WSGI middleware like http://flask.pocoo.org/snippets/35/
Everything else is hardcoded to use 'http://localhost/service_name/foo & nginx proxies to the appropriate resource. Obviously, this doesn't work for non-HTTP services!
This. I'm tired of Django projects not being packaged/released correctly, and deploy scripts that simply git clone in production (yiikes).
I agree the annoyance wrt needing the same environment as the target. We tend to have two platforms that are supported: CentOS 5.x and MacOS. Keeping the build machine on the same platform as the deployment machines is simple. Creating the eggs for MacOS developers is more difficult, but still not too bad. That might seem odd, since we could just use pypi.python.org, but we have an internal PyPi server so that we can easily share internal libraries. Adding a line to a project's setup.cfg makes this trivial for the application developers.
There's another annoyance with sdists. I don't want to compile during deployments. So, I build everything that can possibly be built as an egg as one, and fall back to sdist for everything else. I push those to the internal PyPi server. At deploy time, I create a virtualenv and easy_install the appropriate artifacts. I know the correct artifacts because I `pip freeze` the requirements at build time.
We're also extra paranoid, so our stage and production VPCs are on different AWS accounts. We have one PyPi server per VPC and flow artifacts forward as needed.
> What's wrong with a git clone deployment, why is deploying as a package better, (...)
The advantage is that you can have a release process, where you update the package's __version__, compile/minify the files you need, build documentation, and so on, until you have a deployable branch that you can tag and upload to the repository. This way you have a reproducible history of releases, it's easier to inspect which version is deployed, you have hooks for installation (for instance, you can abort installation if the tests fail on production), etc. Mainly this:
Crank out code -> Run a makefile/fabfile to update version/compile/minify/build whatever -> Export a tag -> Build an sdist/bdist -> Install on production
I believe too many things grew inside Django (e.g. collectstatic) that really shouldn't be part of the framework at all. Another thing that bothers me is South: you need to push a release to production, then run a migration, because the migration is part of the codebase. Well, the migration really should be part of the installation process. There are corner cases where this is an issue - for instance, worker processes reloading before your migration is complete would use the new, wrong model definitions, and suddenly you have a broken release on production.
For all effects, your Django project should just be a valid Python package that you can pip install in a virtualenv inside your server (lets say, /opt/<myproject>) and you're done with it. This way you can freeze the environment on production, pip can handle upgrade/downgrade, you don't have to care about *.pyc hell, etc.
> and how does one implement it correctly?
I should probably upload a project template with a workable setup.py to github.
- use South (right away!)
- Class Based Views
- (not django specific) use virtualenvs!
http://www.kickstarter.com/projects/andrewgodwin/schema-migr...
South is pretty obtuse to learn and definitely has it's warts and issues, which Andrew seems well aware of and wants to fix so check out the kickstarter link below!
That said, our project has over three hundred tables, a couple thousand migrations, and I've been very happy with it compared to the systems I've rolled myself in the past. I'm certainly unaware of an alternative that holds a candle to it, assuming Django's ORM.
Automatically generating properties after querying the database is seen as unpythonic. PyLint and some IDEs/editors will not be happy if you try.
Ideally, we should use the declarative style of Django models and SqlAlchemy on the object, along with the scheme generation tools of Rails migrations.
Write SQL and interact with your RDBMS directly?
so I ask again: what's your alternative (given that avoiding raw SQL scripts is the problem you're trying to solve)?
But I mean idempotent SQL takes care of the vast majority of all this stuff you mentioned. Write your SQL properly and it doesn't matter whether or not it's been applied before.
Even if you don't use South's automatic migration generation, its database-independent schema modification api [1], dependency tracking, and tracking of which migrations have been run, are useful and necessary. In fact, the automatic generation of migrations is an optional feature of South that was not in it when first released.
When you need to write SQL and interact with your database directly, using `db.execute` inside a South migration is a nice way to do it.
[1]: http://south.readthedocs.org/en/latest/databaseapi.html
I do, however use class based views that I've built myself. We extend them and add mixins and I much prefer all of this to the decorator soup that is the alternative.
1. Use rbenv or rvm. Half the problems I notice with beginners stem from installing rails with apt-get or something.
2. Your controller is meant to be simple glue between model and view - if you're putting tons of code in there, you're doing it wrong.
3. Unit test. A lot. This is especially good for beginners because you often make little errors when you're starting out.
4. Unit testing isn't enough. Functional testing is important too.
5. Learn about your hosting provider. Specific to Heroku, if you're on the free plan, every dyno spin-up takes absolutely forever if you don't have a steady stream of visitors.
Yes, fat models are the way to go. This is just a good rule of thumb for any MVC framework. I feel like no framework docs actually explain this. Newbies end up shooting themselves in the foot because they don't understand the reason for separation or understand where logic should be implemented.
I don't want to get into exactly what MVC should mean and whether any particular framework does it right though.
Also, if I want a user to input numbers to do calculations, where does the computationally heavy code go?
The article that edavis mentions does make one good point, which is that the term "view" was chosen to emphasize the notion that the python callback function (or object) sets up a view of the data represented by the models. I think there's something good in this choice of words. I find the terms "view" and "template" more intuitively meaningful than the corresponding "controller" and "view".
One important, substantive difference that I'm aware of between controllers in many MVC frameworks (probably not all) and views in Django is that a lot of MVC frameworks map requests to controller actions through URL traversal, whereas Django offers a more hands-on approach with URL confs.
Also, whereas controllers are often implemented through classes in MVC frameworks, views in Django may be implemented through any callable object which accepts a request and returns a response.
As far as where your code should go, it probably depends on what kind of data the user is entering. If the data must be persisted in the database somehow, such as with transactions in a bank account, then the code that crunches the numbers would probably be in a method on a "BankAccount" or "Transaction" model class. If it's data that only really needs to persist for the user's individual browsing session, then it's probably safe to put the code that works on it in a view. Django limits what kinds of things you can do in its templates (thankfully), so you probably won't have much luck putting your computation code in a template.
1. Tracking tasks status in SQL will result in a lot of queries! (Even when using Redis/RabbitMQ as brokers.)
2. Crontab style tasks are great... until they take a long time to complete and Celery kills them. And then you're back to regular crons (and there is nothing wrong with that).
3. Use UTC everywhere! New projects get this by default, but don't make this mistake.
RabbitMQ for the result store is madness.
For a start, rabbit handles OOM better. It has lower per-message overhead. It has a more flexible queueing model in general so that you can use it for both work queue-y stuff and other queuing needs. It's interesting how often queues seem handy once they are easy to use.
Then there's the fact that redis is good for lots of things other than queues, and you'll be tempted to use it for that, and that will crowd the queue use. If you hold a transaction on redis or do a large set intersection or run a lua script, you block everything else, including your fanned-out celery worker pool.
The most comprehensive best-practices resource of Django I've come across by far is "Two Scoops of Django"[1] by "pydanny"[2]. Its absolutely worth the $17.
Seriously, between the two of us she's the better coder, taught me LaTeX, and knows Strunk and White. I just write what I think and let the grammar experts fix it. :P
One of those tools that you start off thinking isn't useful but quickly turns into a must have.
Why not just go nginx + django the usgi way? Now you have friendly tutorials and docs for this like https://uwsgi.readthedocs.org/en/latest/tutorials/Django_and... . Maybe I'm weird or I've never worked on large enough apps, but I fail to get what the shiny unicorn and gunicorn bring useful tot the table...
Mongo has collection-level locking? I thought there was some preliminary work to support it in 2.2 (It just does database-level locking for now), but it's still unsupported. Source: https://jira.mongodb.org/browse/SERVER-1240
Also, I wouldn't recommend using any of the aggregation stuff unless its for infrequent ad-hoc stuff or scheduled tasks during slow periods.
If you want something a bit more distributed (read less prone to single points of failure) than Supervisor, check out pacemaker. Very powerful and useful for keeping resources alive in any sized cluster.
edit: looks like they're using flask now, I don't see itty being used anywhere. They also split the web interface (resweb) into a separate project, so pyres doesn't depend on a web framework now.
It is not transactional, at least not with a MongoDB back-end. If you stop a celery service, any tasks that were in progress may or may not complete, but none of them will be picked up again when you resume.
No prioritization of queues or tasks. It is possible to set up separate services that handle different queues, but that's a PITA and there's still no guaranteed order of processing.
[1] http://highscalability.com/blog/2013/4/15/scaling-pinterest-...
What do you mean it "compiles over the wire" in development? In development, pipeline just renders individual js/css tags for each of your static files, un-compiled...
the basic premise of the django defaults is that app modules are pluggable and self-contained. all of the static assets, templates, model code, view code, migration files, management commands, etc. related to a particular app should go in that app's folder (which is treated as a Python module with its own __init__.py file). conventionally the app's folder is a first child descendant of the top level project directory (i.e. its in the same directory as manage.py)
[1] This was a few versions of Django ago; if they fixed this since then, I welcome corrections.
Deploy to Heroku (or similar). Saves a ton of headaches, saves boatloads of time/money, and for most startups in the early stages, is well worth the tradeoff.
I see it as a build step before fabric or puppet would take the output and deploy it.
Several issues:
the buildout process is long and monolithic and not conducive to minor adjustments. I'm using ansible now and its much better for just changing one setting on an nginx config file or settings file and reloading. I chatted with the author of buildout and he said he was building to an rpm and then mounting that as his means to do a live deploy.
I often wished I had virtualenv, so many things work well with it. for instance python-vim and sublime lint / rope like to have a virtualenv to get the python paths. also ctags is happier if I can just enter the virtualenv and run it.
there may be a recipe for that but I never found one that worked nicely.
For example, if Apache can handle your traffic just fine, why spend time replacing it with gunicorn? Or if you never really migrate your database ever, why waste time fiddling with South?
Just a gentle reminder to take into consideration the present and future needs of your project to avoid needlessly adding complexity to it.
http://www.michielovertoom.com/freebsd/flask-gunicorn-nginx-...
I'm running mine under NGINX->uWSGI....but might be switching to Gunicorn. (I'm still researching our path to production).
1. The right directory structure, the default use for the media folder is supposed to be using for stuff uploaded through your app, not the files you develop, your static files should go in static and your compressed files should go in media as they are dynamically generated or in your global "static" folder that is pointed to from your web server. You should also have different settings.py files on a per-situation basis. Your development servers, staging, and production environments will probably all have different settings because at the very least development will have `DEBUG = True` and production `DEBUG = False`.
2. This is fine even though it seems wonky to use it for cron jobs to me.
3. This is fine too but I personally use nginx+uwsgi w/ emperor mode.
4. Up to you on this one, there are some nice Django branches that support key-value store databases but don't use the standard Django with key-value databases because it is 100% built around RDBMS.
5. Only piece of advice I agree with completely.
6. As partially discussed in #1 by me. I don't like his override method. I think an import of a base settings into production.py and development.py is cleaner.
7. Supervisor is good here but also if you use uwsgi in emperor mode instead of the suggested gunicorn it can handle the same task saving you an extra install and configuration.
8. Django very clearly has a nice Mixin for JSON responses in the docs or you can build a nice easy API using Tastypie: https://docs.djangoproject.com/en/1.5/topics/class-based-vie...
9. It's up to you if you want to use Redis, I don't personally need it for all the suggested things and I like how well memcached works with Django out of the box for caching.
10. Munin is great! I'm often too lazy to set it up and am fine reading log files.
11. This drove me to write this long comment... I actually get really annoyed when people try to mix in items from other stacks when there are many solid solutions already that don't force you to install another entire stack. Django has django_compressor which works great in this situation (https://github.com/jezdez/django_compressor) and a quick Google search will find many other similar solutions that won't require you to install Ruby to work with your Python web app.
Source: 6 years of developing Django apps and doing everything that was suggested here and more.
It uses classes to define your settings, and you can use inheritance to override and mixins to augment your bits of configuration. I usually have a "CommonSettings" class, with a few others extending it (Production, Development, etc.). At runtime the correct class is set based on an environment variable (you have to add a line or two to manage.py and wsgi.py to start the magic).
Anyway, it doesn't seem that popular but I've enjoyed using it (and done so without problem).
> If you’re hungry for acronyms, you might say that Django is a “MTV” framework – that is, “model”, “template”, and “view.” That breakdown makes much more sense.
right?
http://zachholman.com/screencast/vagranception/
and have been meaning to mess around with it ever since
That said if you are storing data that is really just documents, Mongo or another document db is likely a good choice. For example if you storing interrelated performance measurements (like in a factory setting ) Mongo would likely not be the best choice but say in a setting like an app for applying for a job or registering for events ( natural document data segments ) a document db would work well for modeling the domain I'd think.