You can swap out Django's built-in ORM for something else (like SQLalchemy) and you can also swap out the django templating for something else if you want.

Flask http://flask.pocoo.org/ is pretty awesome and I've used it for a project that was really light on server-side code (http://palettecomp.com/), but I've never used Sinatra so I don't know how it compares.