Switch to the postgres backend. The default sqlite one can get quite slow in high concurrency situations, although I've actually had 50 peple running quite happily off the sqlite backend.
Are you positive switching to the postgres backend will help when there's almost no concurrency and the slowness is persistent and deterministic?
How are you hosting it? I'm using Apache 2.2 with mod_wsgi on Ubuntu 10.04LTS. Don't use mod_python - it's terrible.