Have you experimented with a log-structured filesystem? I once saw similar high-variation behaviour in a logging problem on extfs. Switching to NILFS pretty much got rid of variable latency entirely.
In a few cases in the past when dealing with unimportant data I have downgraded to ext2 for a nice performance bump.
The first has a write-ahead log, the latter is basically just a log. So immediately writing to disk is relatively simple.