In my experience, without careful tunning, postgres performance starts dropping significantly around 10 million rows.
In my experience performance is often directly related to the ratio (total RAM available to Postgresql, shared buffer and buffercache included / total size of tables and indices ), save any weird usage pattern.