If you want speed, just have your database stored in the same place as your application, locally, rather than hopping across the world to retrieve data that can be located next to the code.
That would probably be the easiest thing to do to get a real measured performance gains.
As other commentators pointed out, computers are extremely powerful. This isn't 1995, you can easily host everything in the same local area and get a very responsive application with very minimal needs to worry about resource constraints.
Given how primitive SQLite's optimizer is and how similar the storage and execution engines between the two are in terms of architecture, this seems unlikely to be the norm unless you did something wrong on the Postgres side. (Of course, no RDBMS optimizer will always give the best answer, so there's bound to be such cases.)