There's no reason a PHP based site should take that long to load. It would seem that you are missing some caching settings, regenerating something every page load, or have something wrong with your server setup.

Make sure you are using a recent version of PHP and appropriate WP caching plugins.

Also check to see if your DB is hanging when a connection is made to it, perhaps due to trying to reverse lookup the IP for logging purposes and waiting for that to time out, or something else simple like that.