For some combination of Drupal's general heavy-weightness, lack of caching, slow server, your front page performance is sporadic and mostly very poor. I get times mostly in the 0.6 - 1.4 sec range just for loading the HTML, and that's with about 0.15 sec as the baseline for just loading a similar-sized static image from the site. I'm not a Drupal expert even a little, but if you could fully cache the front page (so that apache serves a static file directly) and update it, e.g., once every 10 min, you'd be flying, comparatively.
But about what's going on, I'd lay money it's all of the above: you have a low-resource server, drupal isn't fast/light by default and you haven't made it any better, you're not using any caching where it would really help. You should work on fixing all of those.