To be fair, I should really switch it over to being a static site, I am using a complicated templating engine to pop out a few paragraphs.
Of course that is the ultimate irony about WordPress. Like any other swiss army knife framework, it will always be slower than a proper dedicated tool.
Another issue (one that a lot of modern development practices share) is that WordPress lets you quickly pull in components to Get Things Done Quick, but those components pull in more dependencies. I was looking at adding some more smooth scrolling effects, and the suggested way was to pull in jquery (fair enough), even though the template I'm using already pulls it into the page. Repeat that a dozen or so times to get the page looking Just Right, and, well, there goes perf.