ActivityPub involves a HUGE number of HTTP requests. Every time you create a post your server has to deliver that post via HTTP to every other instance that has at least one of your followers - this can quickly grow to hundreds of deliveries if you have a few thousand followers.
Mastodon uses a Sidekiq queue and Ruby threads to handle these deliveries.
I have a strong hunch that switching to delivery through a more efficient mechanism - async IO, or a lower level language - could have a big impact there.
My web browser doesn't blink at loading 100 resources to render a web page, so delivering a few hundred HTTP requests shouldn't be impossibly hard to scale, even when you take retries into account.