Mastodon uses a Sidekiq queue and Ruby threads to handle these deliveries.
I have a strong hunch that switching to delivery through a more efficient mechanism - async IO, or a lower level language - could have a big impact there.
My web browser doesn't blink at loading 100 resources to render a web page, so delivering a few hundred HTTP requests shouldn't be impossibly hard to scale, even when you take retries into account.