These days its reputation is mainly in the above, and as a glue language. It's simply not used as much for large standalone applications where people would really notice the speed issue, in comparison to something like Java (which is what comes to mind for me as at least at one time having the "slow" reputation).
And although it might not apply in the distant past, CPAN is large enough now that for any conceivable resource-intensive task, there will be a modules available to wrap an appropriate C library.
The text-processing focus, especially early on, might also be a factor, because if you're mainly banging data through regexes, it's pretty fast. The super-fancy regexes-that-are-more-than-regular do have degenerate cases, but they're fast in a wide range of cases, and in my comparisons usually beat sed.
Perl hasn't had a popular framework that has built as large a community of users (and detractors) as any other. They do have frameworks (like Catalyst), but nothing that has gathered as much attention.