On the higher quality end of the spectrum, I think one of the main differentiators is just how seriously the staff takes the production. They lean into the cinematography, storyboarding, directing, music, choreography, etc just as much or more as one might expect from western live action prestige TV. This is exceedingly rare in TV animation in the US and is only really seen in our animated movies (which are rare these days, at least in 2D form).
The other thing is that high quality anime is exceedingly good at building atmosphere and creating vibes. Half of the enjoyment of Cowboy Bebop for example is just basking in the ambience.