The downsides are the characteristics that make h3 or s2 useful. For h3, the single neighbor type means it is well suited to flow analysis and S2 having exact cell subdivision means it is great for simplifying geometry.
However, there a number of use cases where choosing a spatial index is a more stylistic choice, like for visualization.
The aim of A5 is not to replace S2/H3 but rather to offer an alternative that has different strengths and weaknesses compared to existing solutions