Is this really an issue with Java's type system though? If you get an ArrayList<Number> and you assume that it contains only whole numbers, that's on you.
FWIW, my personal opinion is that erasure is better than templated generics for most use cases. Even when it comes to performance: a good JIT should be able to specialize generic code that end up on a hot path. Erasure does cause some issues with reflection and value types, but I think that it provides a better foundation to start from when designing a type system.
Then again, I am quite a fan of Rust's generics and dyn/impl {Trait}. Might be worth doing something similar where you have two separate generic systems: one for boxed values, and one for unboxed values.