We can and do! Use SML!
People just don't find out about the good type systems because the market is crowded out by mainstream shit.
And people write pieces about how good this shit is: "I was going to add an Int to a String but the compiler stopped me! Less [sic] bugs!". With arguments like that, why would you switch camps?
Or sometimes you get fed shit and get told it's inference: "I can replace the first word on the line with 'var'!".
So what are the downsides?
Inference doesn't play well with inheritance or mutability.
If you read Effective Java and see the points about avoiding mutation and inheritance, then these downsides may very well seem like upsides to you.
Why did you stop coding in SML?
SML uses Hindley-Milner (with some modifications to support imperative code). Many statically typed functional languages descend from it: Haskell and OCaml are the most prominent.
> Other languages haven't adopted a similar system so there must be some kind of trade off. What was the downside?
If you're asking why existing popular languages don't adopt it: they can't. HM type inference requires a HM type system, and they've committed to a different one. At best you could get good inference on a HM-typed fragment within the language (though even that would mean making a lot of painful tradeoffs) but it would break down the moment it made contact with existing code.
If you're asking why HM languages didn't become more popular? Historical accident, in my opinion. I don't think it has much to do with the type system, given how fundamentally awful Java's is.