Using primitive types “too much” is a frequent anti-pattern. It has a name: Primitive Obsession (https://wiki.c2.com/?PrimitiveObsession)
I think people do it because it is quite some work to implement proper value types in common languages like JavaScript, Java and C#.
Languages like F# or Kotlin give it to you almost for free.
The rule of thumb is exactly as you observed: if values have different semantics they should have different types.