And if you just have unit type, empty type, sums and products, then you can only make types with finitely many inhabitants. Adding the integers to that makes you able to make any countable type, but potentially in a trivial way as described above.
Therefore just viewing the types as sets does not give a very satisfactory basis for determining whether a type is an "algebraic data type". Maybe we could do better if we consider some more aspects related to type safety, but it's not immediately obvious to me how.
I agree that viewing the types just in terms of their cardinality isn't terribly insightful. But I do think it's helpful to realize that arrays aren't some other kind of thing that's separate from ADTs, and that you can bring the same algebraic reasoning tools to bear on arrays (and other pedestrian data types).
Now that I think of it a bit more, it seems to be a category error to say that a "set" or an "object" or a "dictionary" is an algebraic data type. Algebraic data types are types (values) constructed out of simpler types (values) in a certain way, and that can be deconstructed in a way that mirrors the way they were created. You can implement various APIs/interfaces on top of that, including dictionaries and sets and whatnot.
But my point was that you can do the same on top of just integers (assuming you have the common integer operations), and you could even package it up in a type safe way so that the fact that you used integers in the implementation does not leak out, assuming the language has some modularity/abstraction mechanism so that you can make stuff private to a module. But based on this it seems to be a stretch to say that dictionaries are REALLY just integers or bitstrings.
And just like you probably wouldn't want to bring "integer reasoning" to bear on dictionaries, you probably don't want to use ADT reasoning to reason about dictionaries either. Because then you are really reasoning about the concrete implementation details and generally speaking those details are hidden from clients of the dictionary (and subject to change at any time).
I think a nice corollary to your second point is that the existence of an isomorphism between two collections doesn't mean they are the same thing. This is easy to see when you discuss integer reasoning. E.g., if we have a infinitely countable data type, so we have an isomorphism to Z. But that doesn't automatically make our data type a ring (not in any meaningful sense).
I guess you explained it better than me. What makes them algebraic is the way you think about them and the full set of types they are part of. For the types themselves, the name is meaningless.
For what it's worth, not all functional languages emphasize the ADT notion per se. E.g., the Ocaml manual generally talks about records, tuples, and variant types; "algebraic" really only comes up with GADTs are discussed.