There is something fundamental about zero that I've been trying to figure out how to state clearly, but haven't been able to put in words just yet. It is closely related to the above.
In this world, 1 is the empty set, primes are the singleton sets, and composites are anything with at least two elements. 0, being divisible by everything, is a superset of everything; it probably makes the most sense to treat it as {p->aleph_0} for all p, though it's rather undetermined -- you could pick another cardinal if you wanted.
The inclusion of negative numbers would basically freely adjoin a copy of this lattice with itself. You'd get two "empty-ish" things that are divisible by each other, two ranks of "singletons" that are divisible across the signs, and so on. Modding out by these cycles gets you the original lattice. There may be some value to keeping the negative numbers around -- certainly in category theory we like to keep equivalences explicit rather than modding out by them -- but "morally" you don't get any new relations by adding the negative numbers.
Many people think of Complex Numbers as vectors, and I think of negative numbers as vectors too. They have both magnitude and orientation, even if the orientation is only 1-dimensional. When you multiply complex numbers, the resulting "vector" has magnitude equal to the product of the magnitudes, and angle (from the positive x-axis) equal to the sum of the angles. So you can actually calculate these products without actually multiplying the numbers themselves, but instead only considering magnitude and angle (this only really matters with complex numbers). But the definition of a vector, that is has magnitude and direction, doesn't apply to 0. It doesn't have any direction, so it's technically not a vector. Adding angles doesn't make sense to something that just doesn't have angles. Zero is actually a mess when considering multiplication, and is generally excluded from the set of numbers you get to play with under multiplication. Another number you don't get to use for the same reason is infinity. A professor once told me an interesting insight, that zero is messed up in the same way that complex infinity is messed up. Neither has orientation, both have magnitudes that consume the other number when multiplying. And if you think about it, you could `s/0/infinity/g` in integer multiplication tables and they'd still work exactly the same.
[0] https://en.wikipedia.org/wiki/Gaussian_integer#Gaussian_prim...