The argentine ant supercolony is an enormous swarm, but we don't really expect to be able to have a conversation with it, or see it deal with abstract concepts, but they can solve lots of local approximate optimization problems.
Slime molds are single cells but you might think of them as a swarm of nuclei and organelles that can solve hard optimization problems when in the right form, but again they don't seem like a general intelligence.
Is the missing piece that swarms must be of agents with certain properties? Or that the interactions between the members of a swarm need to have some specific properties?