It's probably small bits of foam or maybe even sand. When there's air in the balloon, there's room for the sand to move freely, and it's "soft". When the air is sucked out, the sand has to move past other sand, which is a lot of friction to work against, so it's firm.