Sets are one way (among others) to encode such mathematical structures. This doesn’t mean these structures are sets. For example, is the empty set an element of pi? This question is meaningless because the answer would have to depend on the encoding being used, and not on the properties of pi itself. It’s imposing additional structure above and beyond that of the mathematical structure in question.
See here for more discussion: https://news.ycombinator.com/item?id=16080027.