Why not? There's not much tethering our axioms-on-paper to what is necessarily true, past what we can empirically observe. For instance, a universe that is "exactly like ours, except the truth of the continuum hypothesis is flipped" seems no less conceivable than our own universe, given that we don't even have any solid evidence for its truth or falsehood in the first place.
If we're willing to treat mathematical and logical ideas as physically contingent, then it's only a few further steps to "the concepts of identity and discreteness and measure in this universe are different than ours, so all our mathematical axioms are not applicable". Though it would be very difficult to translate any stories from such a universe into our own ideas.