To me, the big question is what do we mean when we say "quantum computer". This is a thick stew mixing together complex math -- and computer algorithms and architectures. It's like a trifecta.
When Google comes out and announced "quantum supremacy" only to get pushback from other quantum teams suggests we still don't even know what we're trying to build.
I think everyone knows what we want to build (A large-scale quantum error corrected device that we can run shors algorithm, etc on). But we are nowhere remotely near that so we have to argue over much less impressive milestones (not that what google did isn't impressive, it is. Just far off from our quantum dreams)
We know what we're trying to build. Pushback came from that shared understanding, not from lack of it.
The part about replacing sets with categories is inaccurate. Some people are talking about categorical foundations, but as far as I know, most work is aimed at HoTT and/or (infinity-)groupoids. Thid is of course not divorced from category theory but the focus is important.