You’re right to call out just how enormously expensive nuke boats are. Anything carrying around a nuclear powerplant for that matter. They are the modern day battleships (Just see how the US Navy names theirs). For AUKUS, I tried to keep my argument to the warfighting value of the boats separate from the economics. But there’s a whole line of argument there you call out, which I think finalizes with the question “Can Australia afford to build forces necessary to participate in a Pacific conflict away from its shores?” Not off the Australian coast, but north of Indonesia where they envision defending Taiwan, protecting the Malacca strait, etc. I don’t think that’s an easy question to answer. I don’t think their previous plan of getting new AIP boats was a good one if that was their aim. But it’s fair to question if they can afford nuke boats either.
I disagree that rocket forces can be a stand-in for submarines. They’re great defense weapons: certainly for shore defense, questionable for longer distances that that (I’m making an obligatory nod to China’s ballistic anti-ship efforts), and of course, air-launched AShMs were a primary threat during the Cold War. But these all lack persistence, and require a lot of effort to target the thing on the other end; something made much more difficult the further away it is. And that’s the thing a submarine VLS cell gives you that an air-launched or shore launched missile doesn’t; It’s right there. I don’t think the subsonic speed is much of an issue when it pops up 10km from your task force. It’s the persistence, the endurance, that gives submarines (and surface ships for that matter) their role that land and air based missiles can’t. The latter can make things painful, but they can’t truly control a body of water for duration in the way those platforms can.
But you’re right to point out that China’s math is likely different than that of say, the U.S., because the latter has to come across the Pacific to do so. Force projection is expensive, and this is the challenge facing Australia. The U.S. can afford to do it (though I think it’s slipping in this regard), other major militaries such as the U.K. are facing hard questions about just how global they want to be. Australia is far enough away that if they want to help in those aforementioned scenarios, they need to bring the fight north, and that’s an expensive prospect for a country of just 24 million.
The last thing I’ll mention is while I think your points about the challenges of operating in the South and East China Sea would be very difficult in the face of enemy ASW, those submarines would not be operating alone. They operate as part of a whole (land, sea, air). It's not the pacific submarine campaigns of WWII and lone wolves hitting shipping. Though if it turns into that, we’re in real trouble.