Maybe for the next community, instead of making it public - potential users will instead have to find the site by solving a bunch of hidden puzzles. Think secret club.
Jokes aside: a major part of why HN works is that the community here self selects for several traits which ensure healthier communication.
As a forum becomes more mainstream, many things happen to attenuate the signal to noise ratio. (Pg had an excellent essay on the mid brow argument effect, which for many other forums is a problem they wish they had)
Eventually part of the crowd who came first realize it's not as useful anymore and decide to go somewhere else. The question is where.
We've seen eternal September come to almost every place and there's only a few traits which can help protect against it.
From my experience with many forums/modding, demographics are 80% of the issue, the remainder is topic of discussion, goal orientedness in discussion and moderator quality.
So the best solution is to create the forum and never advertise it. Let potential users figure out how to find it and how to get in.