They should have made it extremely clear in their original manuscript their CT results are against a non-pretrained version that was not always trained to convergence, and so the CT results presented are a just a lower bound on performance.
They got confirmation from a Google engineer on only one block, which they trained for 1 million steps. For the other blocks, they didn't train anywhere close to that many steps, and their tensorboards don't show convergence on total cost.
ML is difficult! It's not enough to confirm that your implementation and installation are correct - you also need to train to convergence. You are free to complain that this is onerous or expensive. It's a common problem in academia vs industry in other areas of ML -- ML takes a lot of compute, and academic budgets are extremely tight. That said, CT is still orders of magnitude smaller / cheaper than most modern ML models (not surprising! CT is pretty old at this point by ML standards (3 years is a long time in this field)).
Kahng's FAQ admits that pre-training led to a better final proxy cost in the Nature paper, and that it significantly reduced compute time, and that this is the method that the Nature authors recommend and used in their Table 1 evals.
This whole situation is such a shame. Prof Kahng wrote a really great article on CT a few years back, and it gave me hope that the whole field was going to embrace ML and really take chip design to the next level: https://www.nature.com/articles/d41586-021-01515-9
Most of the field in fact has, but Kahng himself seems to be backsliding here.
Unfortunately, I suspect that Kahng's been listening to some pretty toxic people (the same day his paper was posted to arxiv, it appeared in an SB author's court filing). It seems like his group ran into some difficulties jumping into machine learning, and rather than take that as an opportunity to grow and learn, they took that as an opportunity to attack.
While I'm glad that others in the field have built on the work successfully (see Appendix of https://www.annagoldie.com/home/statement ), it would be really tragic for Kahng himself to get left behind here, and so so unnecessary!