dogleg77

Nature papers get retracted every year. I have not heard of DAC papers retracted.

If the Nature paper made it clear that RL is not seriously expected to work on non-TPU chips, it would have have probably been rejected. If RL works on many other chips, then evidence should be easy to publish.

The point is that the Cheng et al results and paper were shown to Google and apparently okayed by Google points of contact. After this, complaining that Cheng et al didn't ask someone outside Google makes little sense. These far fetched excuses and emotional wording by Jeff Dean leave a big cloud over the Nature work. If he is confident everything is fine, he would not bother.

To clarify "you'd expect" - if Jeff Dean is correct, he'd deny problems and if he's wrong he'd deny problems. So, his response carries little information. Rationally, this should be done by someone else with a track record in chip implementation.

Chip designers run EDAtools on premises. How would EDA companies have access to customer data? Maybe for debugging purposes under NDA, but that won't allow training on customer IP.

No, it doesn't suggest that. Complaints are often dismissed on technicalities or because they are written poorly.

Google claimed their new algorithm as a breakthrough. If this were the so, the algorithm would have helped design chips in many different cases. Now, the defense is that it only works for some inputs, and those inputs cannot be shared. This is not a serious defense and looks like a coverup.

That is misleading. The first two authors left Google in August 2022 under unclear circumstances. The code and data were owned by Google, that's probably why Kahng continued discussibg code and data with his Google contacts. He received clear answers from several Google employees, so if they were at fault, Google should apologize rather than blame Cheng and Kahng.

I am referring to direct comparisons in Cheng et al and in Stronger Baselines that everyone is discussing. Let's assume your point about "might have been insufficient". We don't currently have the luxury to be frequentists, as we don't have many academic groups reporting results for running Google code. From the Bayesian perspective, that's the evidence we have.

Maybe you know more such published papers than I do, or you know the reasons why there aren't many. Somehow this lack of follow-up over three years suggests a dead-end.

As for "belittle", how would you describe the scientific term "regurgitating" used by Jeff Dean? Also, the term "fundamentally flawed" in reference to a 2023 paper by two senior professors with serious expertise and track record in the field, that for some reason no other experts in the field criticize? Where was Jeff Dean when that paper was published and reported by the media?

Unless Cheng and Kahng agree with this characterization, Jeff Dean's timing and language are counterproductive. If he ends up being wrong on this, what's the right thing to do?

At least they are acknowledging skepticism now. That's a step in the right direction.

Markov's article has been on arxiv since 2023 before the perceived conflict. https://arxiv.org/abs/2306.09633 All his affiliations have been disclosed and likely known to Jeff Dean. Jeff is simply unable to respond to Markov's article in the technical dimension and is resorting to dirty tricks instead.

You must be kidding. They did reach out and documented their interactions, with names of engineers, dates, and all. Someone is ghosting someone here.

How would I know either way?

Someone mentioned here before that Google folks have been using "hyperbolae". So, if MediaTek can clarify how they are using AlphaChip, everyone wins.

Please specify what you mean by "had a hand in TPU" and where else he had his hand. Thank you

Something is off with your source references. Chatterjee's amended complaint is a legal document filed under penalty of perjury, accepted by a US judge, available publicly, apparently not "thrown out". How does an earlier document figure into this? How do we know it was "thrown out" and for what reason? Obviously, a later document is what matters.

Also, you are using an unreviewed document from Google not published in any conference to counter published papers with specific results, primarily the Cheng et al paper. Jeff Dean did like that paper, so he can take it up with the conference and convince them to unpublish it. If he can't, maybe he is wrong.

Perhaps, you are biased toward Google, but why do think we should trust a document that was neither peer-reviewed nor published at a conference?

The Google internal paper by Chatterjee and the Cheng et al paper from UCSD made such comparisons with Simulated Annealing. The annealer in the Nature paper was easy to improve. When given the same time budget, the improved annealer produced better solutions than AlphaChip. When you give both more time, SA remains ahead. Just read published papers.

I am trying to understand what you mean here by potential to disrupt. AlphaChip addresses one out of hundreds of tasks in chip design. Macro placement is a part of mixed-size placement, which is handled just fine by existing tools, many academic tools, open-source tools, and Nvidia AutoDMP. Even if AlphaChip was commonly accepted as a breakthrough, there is no disruption here. Direct comparisons from the last 3 years show that AlphaChip is worse. Granted, Google is belittling these comparisons, but that's what you'd expect. In any case, evidence is evidence.

What did the TPU physical design team say about this publicly? Can you also point to a statement from MediaTek? (I've seen a quote in Google blog, but was unable to confirm it). Who in the TF-agents team has serious physical design background?

Reading those papers and looking at the code, it doesn't look easy. However, let's imagine that the Cheng et al team comes back with results for pretraining a few months from now, and they support the conclusions of their earlier paper. What should they do to help everyone reach a conclusion?

When Google published the Nature article, Nature included a rosy intro article by a leading expert in chip design. His name was Andrew Kahng, and he apparently liked Google at the time. But when he dug into Google code (released way after publication), he retracted his intro and co-authored the Cheng et al article. You see how your theory breaks down here.

Good point, but Google is buying Nvidia GPUs for some reason. Please remind me who's buying TPUs.

That made sense when Jeff Dean gave talks in 2020, 2021, and 2022. He is now responding to skepticism from the EDA community by unscholarly personal attacks and vague references to "many companies" using the work. He is beyond benefit of the doubt, and into the realm of probable cause.

That's an appeal to authority, and not an effective one. Jeff Dean doesn't have a good track record in chip design.

Yes, that's unfortunate. In that case, the Google Nature paper doesn't belong in Nature. Google is definitely free to claim that they revolutionized their business, but without scientific evidence this is just marketing.

Fair point. I looked at the addendum, and it doesn't really address the critiques. They show an "ablation study" on one additional circuit without describing how big it is, etc. The few numbers they give suggest that this circuit is unlike those in the Nature paper: possibly a newer technology (good) but much smaller in the total length of wires and hence components (bad!). They are trying to debunk the Kahng work, but they aren't addressing many other complaints. Without showing results on public benchmarks in a reproducible way, they are just rehashing some excuses. Maybe their results are no good, maybe they stopped working on this. But with claimed runtimes under 6 hours, they don't need 3 years to add thorough benchmarking results. Anyone who designs competitive chips is doing benchmarking, but Google isn't. Draw your own conclusions.

Science is not done by quotes from VPs, and we don't know how MediaTek used these methods. Also, would you like to hear from VPs who wasted their company resources on Google RL and gave up?

The more marketing claims we see, the less compelling the Google story is.

Your perseverance is as admirable as it is suspicious. You are the lonely voice here defending the Google announcement.

The problem with the Google Nature paper is that its results were not reproduced outside Google. You can attack attempts to reproduce but that only reinforces the point: those claimed results cannot be trusted.

Other commenters already addressed the pre-training issue. Please kindly include a link to Kahng's 2023 discussion addressing your complaints. Otherwise, you are unfairly supporting those people you know.

Kahng's placer is open-source and was used in the Nature paper. It does not make sense to accuse Kahng of colluding with companies against open-source.