Comment by lrossi | Hacker News Reader

Comment by lrossi | Hacker News Reader

Parent Full thread

This is an interesting view on DeepMind’s protein folding result, from the perspective of a researcher working in the same field.

I liked in particular the section where he compares research work in the industry with academia.

Advantages of research work in the industry:

- team structure favors focused work: no administrative chores, fund-seeking etc;

- stable teams consisting of experienced professionals, instead of grad students who do not have as much experience, and labs with a lot of churn;

- large groups working together for the same goal in a “fast and focused” approach.

He also points out that AF2 has a large team of 18 coauthors, which is something very rarely seen in academia, where teams are almost always small.

I have seen the same in many CS fields. Major papers published by FAANGs always have very long lists of authors, in addition to dozens other engineers credited inside. It’s becoming very difficult for university labs to compete with this kind of groups, since it takes a lot of effort to bring together even small groups from 3-4 different professors.

The author points out that there are important drawbacks caused by research work shifting towards industry:

- little visibility into how they reach their results: there are no lessons learned about what was tried and failed, and why, at least not outside the company;

- it’s often not clear which parts of their solution are important and why, and why certain parts were fine tuned or combined in a certain way;

- dissemination of knowledge outside the company is often minimal;

- focusing on results and metrics minimizes exploration of related ideas that are not directly important for the current objective and metric.

On top of that I would add the risk of accepting misleading results due to limited peer review (see Theranos).

I think these points are spot on. While on one hand it feels like we are making major technological progress, on the other hand a lot of the knowledge and the ability to use that knowledge effectively may become locked inside a small group of companies. While their current intentions are noble, I think it would be better for society if major research work continued to be carried by institutions that operate in the public interest.

He also points out that AF2 has a large team of 18 coauthors, which is something very rarely seen in academia, where teams are almost always small.

In physics, we routinely have author lists of hundreds to thousands of authors on the big experiments. When resources are pooled to make a measurement, author lists can be huge.

University groups can absolutely compete with teams from industry, especially in consortia. The big difference I see between industry and academia in this context at the moment is that academics have to submit a grant proposal and compete for 6-12 months with scientists nationwide for $1M in compute-credits, whereas someone inside a company looking to flex its ML muscles may be able to requisition the same with an email.

If computation is the bottleneck, one might imagine that crowdsourced computing like Folding@Home and BOINC could offer an opportunity for less resourced teams to compete.

Such resources are only really good for a particular kind of distributed computing: mostly the kind which is easily verifiable, and easily broken down into small embarrassingly parallel chunks which need a lot of computation done relative to their size.

Training large machine models is almost the opposite of this: the datasets are huge and don't split up easily, and iterations (which touch the entire model and a chunk of the dataset) are relatively fast. Even with dedicated hardware a lot of the challenge in getting these systems to work effectively is in moving the data around between different compute nodes. Something like folding@home just will not function for such a workload.

While interesting, I think this model is only viable for a certain subset of computational workloads.

AF2 took like two weeks of computation on 200 TPUs to train. I’m not sure if they used a data parallel or model parallel distributed training approach, but I don’t imagine either would scale particularly well to a globally distributed network of worker machines in a folding@home analogue...

The problem of experienced professionals vs grad students seems to be a real problem for science. Academia could really do with a build-out of experienced research scientists able to make careers out of consistently building experience and knowledge.

Instead most labs are oriented around a PI that has little time to look into details of most projects due to grant-writing pressures, post-docs who are not expected to stay long-term in lab but whose modal career advancement expectation is the incredibly difficult "land a tenure-track position" option, and graduate students who leave as soon as they've accrued enough scientific knowledge to be truly useful.

It's purely a cost-structure decision on a per-lab basis, but on a societal level it's costly to have most science being spearheaded by inexperienced graduate students (messing up experiments in ways more experienced researchers would not) and costly to train so many people in niche science fields that they then leave to join unrelated tech companies. Salary may play a part in this decision, but many scientists are passionate about their field of study and seems like the real deal-breaker is a lack of career options post "researcher-in-training" phase.

There are plenty of grad students that do their internships in companies rather than academia. (YMMV depending on the field of course...)

Not just 18 coauthors, but 18 co-first-authors! Implying everyone on the large team did equal contribution. Large teams of coauthors are not unheard of even in biology, but you rarely have more than 2-3 co-firsts at most.

This seems in character for DeepMind. Last time I saw their org chart it was perfectly flat with absolutely everyone reporting directly to the CEO.

I'm not sure when you saw their org chart, but it's definitely not true anymore.

Coming from a software generalist perspective, I found this phrase very familiar: "experienced hands, high coordination, and focused research objectives—are great for answering questions but not for asking them".

The best engineering teams are excellent at execution, but you need something more to choose the right problems.

Reply on news.ycombinator.com