Hacker News Reader

Since this is getting a pretty high rank, maybe I should use it to pitch an interesting experiment to the DeepMind team (if they are reading this). Take random amino acid sequence, maybe bias pulling out of the bag of letters in a reasonable distribution, then pick sequences that AlphaFold2 thinks will collapse into a volume in any way smaller than their expected free chain radius; then start mutating this sequence, selecting for sequences that decrease the chain radius. Sort of a GA descent on chain radius.

My question are as follows:

1. Does AlphaFold ever converge on structures in this fashion at all? As in, is the heuristic able to identify partial folds, or, is foldability somehow encoded either through evolutionary history?

2. If it does, are the structures that it identifies truly compact (measurable through sedimentation centrifugation) and solvable using conventional techniques? I suspect if the first than definitely the second because of sample bias.

3. If it does identify compact structures, are they novel sequences or sequences homologous to existing structures? If the former, then AlphaFold is truly extrapolating. If the latter, then it's only interpolating.

It would be super cool if AlphaFold discovered a "new fold" (for those who are unfamiliar with the jargon, a fold is a family of homologous sequences that create defined structure, novelty is gauged by homology, not the end structure; you can have the same structure created by wildly different sequences). It would be even cooler if AlphaFold discovered a new motif (besides alpha helix, beta sheet, and the known turns/chains). But I think that is rather unlikely.

4. If you change your bag distribution by using a primitive aa set (delete histidines, tryptophans, enrich for leucines and serines) how do your search dynamics change?

Lots of cool questions! I'm very excited for code and training data to be made available (and in a perfect world the pre-trained model and hosted prediction as-a-service). Then folks can answer all of these questions themselves, without having to go through DeepMind.

Thank you for detailed and easy-to-understand explanations. This is why I come to HN.

As someone who used to work in adjacent field, I have a question that I hope you can help me with. DL usually requires a large volume of data, but I don't know whether there is a huge pile of experimentally determined protein structures for training. Did DeepMind find a clever method to get around data-size issue or there is really a lot of known protein structures? I skimmed one of their earlier papers and it seems training data was in tens of thousands. I am surprised that is enough for training DL structure prediction.

PDBs are not huge files, they are human-readable text format. Here is a rather large protein clocking in at kilobytes: https://files.rcsb.org/download/2RIK.pdb

Also I am not in ML research, but IIRC the spooky/counterintuitive nature of high-dimensional gradient-based search is that you can get better results from less data by increasing the number of parameters, as long as you have a sane set of regularization techniques (or have we even ditched those too?).

> as long as you have a sane set of regularization techniques (or have we even ditched those too?).

Maybe. In the context of natural language at least, Transformers require less and less data to reach the same result as you increase the number of parameters. No regularization needed. See Figure 2 in the paper Scaling Laws for Neural Language Models (2001.08361).

It's quite odd. Who knows if that will hold for other domains, like protein folding. It may very well be the case though, since AFAIK DeepMind's folding model used attention to reach these landmark results.

I remember reading somewhere that this was likely to be the consequence of how distance metrics work in high-dimensional manifolds.

In figure 2, those curves are all from one epoch each right? Which is to say the model has never seen the same data twice?

It's true you don't need regularization if you've never seen the same data twice, but that's a similar regularization to early stopping. You'd expect the larger number of parameters would make the training error drop faster with fewer tokens as well due to improved optimizability. But rather than "larger models need less data", I'd say the take away is more "larger models need fewer steps to optimize training error". None of the models get "good" until they've seen a number of tokens similar to the number of parameters.

Unless you've run many epochs on small data sets and seen the same results, in which case that's pretty weird/cool.

Figure 4 is for dataset size and seems to shows the same thing.

Yeah, figure 4 is more clear. This is the early stopped loss though, so the regularization is more explicit. If you trained the large models to completion on small data sets they would do much worse on test error due to overfitting.

It is interesting that larger models with regularization (early stopping) seems to work better than than training smaller models to convergence though.

Be careful on who you trust. The author of the article is a professor in structural biology (the field on protein structure) at Imperial College in London. The article is well written and the criticisms are fair and carefully stated. Dont come to HN to get scientific opinions. This site is pretty mediocre at that. There are exceptions of course.

"It is difficult to get a man to understand something, when his salary depends upon his not understanding it."

Talking about the professor or the commenter?

The AI that I currently help orchestrate is not in the sciences. I do plan to go back into biotech someday, but not in anything that requires this in a major way (the intended disruption is socioeconomic, not technological)

sure but also keep in mind that a professor in structural biology is not necessarily a consumer of structural biology (versus me, plus I have citations proving that I have consumed them) ;)

Pretty sure Prof. Stephen Curry could crush most of his critics in 1-on-1 basketball as well.

My understand, based only on having watched Yannic Kilcher's video, is that they had in the order of 10k training cases. So not a lot.

But they added a ton of highly domain specific features. And since it's essentially a natural phenomenon, I would expect the signal to be relatively strong.

I recommend the video, or any video from Yannic:

  https://youtu.be/B9PL__gVxLI

I think they said they used 140k structures as their training data.

"Jeez. Clickbait title, tons of fallacy, like goalpost moving."

Pretty much everything in the post is true, the title isn't clickbait, and I didn't see any fallacy. It's only "goalpost moving" if you confused the location of the goalposts in the first place. For the record: I have worked on protein folding, I did a PhD in a closely related area, and I work in computational structural biology professionally now.

The OP is saying that the general problem isn't solved, and provided (IMO factual, true) explanations why there's still a long way to go. Arguing that it's OK for some uses (which is what you're doing) is true, but not a rebuttal.

Maybe it's possible to use this method to come up with models "good enough" for rigid-body docking, or to get the overall idea of a fold. But it's also true that these structures aren't going to be good enough for any kind of drug discovery, which is what TFA says.

I did my PhD program at one the world's foremost structural biology program at the time (TSRI). I've solved loose structures of metastable peptides in NMR (by hand-- https://pubs.acs.org/doi/abs/10.1021/bi800828u, those assignments were painstakingly conducted by yours truly with assistance from Jane Dyson). I've done thousands of crystal drops with insulin to try to cocrystalize it (and failed; the x-ray structure I got did not have the cocrystal partner). I provided the critical insight for the prep enabled EM structure solved by the postdoc at the bench next to me: https://www.nature.com/articles/nature04339 . I'm pretty familiar with many aspects of the world of structural biology, but more importantly it's never been my primary - I've always been a consumer of the results first, a practitioner of it second.

It is my judgement that if you presented these results in 2003 anyone would have said "the protein folding problem has been solved". Nobody then expected "the solution to protein folding" to solve every single protein, and it's silly to now. If you're complaining about the 33% error rate, you don't know enough about ML to realize that that gap is going to get closed, and rapidly.

My argument is that it is "good enough for the uses that are in the domain of protein folding". There are a lot of things that are adjacent to protein folding (enzyme pocket analysis .e.g), but they never have been considered to protein folding per se.

So you're a structural biologist. Good for you? I'm not sure how it affects that substance of the argument you're making. I'm not trying to argue from authority here -- I'm pointing out that I have direct experience in this specific area, just so that you know I'm not some random commenter.

"It is my judgement that if you presented these results in 2003 anyone would have said "the protein folding problem has been solved"."

Your judgment is your own. At the very least, your use of the word "anyone" should give you pause.

"Nobody then expected "the solution to protein folding" to solve every single protein, and it's silly to now."

Not the argument I'm making.

"If you're complaining about the 33% error rate, you don't know enough about ML to realize that that gap is going to get closed, and rapidly."

Not the argument I'm making.

"My argument is that it is "good enough for the uses that are in the domain of protein folding"."

I have no idea what that means. Uses "in the domain of protein folding" are equivalent to uses "in the domain of structural biology". So obviously, it depends on the quality of the structure being generated.

"There are a lot of things that are adjacent to protein folding (enzyme pocket analysis .e.g), but they never have been considered to protein folding per se."

You seem to have some internal dialogue where there is a field called "protein folding", where there is an unambiguous threshold for success that has now been crossed. There is no such field, and there is no such threshold.

Are these structures good enough to do some kinds of things, like structural genomics or docking? I dunno...maybe? Are they good enough to do structure-based drug design? Not from what I've seen. TFA seems to be saying the same thing, and your comment was that TFA is just wildly wrong, which seemed...unfair, at best.

> FTA: For one thing, only two-thirds of DeepMind’s solutions were comparable to the experimentally determined structure of the protein.

This seems to be the core of the article's arguments. Is this ratio high enough to claim the problem is "solved"?

If your goal is drug discovery by checking out relevant protein structures, then you'll be lucky with useful structure information ~2/3 of the time (perhaps less, if you are depending on interactions of more than one protein).

If it was possible to accelerate drug discovery by looking at computed protein structures rather than protein structures established experimentally, it has now become plausible to do so. It has crossed a tipping point.

Since I'm not aware that there has been any drug discovery that has strongly depended on computational protein folding, this possible benefit remains to be seen.

[I worked in a lab which was heavily involved in CASP, the protein folding competition, but I was not myself involved]

Algorithms only get better. Did Google "solve" computer Go when AlphaGo beat Lee Sedol, when Master beat Ke Jie, or when AlphaZero learned Go from scratch?

Even though Google only won 4/5 games against Lee Sedol, that was the moment that sticks in peoples minds as when Google solved the problem of computer Go. What happened afterwards was a process of perfecting the algorithm. It seems likely the next few years will bring the same for AlphaFold.

But in this case AlphaFold is only marginally better than other algorithms. Since we are comparing computer algorithms (and not a human to an algorithm), it'd be like saying Usain Bolt "solved" the 100m dash when he set a new world record. Is it solved though, without knowing how fast those other algorithms are advancing?

have you seen the performance graph? it went from 40% (other teams) to 60% (DeepMind 2 years ago) to 92% now.

here is a nice case where alphafold did way better than others

https://twitter.com/TassosPerrakis/status/133353559400213299...

Tassos is also a professor in structural biology. Discovering structures the experimental way.

EDIT: on the other hand he also agrees that it's not the end of experimental structural biology

https://twitter.com/TassosPerrakis/status/133402467831107993...

No it’s not. AlphaFold is a huge advancement on all the other solutions. Solutions that were based on the already massive improvement that DeepMind brought in the previous edition.

Could you shed some light on the 66% part, too? As a layman and general sceptic of hype topics, this one is the biggest obstacle for me. How are we going to procede with models that are 33% wrong? In other engineering areas, uncertainty is usually covered by the modeler. But how would that work here?

who knows. You can't generalize 1 error out of 3 to 66%. Statistics of small numbers. Knowing how ML works, I'm surprised it didn't give any indication of low confidence.

Not to give DeepMind too much credit, but it's also entirely possible that "both are correct". By the nature of how targets are selected for CASP, year over year there is higher likelihood that "pathologically difficult" proteins are presented for the competition, for example - a protein that exhibits polymorphic structure where the technique (say Xtal vs NMR) biases the "solved" fold in a huge way. I believe CASP tries to weed out proteins that we know are polymorphic, but you can never be sure, and again, as time marches on, those are the proteins that fundamentally are harder to solve, so it's likely they will be enriched in the test pool.

An extreme example is insulin. The structure of insulin is has been solved for 60? I think years, but when it's in the environment of its receptor, it looks TOTALLY different (solved 5? years ago). Having said that, doubtful that DeepMind could ascertain that structure, since the environment is super-super different.

I think that the modestly high error rate is an indication that deepmind is mostly interpolating and has solved the broad protein folding heuristic. It probably will get better.

> You can't generalize 1 error out of 3 to 66%

Right but we also can't generalize from 2 out 3, and without knowing this figure it's really hard to say how useful this is, no?

> Knowing how ML works, I'm surprised it didn't give any indication of low confidence.

This is actually a hard problem in ML, for example in NLP, many people assume a high log prob score means a high confidence but it is not true at all.

> for example in NLP, many people assume a high log prob score means a high confidence but it is not true at all

Thank you for the clarification. I wasn't aware of this, as I'm most familiar with super-basic/old NLP techniques like BOW/RNN/LSTM/GRU techniques, where log prob scores seemed to me to be roughly correlated with result quality. I'm aware the landscape has changed recently with insights about high dimensional searches...

An interesting paper on this subject from just last week if you want to know a bit more:

http://phontron.com/paper/jiang20lmcalibration.pdf

There is no difference in the log prob between LSTM and transformers. It just depends on the number of updates you make (batch size, epochs), if you train it to overfit, then your log probs will be pushed closer to 1 or 0.

But after training you can recalibrate the temperature of the softmax on the test set and still get meaningful confidence scores (temperature calibration). Or you can use a variation of cross entropy called Focal Loss that will leave your logits un-squashed.

I’m not a specialist in the biology domain, but more generally, for some models 66% accuracy is amazing. For example, a model predicting the next word with 66% accuracy.

My machine learning professor would always remind us that 50% is analogous to random chance in a binary decision, but in a model where the prediction is not so black and white, 66% doesn't sound half bad!

It seems like this protein folding problem is something like the binary decision "guesses the winning lotto numbers" vs. "does not guess the winning lotto numbers".

If it got answer right 1 time in 100, that would be amazing and you'd be foolish not to use it!

>If it got answer right 1 time in 100, that would be amazing and you'd be foolish not to use it!

Except you have no way of knowing if the answer it gives you is one of the 66 right predictions vs one of the 33 wrong predictions. You could say it's likely correct, but not to a high enough degree of confidence that you could really trust it without verifying using the old established techniques.

You missed my point - there are not 33 but billions (an infinite number, really) of wrong protein structures. There is only one right structure, not 66.

Also important - verifying that a given model has a signature that matches the established techniques is far easier than using those techniques to generate the complete model from scratch.

The 33 vs 66 is in reference to the percentage chance that a prediction is correct. But if you have no way to tell if a given prediction is correct or not without doing the tests that you were trying to avoid in the first place then it's not really worthwhile except for perhaps some exploratory research.

I'm not really sure what your point is.

> But if you have no way to tell if a given prediction is correct or not without doing the tests that you were trying to avoid in the first place then it's not really worthwhile except for perhaps some exploratory research.

This isn't really how it works.

To quote the CASP competition organisers:

The organizers even worried DeepMind may have been cheating somehow. So Lupas set a special challenge: a membrane protein from a species of archaea, an ancient group of microbes. For 10 years, his research team tried every trick in the book to get an x-ray crystal structure of the protein. “We couldn’t solve it.”

But AlphaFold had no trouble. It returned a detailed image of a three-part protein with two long helical arms in the middle. The model enabled Lupas and his colleagues to make sense of their x-ray data; within half an hour, they had fit their experimental results to AlphaFold’s predicted structure. “It’s almost perfect,” Lupas says. “They could not possibly have cheated on this. I don’t know how they do it.”[1]

So you have experimental results, but still don't know how it folds. You aren't trying to avoid the all the experiments, just understand them.

[1] https://www.sciencemag.org/news/2020/11/game-has-changed-ai-...

I guess but wouldn't that only be useful if there were only two possiblities. If the next "thing" in a sequence is from thousands, millions, or quintillions of possibilities then 67% is hellatiously better than 1 in $really_huge_number

Exactly. I might not buy a satnav that was 66% accurate, but I would definitely buy a lottery number predictor with that accuracy. I don't know whether protein folding is closer to the former or the latter, but you can't make a blanket statement without considering the context.

It's not strictly 60% or 92%, it depends on the threshold of what is considered "good enough", 0.3Å or 1.6Å, which depends on what you want to use the prediction for.

When you set a threshold you improve precision at the detriment of recall. It's a tradeoff you can play with, but the score depends on it.

> Has protein folding been solved? Yes.

No. Look a the actual competition results.

> As a practitioner, what I want to be able to do is pull a sequence that I've retrieved from DNA, drop it into a computer, and get the structure out. Intuitive insight can FOLLOW from those results.

According to the competition results, you cannot do that using AlphaFold2.

"We have been stuck on this one problem – how do proteins fold up – for nearly 50 years. To see DeepMind produce a solution for this, having worked personally on this problem for so long and after so many stops and starts, wondering if we’d ever get there, is a very special moment. "

- Professor John Moult

Co-Founder and Chair of CASP, University of Maryland

Gee, the Co-Founder of the competition calls it a solution. I wonder if that might be because this is a solution to the problem as he A) Knows much more about the problem than you B) has actually personally seen the results and C) has far more context than you to base his decision to call it a solution.

He should also look a the actual competition results.

According to the competition organisers, you can:

So Lupas set a special challenge: a membrane protein from a species of archaea, an ancient group of microbes. For 10 years, his research team tried every trick in the book to get an x-ray crystal structure of the protein. “We couldn’t solve it.”

But AlphaFold had no trouble. It returned a detailed image of a three-part protein with two long helical arms in the middle. The model enabled Lupas and his colleagues to make sense of their x-ray data; within half an hour, they had fit their experimental results to AlphaFold’s predicted structure. “It’s almost perfect,” Lupas says. “They could not possibly have cheated on this. I don’t know how they do it.”

Sure, you want to validate thing experimentally, but that isn't different to imaging.

[1] https://www.sciencemag.org/news/2020/11/game-has-changed-ai-...

just for clarification, I don't know for certain, but it sounds like this is what they are talking about in science-journalism speak:

https://news.ycombinator.com/reply?id=25262458&goto=threads%...

So they bootstrapped the phases of the X-ray diffraction pattern with the phases from AlphaFold. This does mean, though, that we must be critical - it is possible that they are converging to nonsense that AlphaFold has generated. To be truly sure, you must have some sort of independent confirmation. Might still be "good enough" depending on what your application is.

I know what I'm about to say is off-topic, but:

>I have 15+ years of postgraduate chemistry/molecular bio/biophysics/biochemistry experience, then quit to go into tech, worked for AI hardware and now AI-driven software startups, mostly as backend

What made you quit biochemistry and enter tech and AI?

I have a strong anti-IP position: Didn't want to work for pharma, lost the battle to get into academia, didn't appreciate how much academia doesn't care about your science and how much you have to play politics to get in. I was always good at programming, figured it pays well (it does). Open source feels good, even with its warts.

so the answer is essentially YMMV depending on what project you're working on. This will accelerate certain aspects of structural biology and drug discovery, for others not so much. I don't see cryo-em, x-ray crystallographers, and NMR spectroscopists all losing their jobs anytime soon. At the end of the day you're going to have to validate and verify the claims of your simulation.

Could you use DeepFold to just get in the ballpark, then use that to bootstrap classic brute force methods (like Folding @ Home)?

probably depends on just how far off the "wrong" structures are. I presume the tally of wrong structures will get lower as the model gets refined.

If protein folding has been solved, can DeepMind be used to engineer new prions for study and/or weapon use?

> Probably deepmind will have difficulty solving the fold of amyloid fibrils

A weaponized prion might be the scariest thing I've ever heard of.

Eh, having studied prions (and related conditions, like diabetes, and alzheimers) I think it's just because they're "exotic". If I just called prion disease "contagious alzheimer's"... You'd probably be more scared of ebola, because the way that it kills you is more painful, and you're going to be isolated, so you will die alone except for by people in spacesuits, with the only major difference being that the prion diseases are currently incurable. But if an agent is being successfully weaponized, infrastructure for curing more conventional conditions would be stressed to the point where you would be likewise incurable (we were stressed in many places by COVID-19 and outside of the risk that you go critical, that is very much not a scary condition for a huge chunk even of people that get it).

I think the "completely incurable" aspect of it is the scary part.

I'm not AS concerned about something like Ebola, because while being horrible, it's horribleness is self limiting because people die (relatively quickly) before they can spread it too far.

On the other hand, weaponized vCJD could have a population die off en masse years after they're infected. Since it can be transmitted via blood transfusions, it'd essentially shut down blood donations (although if everyone had it, I guess that wouldn't matter). Also just the fact that prion diseases can cause psychological symptoms make the implications different from something like a classical infectious disease.

> On the other hand, weaponized vCJD could have a population die off en masse years after they're infected.

Maybe the infection could use genetic mechanisms so that the infected pass it on to their children. Maybe this already happened thousands of years ago. X-Files Theme.

Personally, I would definitely choose to die of Ebola rather than Fatal Insomnia. You get slowly driven insane by your inability to sleep. Truly one of the worst ways to go.

Why are you pessimist about AI for drug discovery in this innovation cycle?

I don't think ai will be able to dock small molecules with proteins.

You can bin all astounding ml results into one of two categories, convolutional forms or sequential forms. The problem space does not fit into either of those so cleanly.

Yes, you can turn a molecule into a graph, and there are graph ml techniques, but how that interacts with a protein also does not have a lot of source data (real or synthetic) nor is the interaction space obviously reducible into a differentiable form. Either would be sufficient for today's breed of ML to excel at this.

You want to increment your count of useful Google services? ;)

Take all my upvotes please, this is a wonderful response.