How I Transitioned from Physics Academia to the ML Industry | Hacker News Reader

How I Transitioned from Physics Academia to the ML Industry

dluo.me

52 pointsdatasciencer48 comments

I worry there are long-term implications to the exodus of intellectual capital from academia -> industry.

However, I don't blame anyone who makes this obvious choice:

A)Years of underpaid, thankless, exhausting work, for a... fractional probability of getting even a livable wage.

Contrast this with:

B)$techCo, where even a decent undergrad CV will get you a six-figure job, advancement opportunities, and maybe even a modicum of respect.

What sane(or not blindly idealistic) individual would willingly choose option A?

If there isn't some kind of concerted effort by higher-ed orgs to reduce the administrative and bureaucratic overhead of post-grad education, it's going to have long-lasting consequences.

Note that this also depends on field. For example, public health tends to have a much higher proportion of it's graduates be able to stay in the field. At the same time, physics (and applied math) is remarkable for just how mobile their graduates are, thanks to considerable overlap between the skills needed for modern physics research and several industries.

In contrast, biology PhD students, for example, don't have nearly as obvious an industry-based pathway.

Also, having taken Option A, it's not quite as clear as you lay it out to be.

I'm not in biology, but I imagine that doctorates in biology depending on their specialization could find work in biotech.

Industry was the pathway for doctorates in chemistry & the life sciences until 2008, when the jobs went away. With a physics degree you have a decent grounding in mathematics and programming, so you can go to work on Wall Street, in the life sciences, not so much.

Wall St., data science, etc. Way more pathways than the life sciences, which had biotech, at best.

I have a huge beef with academia and science and all as it exists today, but your description misses how thankless and exhausting $techCO jobs can also be, on top of working for companies like fb or google that are endlessly wrapped in intrigue, or working hard for what at the end of the day is a mundane purpose like increasing ad revenue which is fundamentally about making other people more wealthy than you.

A lot of the issues with academia depends on what specialization you choose, especially if you choose something closely related to theory. Particle physics is a great example of this and given that those who popularize science seem to hold up such pursuits as "pure", "fundamental", and thus important.

My favorite though is the all the masters and phds that end up teaching k-12.

I know a couple of people who went into a masters/phd programs with the intention of teaching in schools later after finishing. They wanted to learn some mental skills and they wanted to dedicate their lives to teaching. Nothing wrong with it.

I just dont think its economically efficient. It would seem like a huge waste of talent and potential.

I don't think anybody is obliged in any sense of the word to work a job that is the best return to society or themselves. Society can encourage people with certain talents to go for certain jobs, but people should be able to refuse such encouragements and instead do what makes them happy. Isn't the purpose of life to do what gives you satisfaction and makes you happy?

That’s funny, seems like reinvesting to me. Better long term returns.

Let me put it in worker placement board game terms.

If you have invested in making a worker scientist you probably want to place them on the location where you get a chance of getting a Tech upgrade. You likely dont want to place them on the location where you have a chance of upgrading another worker to a scientist worker when a much less specialized worker will do. And if you have too many tech upgrades then it makes no sense to try to upgrade normal workers to scientist workers.

Anyone that plays games actually has an intuitive sense of economic calculation.

I think there’s societal level value in having normal workers with a better appreciation for and understanding of science. Something like at a particular saturation level: +10% bonus to research lab output, 25% reduction of policy update costs with presence of overwhelming scientific evidence (stacks), bonuses to population health improvement with Medical Lifestyle Research unlocks, 25% reduction of knowledge rust, 5% reduction in knowledge related production costs, faster to upgrade to scientist, 2% chance that a normal worker will automatically upgrade to a scientist worker, etc. Obviously there are opportunity costs.

You're definitelly not from a Scandinavian country.

When I was last interviewing candidates for a data scientist position, I can't count how many physicists applied for the job when they didn't even understand what overfitting was.

There is a definite sense of "physics is hard, therefore everything else must be easy" groupthink, combined with Dunning-Kruger syndrome.

That's sad because overfitting predates ML by a long shot, and most experimental physicists of my generation (PhD 1990) knew what it was, even if we didn't use that term.

In my view it's not so much that "everything else must be easy," but that physicists are expected to take a multidisciplinary approach to our work. In the process of getting my thesis project to work, I had to teach myself electronics and programming, not to mention a variety of mechanical skills such as machining and vacuum plumbing.

Naturally, the job market situation encourages physicists to spread our wings beyond physics research, and to claim that we can do things. ;-)

Most physicists are pretty good at math. This actually gives us a head start in quantitative branches of other disciplines such as engineering. Today, when the engineers at my workplace run into a hard quantitative problem, they go looking for a physicist. If math is what makes a subject seem hard to most people, you might discover a few physicists trying to sneak in.

> (PhD 1990)

The title of the post is extremely misleading.

The author of this post graduated from an undergraduate program in 2018. By "transitioned from academia", the author means "graduated with a B.S. (after a few research-y summer internships) and got a job".

That first job is always an accomplishment, but in this cae was definitely not a "transition from academia" in the usual sense.

As a result, the post generally contains irrelevant advice for people actually transitioning (i.e., professors, post-docs, and late-stage graduate students).

That's your chance for a cheap hire? They're probably not going to get hired anywhere else and you can see in-interview if they are able to understand quickly! There's not much value for junior positions to "know concept X", so what you want to know is the derivative...

I'm really curious now. How were you testing their understanding of overfitting? Like what was the setup/questions?

That's really interesting because in my undergrad (which I hold a BS in Physics) overfitting and error were big topics in all labs. In fact, most physicists I know complain about overfitting in current ML models and how people are making these models and just blindly accepting results.

Sometimes I wish physicists or mathematicians or whoever will actually try these engineering or whatever problems. There may be certain aspects of them that your training will make you very able to handle, but these problems are actually very hard and you need a lot of their specific knowledge for them.

Was physicist. Did engineering problems. Didn’t have much trouble (especially compared to those with more specialized educations in the relevant fields).

To be fair, OP seems to have actually successfully transitioned, so they don't seem like the other physicists you seem to have a glut of.

If you are hiring physicist to do Machine Learning, then you are doing it all wrong. Physicist are excellent at mathematical modeling, especially those in their 50s+. Data science from my experience is vastly inferior to mathematical modeling. The OP that wrote the article seems young, and doesn't seem like they've seen much. In the world of quantitative finance, physicists and mathematicians reign supreme. These guys get paid millions in salary and bonuses, because they earn 10x that for the firms. Their entire jobs are to mathematically model things. There are very successful hedge funds that only hire physicist and mathematicians. The current data science and machine learning is child's play compared to the stuff these quant funds do. Physicist, especially ones that have mathematical maturity are incredible assets, they have a lethal skillset that isn't really there in silicon valley. The majority of data science posts I see on HN are mediocre at best. I think age and experience are highly under valued. The physicists and mathematicians that Draper, Raytheon, DARPA, Renaissance Tech, Black Rock or Bridge Water (I would include SpaceX and Tesla in this list) have are superior to any data scientist that you will find in Silicon Valley. Mathematical modeling is incredibly hard, seasoned physicists are very good at it (I can't emphasize how difficult and long this process can be, nor can I articulate the reward for doing things this way).

> There is a definite sense of "physics is hard, therefore everything else must be easy" groupthink, combined with Dunning-Kruger syndrome.

You are severely misinformed, and you severely lack a breadth of industry experience. Physicists (especially the experienced ones)have something called mathematical maturity. Having mathematical maturity is a skillset, you are able to make precise formulations of some phenomenon you want to understand things about, and prove with rigor that your ideas are correct, or that they converge to what you want. This is a lethal mind to have, it's far superior to machine learning or data science. You aren't relying on heuristics like "overfitting". These concepts aren't well defined, and are very subjective.

I can tell you this with 100% confidence, AlphaGo, Duplex, Siri etc. are all gimmicks, no different from Watson or Deep Blue. They are just applying statistical brute force to narrow domains. I'd love to see a machine learning system that consistently matches the profits that the successful quantitative funds do. I know no tech company can do this, they don't have a fundamental understanding of mathematical analysis, they don't hire the right people. The right people are at DARPA, Draper, Tesla, SpaceX, any organization working on the hard sciences. Go figure; pretty much all of the major important discoveries came from either military or laborites investigating the hard science.

Data science/ML, is great for simple data sets (there is a mathematical definition of this, not heuristic). For many industrial problems like what SAP, or McKinsey does it's perfect. The available cloud compute power and a basic Statistics + CS degree will disrupt these businesses. But do know this: they are severely limited. Machine learning will very quickly start showing its cracks.

I don't disagree per se but I think you're over-selling physicists' capabilities (former academic physicist here, PhD and all) and also using the latest hype-based machine learning cult in the Bay Area as a stand-in for all ML.

Mathematical modeling is essential to well-done data science. I won't let my team (I manage some machine learning engineers and data scientists) get away with what I call "toy modeling", that is parameter tuning and basic model implementation, and claim they're "doing data science." They need to provide a fairly rigorous analysis and basic mathematical models to justify their choices for particular algorithms or reaching for the "I give up" tool of so-called deep-learning. And the second my bosses tell me to just let them "do ML" is the second I resign. I won't be part of diluting the quality of my team's work

Out of curiosity, which industry to you work in?

I do agree that modeling is useful in every industry, but there are some industries where it matters more than others.

I think this level of worship of mathematical models is the exact reason that physicists shouldn’t be employed outside of physics. Most academic communities have over the past several decades adopted a Bayesian point of view for their modeling purposes. The idealized perception of a perfect mathematical model is so antiquated, I would consider it the equivalent of a flat earth theory in modern statistal physics. I say this as a prior mathematician who has since seen the limitations of this perspective.

Its funny that in the same comment you mention statistical physics and claim physicists don't understand statistical modelling. Physicists routinely use statistical modelling in astronomy, in condensed matter physics and in high energy physics. The emergence of the Bayesian point of view was originally spearheaded by a physicist Edward Jaynes and the torch has been carried forward by many other physicists over the decades.

I think Jeffreys, Wald and Savage had all started on advocating Bayesian statistics/probability before Jaynes.

I'd love to see a machine learning system that consistently matches the profits that the successful quantitative funds do.

How about Google Search?

As @sidlls rightly commented you are overselling "mathematical modeling" a bit, up to the point it becomes another black box, near to magical ability. I think it is essentially the capacity to abstract and find the relevant key features of what you want to model. And perhaps equally important is to cross check your initial model by seeing if it behaves reasonably in limit and corner cases, e.g. by putting certain parameters to zero or infinity.

>I can tell you this with 100% confidence, AlphaGo, Duplex, Siri etc. are all gimmicks

I think here you might have went slightly over the cliff. Couldn't you not with the same conviction claim those hedge funds only use toy models from statistical physics or wherever to sell you some Sokal hoax like vapor fund (of course they now the tricks that you or the public will pay when the bubble collapses again)?

I'm a bit split on the interesting tangent that @thoroughaway opened of distrusting mathematical models altogether. When I left physics somehow this field of "social physics" tried to establish itself, and I remember how the old lads in statistical physics viewed this field with great suspicion as not being proper physics. (One of the physicists of that field was consulted prior to the Love parade disaster in Duisburg https://en.wikipedia.org/wiki/Love_Parade_disaster - it's not about blaming him, but about the suspicion maybe also raised by thoroughaway that not everything, especially humans in a panic, should be modeled like Brownian motion or something)

It's worth knowing that computer vision originally worked with manually programmed feature identification using more-or-less standard signal processing techniques, but those techniques were wiped out by neural networks. In some domains ML performs far, far better than any manual model could, simply because in those cases first-principles understanding is impractical to obtain. In fact, most textbooks on optimization start out with some comment to the effect of, "such and such a technique is only good to use when you have no knowledge of your problem domain." That speaks for a lot of problem domains.

Tangential rant: I worked in a computer vision lab around the time ML/neural network was on the rise in the area. The chief scientist there was seriously annoyed by such papers because he got into the field to understand at a deep level what vision is. Neural networks get the job done, but after successfully implementing a neural network you haven't learned as much about how you can infer information about the world out there from images of it, as you do from first principles thinking.

This not 100% true. I am not advocating first principles. I am advocating mathematical modeling. Mathematically modeling vision seems pretty much impossible. I do think there are many theoretical parts of topology that might be useful like homotopy. It could be the case that the "next calculus" needs to be discovered. By that I mean a totally new branch of math that will change mathematics for the next several hundred years (the same way Newton and Leibniz did with the invention of calculus). It took a very long time for us to go from algebra to calculus, in the end calculus ended up being simple. The same may be true here.

Or another scenario, instead of modeling vision, model the brain, mathematically (this is the path I favor)

There are mathematicians trying to model the brain. Unfortunately the field is very new, around 50 years or so. Deep learning and neural networks (in their current form) are temporary. There is a tremendous amount of experimental and quantitative work that was done in the last 50 years that (I believe) will solve vision far better than deep learning.

My major question to you is, why do people use back propagation?

https://www.axios.com/artificial-intelligence-pioneer-says-w...

This is the smartest thing Hinton has said in his career. Back propagation is a pseudoscience, sort of like ether theory was.

Deep learning is a pseudo science. Why back propagation? Why sigmoid function? I know the intuitions behind why these decisions were made. It's all questionable, there is no rigor to it, nor is there experimental evidence for any of this. Pseudoscience.

What will replace back propagation and the sigmoid or the relu function?

The guys modeling the brain have a good idea. Right now is a wonderful time, there are institutes all over the world that have done tremendous amount of work in this domain. There's a wide amount of competing ideas and models. These ideas haven't trickled their way into computer science yet, and remain esoteric. Which one is the right one? I have my "team" already picked, and it will solve audio and vision, and basic problems in language. But the proof is in the pudding.

Will it give us AGI? (Hint: no it won't. Not in our life time at least, the math isn't there yet).

Can you expand more on what direction your "team" is researching towards?

As for over-fitting, I am not in my 50s, and I knew what over-fitting was from my undergrad (or highschool?). The idea is very commonplace in engineering. Numerous times, when encountering a "new term" in a different field, such as overfitting in machine learning, I have found myself thinking in overly complicated ways, because I never thought that what I considered so simple could be the central point of discussion. Perhaps the physicists who could not describe what over-fitting was, were in the same shoes. The other unfortunate scenario is that the interviewers expect certain responses that are just not the generalized things one knows of.

But regarding other matters, I hardly doubt that the sentiment you are suggesting is lost on all.

One point of view is that the self affirmation bias is easy to come these days. The material on basics of machine learning are beyond trivial, but the hype makes money. Even traditional engineering does that to some extent, at least that's what I saw in my education. Same goes for basic finance. I see hype attempts in non-"computer science" engineering research, which amounts to noise. Some writers are oblivious, some are doing it for money. The problem is these days every noise is amplified, and at first I was confusing noise for signal. Particularly at a period when I was practically doing solo research despite having colleagues.

All being said, and I could be too quick or too short on this, large scale computation may automate many aspects of our lives, and funding goes for automation whereas mathematical modeling does not automate things in and of itself.

As for new mathematics, I doubt serious researchers would disagree on looking for more.

I came across this today, you might recognize one of the names, haven't read it and doubt will find the time for a while, but perhaps it could be of interest to you. https://arxiv.org/abs/1608.08225

100% right. The only thing I can add to your comment is that “machine learning” is rapidly beginning to expand to include heavily quantitative model-based work.

What graduate level mathematics courses are relevant for these Wall Street jobs? Your comment has me interested in taking some of these courses.

If you are interested in how mathematical modeling works I recommend any course that covers the contents of this book: https://www.amazon.com/Mathematical-Financial-Derivatives-Sp...

Again, experience beats academia. These courses do a great job at introducing you to how mathematical modeling works. You need to go through an incredible grind (I can't stress this enough, it's a GRIND). I would recommend having a solid foundation in analysis. I can't stress how important this is, it's pretty much your bread and butter. I never took any computational finance courses, I wasn't a physicist, but a pure mathematician.

I will say this: from my experience, there is no such thing as a genius. Anyone can be a great mathematician, and gain this skillset. Often times, these so called geniuses from Ivy league schools are over inflated. You will get stuck, and get stuck for weeks to even months. It takes a very special mindset to push through it and those are the minds that are the most successful. Math is hard, no one is naturally gifted at it. You may have had people in your past that you thought were geniuses. I assure you they weren't. You saw them solve "easy" problems. They weren't anything special. Mindset always trumps innate abilities.

Main paragraph:

I grew increasingly disillusioned with academia. It seemed that many STEM undergraduates would rush into PhD programs out of sheer inertia, without exploration of other options, and spend five, six years of their prime youth down the rabbit hole of academic research. This resulted in frequent burnouts. Many PhD graduates turned, in the end, to industry after acquiring a distaste for academia in grad school. Those who continued in academia were met with fierce competition for limited amounts of tenure-track positions, and must go through several rounds of postdoctoral research stints before being accepted for a faculty position, if they were accepted at all.

The academic ponzi scheme thing is overplayed. I'm a Phd Research Scientist at a good school. I'm not faculty, but I have a lot of freedom and I get to do research that is fulfilling. Yes, it's a pyramid (aren't all org charts?) and I'm not at the top, but I'm happy.

...also, in a ponzi scheme there is no product. I aim to produce useful, generalizable knowledge. That's what the NIH is paying for. A valuable product.

The unspoken aspect of the pyramid scheme theory is that academia is great for the top one percent of Einsteins, and bearable for the top one percent of the most tenacious.

I'm interested to see what the future of the Research Scientist role is. It's basically a place for people who didn't do a post-doc and aren't bad-ass enough to not need one. And, one could argue, people who just don't want to play the faculty game. I hope we see that space grow in academia. I think we will.

Doing an undergraduate degree is not being in "physics academia". The author transitioned from graduation to getting a job. But good post regardless.

I initially read this as the MLM Industry!

Downvoted because you don’t like that I misread it? Or don’t like that I shared that I misread it and found it funny?

A lot of HNers don't have much of a sense of humor. (I thought it was funny.)

Reply on news.ycombinator.com