Who would have thought that choosing an LSTM on iteration 1 for embedded C code would have been fraught with challenges.
653 karma · joined October 2, 2016
The reason I did this is because there just wasn't enough hours in the day, and my job was taking ~10 hours a day with commuting...etc. It was a risk, but the idea was that I would be able to transition much more quickly if I worked full-time towards it. I also had the financial savings to support myself for 6-9 months and was willing to get a part time job if necessary. Once it became clear that my job's only purpose was to pay the bills in the context of my goals, and I had enough to pay the bills for the near future, it was clear that quitting was the easiest way to free up a lot of time.
This turned out to be the best decision of my career, but YMMV. I doubled my salary in less than 2 years. It's also nice to be part of an industry that isn't so cost-sensitive. I also have a skill set that's in much higher demand, so you can live almost anywhere and there's a ton of companies that want/need it. With semiconductors, you're much more limited.
It's true that you're giving up some expertise and will start in a less senior position in a different field than if you stayed in semi. Sticking around because you have experience is classic Sunk Cost Fallacy. Think 5 years down the line. If you leave now, you'll have 5 years experience in ML. You'll definitely be giving something up if you leave, but there's huge opportunity cost if you don't leave.
In the context of a big company, I think it makes sense to have a specialized workforce. Why look for the one in a billion person that can publish top quality theoretic papers and then implement them on distributed gpus in an optimal way while also building simple Random Forest models for your business? I’d rather that person do more of the most valuable thing, and then hire someone else to do the rest.
In Amazon, it’s easy to move around, but not between job families. I think it’s a bad idea to join as a SWE and try to transfer because they want people that have done it before, and you’re unlikely to do that sort of work as an SWE. I think it’s better to get experience in the role you want at a less prestigious company. You’ll learn a ton. Pick the best company that will have you.
My personal turning point was when I did free work for a local startup in exchange for them letting me take the Data Scientist title. To a recruiter, it’s totally obvious to hire a data scientist for a data scientist role, and isn’t clear at all what physics has to do with it. Recruiters are the first step when you’re starting fresh, so make it easy for them.
I also somewhat disagree with many of the comments here that textbooks are better than tutorials. If you buy these 1000 page graduate level texts when the idea that you need to read them cover to cover, you’re likely going to give up and fail. Instead, buy the books and put them on the shelf, and then work through tutorials and examples. Then reference particular sections of the book that are relevant to your work to add depth.
Finally, I recommend against starting with deep learning. There’s a whole helluva lot to learn with basic techniques. Very few companies are actually using deep learning in production systems. Start with linear and tree based methods to learn all the stuff about how to frame the problem and build robust systems. Then you’ll have a deeper appreciation for DL.
A reasonable person could disagree and say that there’s so much domain specific stuff around the art of DL that it really behooves you to start there ASAP. I would counter that you’re unlikely to be considered for positions using DL unless you’re pursuing your PhD in it, or have proven yourself in industry. Since that isn’t your situation, I’d wait until you got your foot in the door somewhere and then pursue DL on the side. That’s what I did, and then you look like a hero to your boss. This strategy led to my first publication in the field and I’m now working on DL almost exclusively.
Edit: one more thing. Think carefully about the type of work you want to do. My advice is assuming you’d like to be a person that trains/deploys ML models to solve problems in industry. This is much different than an ML Engineer, who’s implementing algorithms in low level languages and squeezing out efficiency. Obviously that would require a much deeper understanding of SWE. And a totally different person is an academic researcher that’s developing theory or technique. It’ll be hard to do that without a PhD.
So be clear on exactly what you’re trying to do.
Or what if A always caused B, but the impact was slightly less than if B occurred without A? In that case, the sign on A might be negative, but its presence would actually tend to increase the probability of class=1, it's just that the positive impact has already been counted by variable B.
Maybe you try to avoid this situation by adding in an explicit interaction term of A*B, but then how do you interpret the impact of A since you now have more than one coefficient?
If you feel confident making assertive statements about what has been learned by looking at an equation fit on multi-correlated data, then your mathematical intuition is much stronger than mine!
Re: comment on showing "top" variables from a model, I agree this could have utility. But I would add that the devil's in the details, and there are multiple ways to calculate importance values, each of which has its own nuances and pros/cons.
For instance, how do you compare the importance of a categorical feature to a float feature? Do you one hot encode and then add their individual importances, take the average, or something else? Although sampling from the columns is meant to help deal with feature correlation, under what conditions is this effective and how do you know if your feature importances are safe? Moreover, how does this column sampling work in the context of one-hot-encoded categorical features?
This is all a way of saying that while you can devise methods for coming up with metrics, and then assign them handy titles like "Feature Importance", the reality is that these things are pretty nuanced and limited, and upper level management might be fooling themselves by thinking they're "interpreting a model" if they don't recognize the limitations and nuances involved. Or to put it another way, to say "I better understand this model because you gave me a feature importance list and a partial dependence plot," is a dangerous over-simplification.
For instance, take the go-to classification model: Logistic Regression. Many people think they can draw insight by looking at the coefficients on the variables. If it’s 2.0 for variable A and 1.0 for variable B, then A must move the needle twice as much.
But not so fast. B, for instance, might be correlated with A. In this case, the coefficients are also correlated and interpretability becomes much more nuanced. And this isn’t the exception, it’s the rule. If you have a lot of features, chances are many of them are correlated.
In addition, your variables likely operate at different scales, so you’ll have needed to normalize and scale everything, which makes another layer of abstraction between you and interpretation. This becomes even more complicated when you consider encoded categorical variables. Are you trying to interpret each category independently, or assess their importance as a group? Not obvious how to make these aggregations. The story only gets more complicated for e.g. Random Forests.
I think it’s best to accept that you can’t interpret these models very well in general. At least in the case of some models (like neural nets), they approximate a Bayesian posterior, which has some nice properties.
I want to say that I like their promotion process. It’s designed to be transparent and owned by the employee. I’d much rather have an ongoing conversation with my manager, using the doc as an excuse, that keeps the conversation on how close I am and exactly what I need to do to progress. This is far better than just working heads down and quietly wondering when you’ll be picked.
It’s also designed to persist in the event your manager leaves, which is a high likelihood event in the tech world on the scale of years. It would be really bad if your promotion totally depended on the support of one person who then leaves and you’re left at square one. At Amazon, boxes are ticked and they stay ticked if your manager leaves.
Yes, it takes time to maintain this doc. But it mostly is an excuse for you to have an open conversation with your manger and focus the conversation in a way that helps you grow and progress. It may not be perfect, but it’s designed with all the right intentions and in my experience has been very helpful and effective.
I also struggled with trust early on, but that’s not an amazon thing, that’s a human thing. People will tend to not take the new guy whose preaching big changes seriously, especially if he’s at a lower level. In that case, I found the best strategy was to find the high trust members, get their buy in first, and then present it as “I was talking with X and we think...”. That way, you get to borrow their trust level and he heard, and stengthen the relationship with them. Plus, they probably are better and smarter than you, so the conversation will likely lead to you learning something and being better off than just pushing your ideas on people that aren’t really listening.
And F the elitist douche that thinks I’m not smart enough to read Hacker News, as if tech people are just sooo brilliant.
For the first time or two, have a response ready, like “oh I just had to make progress on blah, did I miss anything important?” After awhile, if anyone asks, you can say that you don’t feel like you add enough value to justify the time spent. Your boss can always correct you and ask that you come again, but that at least gets him to evaluate if it’s important.
I have done something similar. To make this point clear to myself, I do not use any sort of tracking, so that I couldn’t figure out traffic numbers even if I wanted to. And I don’t advertise at all or put effort into growth, aside from dumping a url on LinkedIn.
One really nice benefit of writing about technical stuff and lessons learned is that it’s a great resource for future you. Just yesterday I pulled up my own article because I forgot how to do something I used to know really well when I was in the weeds. If you take the time to write it up, you’re free to release it from your brain.
- 1 year: about the same as now - 5 years: still corners social media, but not as front and center in public consciousness - 10 years: like google and apple. Not an innovation powerhouse, but still dominant. - 40+ years: like Sears?
It’s the post 10 years point that seems interesting. We basically know how the story is told for top companies.