Congrats to the Kaggle team! One great thing about Kaggle was that the team listened and sought out feedback from users (even if they didn't always follow the feedback). I hope that doesn't change with the acquisition.
I really wish "data" journalism did a better job of conveying uncertainty. It is incredibly common to see comments like such-and-such increased 10%, without any comment on what the number of such-and-such actually was or if the 10% change was statistically significant. Likewise journalists often make claims that are not supported by the research. Scientists don't have the reach to describe their results to the public, news organizations need to do a better job of conveying the results accurately.
They gave me the standard whiteboard interviews on-site after that. This was for a data scientist position. I did not get offered the job (but I accepted a good job offer a couple of weeks later, so whatever).
The cost of a mishire is huge and obvious, but don't discount the cost of passing up on a good hire, which is much less visible. But every time a good candidate is passed on, the odds of a mishire go up. There isn't any way to determine with certainty whether a hire will work out or not before they take the job.
The thing is, the companies giving homework assignments still go through the regular onsite interview process. So it really is just an added burden of stress and time on the candidates. Arguably studying CS problems also helps job performance, though probably not as much as it could.
It depends how desperate the candidate is for a job. A coding assignment may have a hidden cost of filtering out many qualified candidates, if qualified candidates don't feel the need to apply to companies with these assignments. Anecdotally, these assignments seem like just another filter companies use, they don't replace any steps in the process, just one more hurdle to pass.
Often the assignments are more complex than described. I wouldn't be at all surprised if this was the case here. Also 2 hours is actually a really short amount of time, it can take 30 minutes or more just to understand what the person is looking for if the instructions are not clear.
To add on to this, a take-home assignment should be more than just another filter. The companies I interviewed for with take-home assignments didn't actually discuss the assignments at all in the in-person interview - I spent the time to prepare for and do the assignment, at least pretend it matters.
The lack of feedback from interviews is definitely one of the most frustrating parts. But remember that the success rate for onsite interviews is around 20-50% depending on company (companies don't publish this stat, but see for instance https://www.quora.com/How-many-people-who-take-the-Google-on...). Even if you had a success rate of around 50%, you would fail 7 onsite interviews around 1% of the time - which is unlikely, but unlikely things do happen. So my advice would be to keep trying - getting 7 onsite interviews is a good sign. Also try to figure out what areas of interviewing did not go well and improve on those areas.
I think there is a strong selection bias going into effect here. I know plenty of young people who are struggling to find good jobs and not optimistic about the future, and I live in a "liberal" city.
"On the fact that median male wage was higher in 1969 than it is today"
The obvious explanation for this is women entering the workforce, it is basic supply and demand. While this is good for the economy and society as a whole, clearly men now face more competition for jobs.
I agree that neural nets are state-of-the-art and do quite well on certain types of problems (NLP and vision, which are important problems). But a lot of data is structured (sales, churn, recommendations, etc), and it is so much easier to train an xgboost model than a neural net model. You need a very expensive computer or expensive cloud computing to train neural nets, and even then it is not easy. Ease of implementation is an important factor that gets overlooked in academia. And on non-NLP and non-image datasets, usually the single best Kaggle model is an xgboost model, which was probably developed in 1/10th the time it took to make a good neural net model. Xgboost has come a long way since it was first introduced, with early stopping being an example of a significant improvement.
I've always found it curious that Neural Networks get so much hype when xgboost (gradient boosted decision trees) is by far the most popular and accurate algorithm for most Kaggle competitions. While neural networks are better for image processing types of problems, there are a wide variety of machine learning problems where decision tree methods perform better and are much easier to implement.
The models of the universe with and without dark are fundamentally different, it is not inventing "invisible stuff to explain that the model is still right". The nature and composition of fundamental particles are an integral part of models of the universe. There is also a long track record of predicting the existence of particles and then later discovering these particles, for example the Higgs boson. Of course far more particles have been predicted and not (yet) discovered, such as theories of super-symmetry, but these predictions usually don't have the same type of consensus that the Higgs or dark matter has. While dark matter is invisible in the classical sense, it is observable by its gravitational effects. We have also observed what looks like clumps of dark matter, which modified theories of gravity have great difficulty explaining. With that said, until someone actually experimentally observes dark matter it is still a theory and could well be wrong. But this is true of all competing theories - there is no experimental evidence for theories other than dark matter.
The problem with this approach is that people pay much more money for limited gains they can exploit than for general gains that benefit everyone equally. Advancing academic knowledge provides gains distributed over a very large number of people, but the current system often does not efficiently allocate resources to this problem.
This definition of engineer is so broad as to be meaningless. There is no "fake world" with "fake world" problems, pretty much everyone is trying to create real world solutions to real world problems. Similarly most jobs are focused on the end goal. A data scientist job usually involves a mix of science and engineering.
There definitely does seem to be a bias against academia in industry. From my experience applying to jobs, most employers seem to value 1 year in industry more than 5 years in academia. The data science field is flooded with academic applicants right now, don't believe the rumors. But it can happen, I know of physics postdocs over 40 who transitioned to industry, and grad students do it fairly often.
It is interesting to note that the statistical evidence for global warming is relatively weak (at least compared to particle physics, where 3 to 5 sigma is expected for discoveries), https://wattsupwiththat.com/2016/04/07/no-statistically-sign... it is only because we have a known causal mechanism that we can say with confidence that man-made global warming is occurring. I liked that this article focused on causes of global temperature change, even if it did not present the data in a good manner.
The easiest way to make big changes is to make a lot of small changes. Try to focus on small improvements you can make and work towards them. Don't get frustrated if progress is slower than you expect.
In my opinion, the solution proposed in the article of: " choosing a women over a man when growing your team, just because" is counter-productive. These kind of suggestions cause people to question why a minority got promoted, if it was because of qualifications or because of reverse racism. Reverse racism is still racism, and it breeds resentment.
Most physicists who recently graduated who I know (who don't do a postdoc) have transitioned or are looking to transition into data science. I am in the looking to transition group, so my sample may be biased though.
It is just a rough guess based on my experience as a grad student. The average professor at my university had 1-2 grad students at a time, and with 6 years to finish a PhD and about 35 years where a professor takes on grad students it works out to roughly 10. I wouldn't be surprised if the number was actually as high as 20 though, 10 is probably a low estimate. Another way to look at is that roughly half of people go on to do a 1st postdoc, half of those go on to do a 2nd and 3rd or 4th postdoc, half of those get a tenure track position and around 3/4 of people with a tenure track position actually get tenure. The problem with a career in physics these days is that a postdoc position is a terrible deal. As a graduate student you at least have job security and can stay in the same place for a good amount of time, not so as a postdoc.
On average, a tenured physics professor will graduate a bit more than 10 PhD students over the course of their career. Physics as a field is not growing, so at most 10% of PhD students become professors. If you don't become a tenured professor there are very few jobs that aren't postdocs, and even if you wanted to you can't keep doing postdocs forever.
The hard part of coding professionally is getting a job. Practice is much more important than native ability for most intellectual activities until you get to the very peak of competitive performance.
To get a job at a company you have to be given the opportunity to interview, and to succeed at the interview. If you only evaluate performance at the job level, you don't evaluate performance of anybody who failed the interview. This is one of the fundamental challenges in hiring - you have a biased sample when trying to determine how effective the hiring process is at screening candidates.
What do you mean by large? Probably as your data is in csv format it is small enough that python or R are both good choices. Personally I'd recommend python as it has a large community using it so googling questions about python libraries such as pandas usually provides good answers.
From someone who is looking to transition to data science, the field is terrible if you haven't had an industry job before. I am ranked in the top 150 on Kaggle and can't even get phone interviews without someone in my network recommending me for a position.
When you say "clean data", what exactly do you mean? I've often seen this claim that cleaning data takes a lot of time, but it seems like an ill-defined term.