My sibling comments miss something important: people actually do something for those 5+ years. They don't just sit on their hands. They're publishing papers, collaborating with respected researchers, building a personal network/reputation, developing new tools and techniques, writing peer reviews, contributing to grant proposals, giving talks, teaching, etc.
I.e., they are actually doing the exact job that they are now being hired to do.
> Is it the ability to conduct research - methods, process, analysis, discipline? Or is it the fact that they have knowledge in that particular field?
When you apply to R&D labs, you are competing against people who spent at least five years:
* building research infrastructure (which includes, but is not limited to, knowledge about "methods, process, analysis, discipline"),
* a concrete track record of producing publishable research,
* a professional network and some amount of respect for you previous research output, and
* experience designing a multi-year research project (from writing/helping write a convincing proposal all the way through publishing, evangelizing, and managing relationships with people who want to build on your work).
Do you have 5+ first author publications in good conferences? Have you contributed to R&D software? Do you have multiple influential collaborators who have a high opinion of you and are likely to help popularize/contribute to/build upon your research goals? Have you written and followed through on grant proposals?
If "yes" to all those things, you should apply for these research positions! Even without a phd, you are a good candidate.
If all you have is a six month MOOC, or some R&D software development but no experience with the full research process, then sorry, but there are people with who have already demonstrated they can successfully design and execute on research agendas. You might still be a good candidate, but preferring strictly more experienced and better qualified candidates who have demonstrated, empirically, that they can perform the full job, is not an "arbitrary blockade".
> Every time I speak to someone with a PhD at work (I work in top 10 tech companies in US), in 100% of the cases in my experience do not engage in anything related to their thesis.
"people I have worked with" is a huge selection bias.
IME, in CS this is almost always either a) not true, or b) a conscious choice on the part of the phd holder (and is more of a pivot than a "totally unrelated thing"; you just might not see that there's a deep connection between the thing that the person was doing before and what they're doing now).