So You Can't Find Yocto People
logikalsolutions.com
logikalsolutions.com
The work is challenging and rewarding, but also grueling and frustrating like little else. Embedded wages are miserable. Almost any Yocto developer could immediately improve their quality of life by picking up a Ruby on Rails book and then nearly doubling their salary in a month or two.
That's essentially what I did and I've never looked back.
As long as that holds true, hiring good folks will be challenging.
What's up with that?
They're not you VC, SASS, zero marginal cost software first company. They're hardware manufacturers with physical product margins. The embedded project I work on only ships on a few thousand units/year, and it's only a small component of the overall system.
I think within that group of companies, the comp for embedded vs "software" isn't significantly different. "Software" just has FANG/VC backed salary inflation making the averages look crazy.
I work on Yocto-based products and am responsible for the platform, but working on recipes and kernel mods is only 10% of my work, and by the end you're not touching it at all. If you're making kernel and rootfs changes in validation you have a serious problem.
I'm not sure what projects need someone that is hacking recipes 40 hours a week.
It is mainly handled by armies of contractors fixing stuff that constantly breaks from (uncaring) vendors that all gets chucked into a giant pot that has to work to ship. Vendors patching other vendors' recipes, etc...
Frankly, I think Yocto enables a more agile approach to embedded than not. I love it and I hate it.
We ate the cost to upgrade the hardware simply so we could run mainstream Debian and punt Yocto to the garbage can.
If I'm being more charitable: Yocto has been made irrelevant by the progress of technology. Either I have a system that is capable of running full blown Linux or I have a system that has no hope of running anything that isn't completely bespoke. The number of times I have enough resources to run "I Can't Believe It's Not Linux" but not actual Linux has converged toward zero.
30 - 10 years ago; Yocto (and its kin) served a very specific purpose. Building super tiny Linux distributions. If you have 4MiB worth of flash (storage) there's only so much you can cram in their.
Any "modern" linux distribution might still require more then that (excluding the likes of Linux from Scratch, as that falls in into the yocto category). Being CPU bound tends not to be the major issue, storage and RAM bound however is.
Personally, I've used debian in an embedded setting which can work great, but recently I've been using Alpine Linux for embedded and if you have a bit of storage (say 32 - 64MiB SPI NOR, or just some eMMC) you are good with Alpine as well.
Self-plug on not-yet-finished image builder around alpine: https://gitlab.com/esbs/builder
One other thing I think is greatly wrong with yocto, is the fact that it is super complex to do CI with it (because its one big thing) and build-times of 3.5 hours are ridiculous. Yes if I had to rebuild every single alpine package I'd probably end up with those build-times as well. But once a packages is built and vetted, just go with it (in your mirror if you will)
As for building the whole toolchain from scratch, sure, there might be exotic reasons to do that, but come on. Just stick your build environment in a container, and use that. Getting the toolchain from a "vendor" (alpine being one) is not a crime...
You're dealing with vendors, supply chains, hardware revisions, manufacturing pipelines... It's classic waterfall. Agile is a recipe for burnout here.
My favorite is always "but we defined 3 years ago that it'll do this-and-this fancy 3D thing on this screen, it's a requirement!" "well then maybe you should have selected hardware that can do fancy 3D. Let's talk about what we can change that still is good." "But its a requirement!".
Or on the hardware side, there is a "plan" that can't cope with the fact that suddenly there is a really good reason to do a hardware revision that nobody planned for. (e.g. because its 2021 and suddenly a bunch of parts just isnt available)
Of course totally freewheeling nothing-planned also causes equivalent problems, as does expectations that anything can change within 2 weeks. On the other hand on plenty projects I've worked on the "agile" requirements on the Yocto person are more among the lines of "we'd like this additional software added to the image please".