Where are the low level system engineers? | Hacker News Reader

Where are the low level system engineers?

twitter.com

109 pointspyb149 comments

Here :)

I maintain a C++ service running in production since 2015. I released Windows and Mac kernel drivers in 2012, both of which are still used in commercial products. My NAS runs a patched kernel because the stock dm-cache didn't offload burst writes onto SSD. I once built a virtual filesystem driver to run inside my PS2 so that I could exfiltrate in-game debug data over LAN. My most recent low-level project was PCB+Firmware for a digital camera.

But:

Ruby on Rails just has a much better bang for buck than C++ and customers know it. Powerful hardware is dirt-cheap these days, so nobody wants to pay for the extra +50% you get from C++ over JavaScript.

And if you want to be paid well, call your multi-threaded C++ calculations something like "CPU-accelerated AI deployment optimization" ;)

And lastly: Most embedded C/C++ jobs went to China! We used to design hardware and firmware in the west. But nowadays, you just book a Shenzhen shop to design, code, and produce your product. That's why nowadays almost every device has firmware with Chinese debug strings inside ^_^

We build our own cameras and write our own drivers here in Amsterdam, and sometimes get requests from other companies here that rather pay more than trying to explain their projects to Chinese manufacturers. Some industrial cameras we buy from Basler and the guy who builds them lives right around the block in a village close to Amsterdam. So maybe it's coming back around.

As an engineer I have a really strong opinion about only finance being left in Europe and all the engineering being outsourced to cheaper countries.

I personally think in tech the ladder to the top is made so you have to switch to management at some point in order to promote any further. I see managers and people that actually want to volunteer to become a manager instead of an engineer as the root of this problem. At my company we organize the hierarchy more like a hospital or a law firm, with the best engineers having the highest positions, just like partners at a law firm.

I work at a product design firm here in the US, we're also organized like this. Engineers that can deliver end up getting more clients, so it does end up somewhat like a law firm (their hourly rates are still higher, which IMHO is crazy..).

We also make cameras, I've become the MIPI expert.

Our bread and butter includes mechanical design, PCB design and firmware / FPGA design. This maybe sounds like basic stuff, but includes things like RF and COMSOL / ANSYS physical and mechanical modeling and simulation.

A startup company in India hired us: they had plenty of software developers, but wanted experts to de-risk the hardware and low level firmware.

https://www.nklabs.com/

Our experience with Basler was pretty bad. We got a few daA1280-54uc and all of them had insane noise levels. Their German support never took us seriously and just argued that our 10% noise level was normal.

Basler Vision Beijing later confirmed that it's a firmware issue and indeed switching from RGB to YUV reduced the noise to less than 1%, but it left a very sour taste that the support team in their supposed headquarter didn't recognize a noise issue when we sent them test pictures, like this: https://www.dropbox.com/s/ahht53ht6moa40g/Image__2021-04-07_...

Also, the marketing photos of dart Cameras had a "Made in Germany" sticker on the back. Ours shipped from the US (to Germany) and didn't have a "Made in Germany" sticker.

In the end, our client then decided to go with a custom camera. They wanted full control over the FPGA firmware to avoid future issues like this.

You'd get a lot more than 50% from C++ over Javascript (and definitely way more than Ruby)

But probably not compared to Rust, Go, Kotlin, or Java. Few people are going to opt for C++ for an internet-facing web service given the multitude of better options.

> And lastly: Most embedded C/C++ jobs went to China! We used to design hardware and firmware in the west. But nowadays, you just book a Shenzhen shop to design, code, and produce your product.

That's a key part of my depressing realisation in the last five years or so (my realising, not putting a timeline on the happening, that's rooted in industrialisation basically I suppose) that practically everything is just marketing. Some tiny amount if product and then just marketing.

It's 'drop shipped' 'import' 'Chinese tat' with its bizarre fonts & 'brands' and unintentionally humorous instructions that clued me on to it, but now it's everywhere. Hand soap. As fancy as you like, essence of bullshit and aroma du markèting, it's all made in a big stainless steel vat.

Greengrocers have to be the one avenue you can still find authentic no-nonsense products. Even then you basically have to go 'ethnic'. A British mainstream supermarket will charge a couple of quid for a pathetically sized, masses of food miles (and time) short-dated pack of some herb; the grocerywali opposite will gladly sell you two big bunches, far fresher, for less. I don't really get that? Even in the British supermarket they'll sell you a 10g glass jar marketed to white people for £2 while the same bloody thing is £1/100g (and in that sort of quantity, i.e. spend less for more) on the other side or in the next aisle in a plastic bag sold as 'world foods'.

(I've probably digressed enough...)

You're sadly not wrong. I've seen the financials of selling bikinis.

$2.5 product + $9 shipping + $20 in facebook ads => sold for $75

As for supermarkets, the main difference is probably what they charge you for placement. If you (the manufacturer) want your product inside a supermarket, you typically have to rent the shelf space from them. And depending on where inside the shop that shelf space is located, it'll be priced vastly differently. That is also one of the reasons why you typically only see high-throughput high-margin products close to the cashier, e.g. sugar candy for kids.

But I feel like on the electronics side, it's not that obvious. When you need access to electronic components, being close to the markets in Shenzhen is a huge advantage. Shenzhen engineers can just walk over, chat to the manufacturer, and buy some sample products. Western engineers need to read the spec sheets and then wait 2 weeks for the samples to make it through shipping and import clearance. Also, some top-notch components only have descriptions in Chinese, because the manufacturer knows that 99% of their market is selling to other Chinese companies, so they don't bother producing an English datasheet.

The result is that Shenzhen shops are not only cheap, but also better quality.

> Ruby on Rails just has a much better bang for buck than C++ and customers know it. Powerful hardware is dirt-cheap these days, so nobody wants to pay for the extra +50% you get from C++ over JavaScript.

Depends. Use cases with highly constrained hardware SWaP where inches, ounces, and watts matter, the expense to optimize may well be worth it.

I’m right here, bombing Leetcode interviews because they ask me to write a program that sorts the columns in a CSV file in 15 minutes and I only know C.

Maybe I'm missing something but unless the interviewer expects a 100% correct csv parser that handles all possible corner cases this is basically

- read file

- start at beginning; if you encounter separator, add new entry to an array. if you encounter newline, save entire current array as a new line to an array of lines.

- do lexicographical sort of arrays (can just use qsort for this).

- if the test q actually wants you to parse out numbers and shit like that, nothing about the structure changes, you just have to record some extra info when parsing values and then use a different comparison depending on the type in the function you pass to qsort

I dare say someone programming algorithms in a kernel or writing code for custom file systems should be able to do this. OK 15 min may be tight if you're nervous but you should get there.

You forgot quotes. Unfortunately we're moving forward with other candidates at this time. Good luck in your job search!

It gets very nasty at this point especially because I’ve used popular languages who’s CSV parsing libraries were unable to handle some weird quotes properly.

What languages? No option to configure any character as quote character or delimiter?

Ruby. Python. Several JavaScript NPM packages. Part of the problem here is the standards around CSVs aren’t consistent, nor are they well respected.

Things get even weirder when you throw non-Latin characters in the mix. May such parsers predate widespread use of UTF-8.

RFC 4180!

Python I believe was the issue in my case.

State machine is all you need.

I don’t think you realize how much harder this becomes with C. You probably need to can’t just add strings to an array you’ll need to allocate space for all the arrays (presumably by doing a first pass to determine CSV dimensions) and all the substrings (since they’re Cstrings).

Personally I’d do it by storing indexes into the CSV contents instead of using null terminated C strings.

Doing all of this in just 15 minutes is going to be hard.

By the way, your solution as described is wrong, since it sorts each row independently. What you’d need to do is store all the data in a transposed format i.e an array of pointers to column arrays, then sort the array by giving qsort a comparison function that looks at the first entry in each column.

Likewise. I've known C since the 90s, C++ since the 00s (with a refresher for C++11), and recently picked up Rust—have already written a few thousand lines, it's a great language. But I do DevOps because the pay is better and the interviews are practical.

Holy moly, I thought I was the only one!

"Want me to write a specialised API with custom encapsulation transmitted over UDP?"

"Nah fam, we want to watch you code in our super-duper-wiz-bangy-exclusive IDE that you've never used before. Oh, and you have an hour. Oh, and it's going to be super-specific tasks that only we know how to do."

"So, no on the encapsulation, eh?"

<click>

I'd be tempted to just use sort(1) in that situation and say "what? it's in C" if questioned.

I'd actually hire you for that. You want to not waste time reinventing the wheel? Cool!

But then, when I interview, I don't ask leetcode questions in the first place...

This is missing the point of such questions. They aren’t aimed at asking “how would you solve this easy task?”. What they’re checking is “since we can’t have you demonstrate how you would solve a complex task in the short time allotted to an interview, let’s see how you can solve a smaller pretend task.”

In practice for this particular question they're asking if you've memorized sort algorithms. I have not (or I've forgotten since) and I doubt even the brightest minds of the industry that did invent them did so within an hour long job interview.

Was that (implement sort) the main point of the question? Especially given the 15 minute time window this sounds more like some kind of fizzbuzz-style question (here's a basic idea, can you translate it to code somehow in some language?). Usually when interviewers give me that kind of thing they're literally fine with bubble-sort (which I'd hope anyone could code on the fly), or even just assuming some appropriately shaped sort function exists and writing the rest of the code around it.

No that's not what the question is asking. You should be using the standard library sort function (or making one up if you can't remember the specific API).

Brother !

This must be common, I just bombed a leetcode (tm) interview where I was asked to write code to share a method between two structs using traits in Rust without duplicating code. I couldn't remember a time I'd ever seen it done with generics so I tried to find it in the documentation but nothing. I looked up the solution after and it is not possible without writing a macro which is metaprogramming and seems like a poor way to assess how I'd perform writing Rust on a daily basis when I only had ~20 minutes to do it and was transparent that I am still learning the more advanced concepts. There were other red flags I ignored in the interview process, I look at it as a bullet dodged.

Can't you just provide a default implementation of the method on the trait itself? Unless there are more requirements I'm not parsing from what you wrote, that seems to do exactly what they wanted.

Yeah, I'm not sure I fully understand as well. The only thing I can think of is that the trait was supposed to be from outside the crate (i.e. not modifyable), but then then, I'm not sure why writing a function that was generic over T: ThatTrait that took an instance as a parameter (or even `impl ThatTrait`) wouldn't work

That is not really a leetcode-style interview question. Leetcode focuses on DSA questions not language knowledge.

What they’re describing is leetcode style problems that require the coder to use an abstract high level language that makes coding easy.

What you’re describing is the opposite - some kind of esoteric question related to a specific language.

The real answer here is take whatever the R data.table package is doing and copy that. It's even written in C. It's by far the best csv reading package I've ever used and will sort as it reads, sometimes extremely fast since it can pretty intelligently guess if a radix sort is possible. The only downside is I believe it can only handle csvs that fit in memory, but memory is getting pretty damn big these days. I used it as inspiration a long while back when I needed to write a library to read in ARPA-formatted n-gram language models, which is a fairly obscure file format I could not find any existing libraries for.

Of course, data.table has existed for over a decade and you're not going to come up with something comparable in 15 minutes, but that's why you'd want to just copy it and not come up with your own solution in the first place.

Right here. I was CTO of a hardware + software startup. Built a home automation device, couldn't get scale, sold to FAANG.

Now I manage rEqUireMenTs, syNcHroniZaTiOn oF cRoSs fUnctIONal teams, building a myriad of control planes over open source software, make 5x more, and am 10x less happy. Bills to pay.

My condolences, that sounds horrific.

They’re at FAANGs. I can say from experience that Amazon and Meta employ tons of systems software engineers working on kernels, languages, low-level Unix software, etc.

The specific things he's mentioning are domain specific topics to (it sounds like) OS development that most low-level capable programmers could pick up no problem.

I'm talking about programmers working on any kind of application in C/C++ (or other systems languages) with nontrivial performance and memory requirements. It's not like there are a ton of those either, but they're there.

If for some reason some day some market popped up for OS devs that paid better than the other sub-fields those low level programmers are currently working in, nobody need be concerned about supply of programmers who can do the job.

> I'm talking about programmers working on any kind of application in C/C++ (or other systems languages) with nontrivial performance and memory requirements. It's not like there are a ton of those either, but they're there.

I tried finding a job somewhere around this realm. Half the companies were using Go and none of them wanted to talk to me.

Companies don't want someone who is capable of picking up x no problem. They only want someone who already has a job doing x.

Furthermore, a lot of these companies seem to only want somebody be with 10-20 years of experience already doing x.

FAANGs are generally looking for the for the kind of person who is capable of picking up x no problem. That’s why the whiteboard interview so many on here complain about is the way it is.

people complain because those whiteboard / algo heavy interviews don't actually test for that ability, and it's trivially obvious.

just the fact that people practice them before FAANG interviews and their correctness/success rate and speed at completing those tests suddenly gets better by an order or magnitude or more should tell you how little those tests actually filter for any more fundamental abilities.

a proper test for the abilities you mentioned should not only be doable no problem by any working programmer (especially senior and above) walking into that test naked and unprepared, in fact preparing for the test in any reasonable manner before the interview should give no advantage at all.

Thank you for this argument.

Complaining about any test obviously just sounds like nothing but proof the test did it's job, to those that want to believe it.

But what you just said stands on it's own and doesn't matter that the probably-lame applicant said it.

If preparing for the test works, then that fact itself exposes the test is not testing what you think, or anything very useful.

I'm here brother! Low level for life!!! C/C++/weird bytecodes interpreted by FW. They don't give us the love like they used to, but can't build/optimize/secure/debug systems without us. Employers are leaving a lot of performance on the table and they don't even know why. I feel bad that we are not training the next generation, but at least I will be gainfully employed until I die.

> Employers are leaving a lot of performance on the table and they don't even know > why.

This, 100 times. All performance discussions at my employer are at the upper layers of the stack ("hey look I'm using char vs. string in this C# method, 0.15% improvement!"), while nobody is looking at the lower layers where the real improvements can be made and benefit all platforms.

> Code schools don't seem to be teaching this stuff and some universities have moved up the stack.

ABET accredited CS degree programs still require students take courses including "Exposure to computer architecture and organization, information management, networking and communication, operating systems, and parallel and distributed computing."

I'm not aware of any flagship state universities that don't have an ABET degree option, but maybe I just have massive selection bias.

[1]: https://www.abet.org/accreditation/accreditation-criteria/cr...

UNC Chapel Hill doesn't, probably because we're not an engineering school (actually, we do show up on that page for the BME program but that is a joint degree with NC State, which is).

I suspect there are a few other universities in that position that have reputable CS programs but otherwise no engineering--although CS itself can be said to not be engineering, I would argue that for the 90% of CS majors wanting to get a SWE job, what they are effectively seeking is an engineering degree not a science one. Still, there isn't any expectation on the part of employees that a degree is ABET accredited, as there is for engineering.

C, basic digital logic, and MIPS assembler are part of the required intro sequence here.

Do students or employers care about ABET? I don’t think they do.

Maybe the kind of person who complains on twitter we dont educate enough low level engineers does?

IME : they had to move up the stack, in order to access better employment opportunities.

Seconded. I look around, at this week's "Who's Hiring" thread, for example, and the systems/firmware/embedded jobs are slim pickings. Filter out the super-senior positions and I can count them on one hand. How's a guy supposed to gain the experience to be a senior embedded dev when nobody's hiring junior embedded devs? I'm sticking at this, hoping for a needle in the haystack, because I like it, but most people in our position would just pick up some JavaScript and Leetcode and double their salary.

If you also take into consideration the years it takes to go from junior to senior/experienced in the systems area, salary becomes a much bigger deciding factor. It’s not just 20k a year less for one year, its 20/30/40k less for as long as you’re mastering your area when you could spend less time mastering something else for more money.

Part of the issue is that places like HN cater to smaller startups and places eager to seem 'hip'. Companies that employ a lot of embedded engineers are invariably either FAANG type companies that don't need to advertise that way, or hardware companies so institutionalized that they treat it like any other administrative job and only post on LinkedIn/indeed. Occasionally you'll get a smaller startup/unicorn that's trying to complete with SV tech giants, but that's uncommon.

I'm trying to implement these sorts of changes (and many others) right now at my own place, but it's quite frustrating from the other side too.

Hacker News is just the wrong place to look. This is essentially a web dev site. Just use linked in or zip recruiter or whatever. There are tons of positions for embedded now, same trouble hiring as everyone else, and often they’re open to anyone with software experience, you don’t need to know the difference between Vdd and Vcc or how to use an oscilloscope, though obviously employers would prefer you did. (Well, forget the first one, Vdd vs Vcc is essentially trivia)

I am looking on those other, more traditional sites as well, and junior-level embedded jobs (without moving to California or China) are still far less common.

As someone who has recently left embedded engineering to pursue other avenues, I got tired of:

1. Being paid much less than my similarly leveled colleagues

2. Needing to work with engineers well outside my timezone

3. Being hounded by recruiters to do bluetooth comms and nothing but bluetooth comms

4. Having to be very careful to not put myself in a position where I need to fly to China to deal with manufacturing issues on a moments notice

5. Lack of remote work opportunites for obvious reasons

I do enjoy working on low level code but not enough to deal with the industry in its current state.

I don’t know about the other FAANGs but if you get into Meta you can then select a systems / embedded team during boot camp.

What area are you in? I'm in Detroit, a major manufacturing center, and there's always demand for embedded devs on job boards. Maybe you need to relocate?

I'm in webdev to access any employment opportunities. Moving to kernel/embedded would be fun but it almost feels like jumping to another industry. I have no connections and no idea how to find a good company (e.g. what is the Joel Test for hardware companies?) I have gone on a few job sites and sprayed/prayed my resume, which so far has gotten zero interest from anyone. OTOH I'm lucky to have landed where I did so it might be for the best that I stay in webdev.

Koopman's Better Embedded Software should give you some good things to ask about.

Not even remotely close. As OP posted: there are too few low-level people left. This skillset commands much better pay than "upstack"

How much does it pay? From what I’ve seen, market pay for systems devs is substantially lower than for those up the stack, outside of FAANGs and HFT outfits.

I've got a friend who is a long-time embedded C developer, about 20 years of experience. Really knows his stuff. I just hired him to write python and he got a massive pay rise to come over, because all the embedded shops in our area pay shit.

I'd love to get paid doing low-level coding (or at least try my hand at that). Sadly, I am very passionate about being able to afford food, so DevOps it is (not that I'm complaining).

They also won't give a second glance to those who would like to move down the stack, even in markets where these sorts of skills are practically non-existent.

In 30 years, I have moved all the way from the bottom of the stack doing assembly to the top, where I now do deployment pipelines and other DevOps related tasks.

I'm a specialist in low-level telco code, lots of which is written in C/Perl.

Recruiters are breaking down my door, asking me if I code in Ruby (Rails).

No.

Similar story here. Have lots of experience with low-level scientific compute, ML, and network code in specialized environments like avionics and robotics. I have used Rust in production so it is listed on my skills, but mostly C/C++ and of course Python for prototyping.

Recruiters saw Rust and I got blockchain spam. Nope.

I've moved on to my own thing anyway, but I've still had to put an explicit "no crypto/blockchain" everywhere they might look.

But now all the bots will see the words crypto and blockchain in your profile (or wherever) and you'll get even more spam.

maybe there's a way to include those words like "bl0ckchain" and "crypt0" or even similar looking cyrillic letters eg:

сrʏpto ʙloϲkchain

from https://www.irongeek.com/homoglyph-attack-generator.php

Not so many employers. Very few places/teams in this world care about cache coherency or memory ordering in the go-lang runtime on ARMv9 CPUs. You can count them using one hand fingers. The entry barrier is also too high; you need too many years of serious CS education, usually a PhD. And then find a place at Google or Meta Kernel/Compiler teams, which is obviously super competitive.

I've been employed for a variable mix of system design and low level performance work my whole career... except for Google. There are definitely more than five firms that would pay reasonably well for my expertise, and my PhD was in mathematical optimisation.

> There are definitely more than five firms that would pay reasonably well for my expertise

There are definitely more than five firms that would pay reasonably well for an ex-Googler with a "whole career" and a PhD, yes. The barrier to entry is too high for the rest of us.

Definitely not just one hand (ex: HFT), but perhaps you can count them using 3-4 hands. The nature of most compiler work is that you're pushing the boundary of what's been done, which in academia is almost exclusively done by the top 1% researchers. And HFT is obviously winner take all, which further favors top performers.

A PhD is definitely not required to work on this kind of stuff.

But maybe it is to get hired ?

Definitely not.

> The entry barrier is also too high; you need too many years of serious CS education, usually a PhD.

School taught me none of that(the opposite with more of a focus on big-o and ignore the constants), all the good systems engineers I know either learned it on the job or were self taught.

It certainly requires a lot more skill and patience than web development. I don't think you need an graduate degree, but i think in fields that are time consuming and complex the graduate study experience is generally helpful. Ofc there are plenty of people without the degrees who have the skill and patience.

> Very few places/teams in this world care about cache coherency or memory ordering

A lot of HFT shops definitely care about those things.

Google and Meta certainly aren't the only people employing compiler engineers. Even if other compiler users outsource everything to consulting companies like AdaCore, well, that's where they are.

Around, working on cool things at companies that can afford this increasingly rare skillset. Google employs a lot, Apple does too, Facebook does as well. Most startups cannot afford us, and just keep trying to sell us on "impact" and "get in early on equity".

I did a lot of C, network wizardry and kernel hacking at uni.

I couldn't get a job in that though, so I moved further up the stack and now do backend instead.

Somewhat here. But to be honest, it's a tricky place to be.

As an individual who is very proficient in this areas, I get rarely any reachouts about those skills. 90% of LinkedIn recuiter messages inquire about generic high level programming projects. Then there's 8% crypto related things, and a very tiny amount of other systems level engineering roles.

On the other hand trying to hire someone for such a role is super hard. Most engineers don't deal with low level details and systems programming languages anymore.

Compensation wise the domain does not be better than others. While one assumes it might be more specialized, that seems by far offset by the amount of lower opportunities and thereby less chances to negotiate compensation than in other software engineering parts.

I think another big reason for lower compensation is because a lot of companies working at a low level tend to be companies making physical electronic products. Profits for physical profits don't scale as exponentially as they do with software only products, so companies don't have endless amounts of cash to throw around at engineers.

If you're after good compensation and want to work at a low level, then go for companies that sell both hardware and software. FAANGs are an obvious choice but B2B companies (usually IoT-ish) that have have both a software and hardware subscription component are another option worth looking at.

> Compensation wise the domain does not be better than others.

Yep. I moved into web development because it paid a lot more and carried a lot less stress. The 7 years I spent working on embedded systems paid off in a major way, too, because I showed up equipped to directly address performance problems. I'm still in web development and I'll probably never leave.

Well I’m still here. But when I learned low level systems programming, it was just called ‘programming’

I'm not on twitter...

Working on consumer electronics, VoIP phone systems specifically. It's not sexy, but my boss is cool and it's reliable employment - have been at the same gig over 5 years. I occasionally feel jealous of my peers higher up the stack who get paid a bunch more and/or don't have to explain "how git works" weekly. Day-to-day, I deal with some tedious old school codebases that can be frustrating at times, but I occasionally get to play electronics engineer, and have an ongoing embedded Rust project to keep things fresh.

"How git works", especially demystifying merge conflicts, has been a major pain point at all my jobs from embedded to frontend web dev, unfortunately.

I'm a bit torn about it to be honest. On one hand, it's sad/frustrating to work with colleagues who can't be bothered to figure git out. On the other, it is sometimes nice to have a less technically-difficult distraction (or at least one in a different problem space) from what I actually get paid to do.

Maybe the right thing is to lean in to it, and become a git consultant! I'm a bit scared to see if that's a thing - would be a sad commentary on the state of the software industry.

FWIW, I'm aware of a handful of people that work for a very large AAA game studio, nearby where I live, who are, in fact, git "experts" -- and, really, just extremely knowledgeable about version control systems, generally speaking, since they don't just use git -- for the company and they spend a large % of their time helping these teams deal with version control issues. As I understand it, they've written a lot of automation to deal with recurring problems like merge conflicts, perforce depot / workspace issues, etc. And they make good money for that expertise...

I guess in the microcontroller and IoT world there's still many? Though now there are alternatives like micropython to avoid diving down to the low level.

Plus most of these are hobbyists. I'm not sure if the market contains enough money to sustain people doing this for a living.

I started at a bootcamp-style program. After getting some experience, moving through junior, to mid, to senior level, there was this inevitable progression to learn what’s going on lower in the stack.

Over time, there is more opportunity to run into performance problems, memory/ gc issues, deadlocks, null pointers… And when you don’t know what the hell is going on, you can end up going pretty deep down the stack before realizing you’re off course.

Obviously, this doesn’t happen to every developer, but it does seem like there are always some devs that want to dig into this stuff.

I’m now working on a team that builds foundational things like api templates, common libraries, patterns… It all seems like this long progression as I work towards lower level knowledge.

Since there are a lot of C/++ people in this thread, I figured I should swing this job posting at y'all:

https://flagshipbio.bamboohr.com/jobs/view.php?id=79&source=...

I'm in no way affiliated with the company, but my friend is (honest!). They have had a heck of a time finding a person for the role. Comp maxes out at ~$120k, remote is just fine. If anyone is good with C++ and would like to transition to the bio side of things, this role may be for you.

>Basic Qualifications > • PhD degree in a field related to Image Analysis, or equivalent work experience

I think I know why.

Any other things that stand out to you? Impartial feedback may be good for my buddy to see.

I mean, if comp maxes out at $120k for this job where you want an engineer with mid-level experience, and Google is giving $190k total comp to new graduates who have no experience, what do you expect? The compensation is way below market for a good, experienced C++ engineer.

The Software Industry is headed off a cliff.

People who actually take care of the low-level plumbing are not being incentivized nor given their due. Expertise and Hard-won knowledge is being lost. There is also not much ground-breaking research going on in these domains which doesn't bode well for the future. As a final nail in the coffin, Colleges/Universities are not even creating/training the next generation of "Low-Level System Engineers". We are going to end-up with a handful of Companies holding the keys to the kingdom and holding Society-at-large hostage.

Well I'd love to write C++ as my day job, but it is not easy to get into a C++ job here in the Netherlands.

There aren't a lot of native code positions, and those who are pay generally less than I get now or have an unclear offering.

Once you take a career path, like C# web development in my case, you generally keep doing that unless you are attracted by a company which really needs you or believes in you.

So my C++ experience is limited to side projects like Marlin based firmware and that's it.

Right here.

https://nanovms.com

If you like kernel level c I'm hiring. The best hackers want to work with the best hackers and we are charting a new course for the 2020s - not one from the 1960s. Despite what you might read in the main-stream dev news our work is extremely important and needs people just like you.

You don't have to work at a boring corporate job -- you can instead work at the unikernel company NanoVMs - you just need to email me.

Hey, can you speak a bit about how nanos compares to something Like unikraft?

IIRC nanos doesn't eliminate syscalls in it's entirety like other unikernels.

We don't think that's a defining characteristic no.

We keep the kernel/user cause otherwise you can very very easily change page permissions which means all security goes out the window. You could instantly make rwx on all pages with a few instructions.

If you rewind to ~ 2013 most unikernels were talking about trashing the context switch yet depending on the "context" that implies various things (kthread <> kthread, user proc <> user proc, user thread <> user thread, etc...) They all have different costs and while we definitely agree with the aggressive cost of various user process to user process the kernel <> user was one we felt was not worth trashing.

Clearly though, because of massive adoption, we see 200% extra throughput on GCP and up to 300% on AWS - so the performance and security remain.

From what I gather it doesn't pay as well as other engineering subgenres, so presumably the market is telling us we don't actually need that many?

I got excited because I thought this was talking about linux systems engineers and how that is a almost non-existent career path these days.

I would call people who write c/c++ code 'programmers' - interesting to see that title is not universal.

If you are wondering where all the c/c++ programmers (system engineers) are - they are in the defense industry - or at least all the ones that I know are.

I’ve been hunting a contractor for C and BPF/eBPF work. It’s been a challenge. Would love to know where to have better luck

If you are asking "where?", you are asking the wrong question, and that is likely why you are having troubles. You need to ask "at what cost?" Suddenly, your issues will disappear and you'll have no trouble hiring (at the proper cost)

Generally, when you are unable to find X at cost Y, either settle for worse X, or raise your price-i-am-willing-to-pay (Y)

Haven’t even gotten to the cost part. But yes, I generally agree. People want unicorns without unicorn budgets.

What are you specifically looking for someone to do?

Still looking? What is the best way to contact you? Keybase?

I work in a place that desperately needs a senior systems/embedded engineer, and our upper leadership is willing to throw so much money at that position that I'm jokingly suggesting that I should demote myself from Director back down to Sr Developer so that I can earn more pay and suffer less bullshit.

I actually learned this stuff in school, but I ended up not working in that field at all. I ended up first in computer repair, and then working in a NOC at datacenter, and then moving into management. Life is weird like that. I do more low level crap in hobby osdev on 8088 now that I don’t write software for a living in any capacity.

I believe ECE (Electrical and Computer Engineering) programs still provide a strong background in low-level systems programming. At least, that was the case when I was at Purdue 9 years ago. My experience with low-level programming has been a boomerang: focused on embedded systems in college, went gradually further up the stack after school due to my interests, and then gradually drifted back down to low-level C++ as I began working on more complex problems.

I think a lot of people currently have "the knowledge" to do low level system engineering, but I agree with the comments that the job market drives what people pursue, and the job market isn't particularly favourable to low level system engineering. So those skills are getting rusty out of lack of use, and aren't being passed on in the workplace because that's not the sort of work people are doing in the workplace.

"Low level" covers many things that don't have much overlap.

In the employment market, I see a significant distinction between "embedded"/"firmware" and "server systems engineering". TLDR: Embedded pay is low compared with other software.

As far as I can tell, from inbound recruiter mail and job ads, the pay for embedded and firmware work is consistently low compared with what's possible in webdev, cloud, and software generally. Even for "senior" embedded and firmware work. I suspect this is because it's adjacent to things like FPGAs and electronics, which also pay low compared with software engineering generally.

By low, I mean senior embedded pay at about 60% of senior webdev pay.

On the other hand, seasoned "system level" engineering such as kernels, compilers, database engines, GPU drivers, storage, crypto, and many other such things, especially things that run on servers, can pay well and seems to be in some demand, especially at big companies.

(Device drivers can fall into either category. For example a GPU or storage subsystem driver is at a completely different level from a USB serial driver.)

Right here. I used to do FPGA stuff in finance and low-level stuff at a FAANG, got tired of all the BS and left to start up my own company. I'll let you know how it goes, but it's not looking great...

Why design new, low level systems when our stacks are built on old, debugged foundations and work "well enough"?

Who wants to think about cycles and bytes when everyone else is talking about gigabytes and "spin up another 1,000 instances" is the expected answer to every problem?

Is anyone hiring these people? Look at the SoC world, the stuff that's being made with the expectation of taking advantage of the public code ecosystem... Would you even want to work at a place like that? "Make the minimal system that passes tests; actual functionality is secondary" is not a good motivation.

Even the RasPi people have blobs of ugly and seekrit hardware; and they're the epitome of "open" systems engineering now, aren't they?

Downvoted for the blind cynicism.

It's true that the frontend side is awash with money, and that there are nowhere near as many such engineers, but we tend to be a lot quieter about it. A lot of people went into IoT.

The economics of the hardware market mean that "open" is even more unprofitable than it is for software, which already has a bit of a crisis in terms of the few people holding up the world not being paid properly.

> a crisis in terms of the few people holding up the world not being paid properly.

Who? The farmers and sweatshop workers in frontier or emerging economies?

You care a lot about bytes and cycles especially when you’re going to spin up another 1000 instances. The larger the scale the more optimisations pay off.

It’s actually small to medium operations that don’t care so much about performance.

> where are the low level system engineers

working on operating systems and software loads for UAS flight controllers, some of them. stuff like an auterion FC is very limited in CPU and RAM.

Any recommended sites for finding/hiring embedded talent? I had A LOT of trouble trying to find someone for a driver development project a few years ago.

I’m doing Embedsys Weekly. With a job offer, you can post the opportunity on the job board jobs.embedsysweekly.com

Here! I am a firmware architecture engineer at a Chicago-based startup. I get paid to do embedded firmware, but I like to make little kernels in my spare time.

I don't get this take... there's plenty of low level system engineers graduating from computer/electronic engineering programs, and coding schools target a very different profile, so I don't think they compete with each other significantly.

Honestly I'd love to switch to systems programming from web CRUD. I just don't know how to get into that field.

Start with an arduino, maybe make a simple game. Then move on to some cortex-m style micros. Get comfy with those. That is already a hireable skill.

I am a college student who loves writing low level and reading this thread discourages me :(

Why? What do you want to work on? FAANGS are certainly hiring “low-level” engineers. Apple is hiring and has relatively low-bullshit interviews (depending on the team).

In fact, Apple has lots of open jobs for exactly the kind of low-level engineers this thread is about and you can find the team managers posting about them on Twitter.

Reading this thread made me feel like all the embedded jobs are low paying or moved to China. And whatever is left has fierce competition. That's why my parent comment.

I think most people on HN are web developers who only talk to other web developers and think the only good jobs in the world are at Facebook. There's a definite bias.

Of course, there's a few more problems than low paying or moved to China - it might be on-site somewhere expensive (Apple), somewhere you don't want to live (probably a lot of these), or boring or unethical (a defense contractor).

> Apple is hiring and has relatively low-bullshit interviews (depending on the team).

I agree. I passed interviews for a Systems Software Engineer position at Apple two months ago, and I can attest that the interview process was very down to Earth. No LeetCode questions. Instead, the interviews were mostly system design and and simple programming exercises where finding a solution was less important than the discussion with the interviewers.

If anyone is interested, Apple is still hiring - https://jobs.apple.com/en-us/search?search=systems%20enginee...

They're bogged down fixing millions of issues.

I'm here. QA/Dev integrated systems.

Garmin

It's a living.

JavaScript 4 ever

Bootcamps and anti-intellectual sentiment spreading through the software development community have convinced newer developers that web APIs and other high-level frameworks/languages are actually the bedrock of abstraction, and that nothing important lies beneath (if they even recognize there IS something beneath). Nothing ever breaks down there, anyway. And if something does, it's not my problem to fix.

16 weeks at a bootcamp is enough to get you up to speed with the important parts of a rigorous 4 year education into the foundations of computing. No one needs to know CS or computer engineering anyway. It's just like math - useless.

I'm a software developer. How does software work, you ask? Uh...

Plenty of graduates specifically don't work in systems engineering for different reasons - its not like there is a dearth of people with degrees. The jobs in say embedded programming are fewer, harder to get, and tend to pay worse unless you're at the top of the market.

If you're at Meta making an OS for VR, you're at the top of the market and getting paid very well. If you're at Cisco making router firmware, or F5 making DNS boxes, you're likely getting paid far worse than at equivalent experience to say a web dev - for which there is a lot more work. Employment mobility is far far lower when doing embedded work.

Short term, yes, there's lots of demand for front-/back-end web developers and the lower barrier to entry makes it a solid choice. Long term, though, it's a race to the bottom as more people flood in and eventually web developers become easily replaceable commodities.

(Don't take that the wrong way; it's not criticism of web development. It's just what generally happens when there's an oversupply of anything in a marketplace. We've been in a boom for web developers for a long time and what goes up must come down.)

But embedded engineers are treated more like commodities nowadays already (except at the top or in specific niches like legacy banking software or HFT), because the industries that drove their day are also commoditized and outsourced. It’s relatively easy to hire.

Yes eventually web devs will be eventually commoditized, but that doesn’t mean the return of low level programmers persay.

My take (and sorry if I hadn't articulated it well) was that it's better to be a commodity in an area that one enjoys working than to become a commodity in an area that one went into just for the career prospects and be trapped there later on if circumstances change. Beyond that, the effects of commoditization seem likely to be significantly worse for web development than systems engineering given that the barrier to entry is lower.

It’s my firm belief that NLP will eat all software engineers; web, backend, mobile and eventually even the ML engineers themselves. Embedded programmers are simply translating business requirements ( natural language ) into architectures at the end of the day. To prepare for this don’t think in terms of “I’m an X engineer”, think in terms of “I can solve problems using X, or Y and unknown Z”, because, very soon, Z is coming to make both X and Y obsolete.

It's my firm belief that anyone reading this post will be dead and gone before Z actually arrives and makes X and Y obsolete.

For folks that hate webdev they shouldn't do webdev unless they're not privileged enough to be able to pick and choose. Its a relatively good market and people are paid and treated well overall compared to most other industries as a whole.

The deal with embedded is that it's inherently associated with a business that has to actually make something. That entails a much higher headcount for all of the additional jobs needed for procurement, manufacturing, test, and more. You don't get to be a prima donna rock star in such an environment.

> Long term, though, it's a race to the bottom as more people flood in and eventually web developers become easily replaceable commodities.

Long term doesn’t seem too great for low level stuff either. It doesn’t seem like a lot of new people are entering the field and getting up to speed on the radically complex hardware we have now

We need a better blended system. I'm the only person I know that's written a compiler, an emulator and written my own kernel modules (for fun really), but my degree was not really worth the paper it was printed on because all the tech we actually learned how to use was 10 years out of date.

As an adverse opinion, I worked at a FAANG company as a software engineer for several years and not once did I ever have to implement one of those fancy data structures or algorithms that are taught at universities around the world.

There are of course outliers, but the vast majority of CS jobs out there are simply shuffling data around. They even call it “shuffling protobufs” at Google.

Reply on news.ycombinator.com