My belief is that management & compensation should be completely decoupled, so people wouldn't seek to do management for "career advancement", but because they actually enjoy management. For example, I hired a product manager last year and he really loves the work. It allows him to mix small engineering projects with product strategy, requirements elaboration, and his considerable people skills. But I don't see him as a "superior" to my other engineers -- he is just a contributor, like my other engineers.
The "pay scale" for management should be similar to the pay scale for advanced individual work, in my opinion. Unfortunately, that's not the market we live in, so I can see why people would choose to seek management for "advancement" reasons, but this is just an unfortunate side effect of the market and ultimately causes a lot of talented contributors to do work they don't really love. Why? Essentially, for the money & prestige that needlessly comes with it.
I'm thinking of the recent Ask HN "What happens to older developers?" Which I answered in some detail here https://news.ycombinator.com/item?id=7373301
There's lots of reasons for wanting to get into management, and I've found that one of the main motivators is wanting to get out of direct code-slinging software development as the majority of their day and use their experience for the company good in different ways -- basically new challenges and all that...but keep a similar level of pay. And very few people these days want to stay at a single job for more than a few years. So for a very experienced developer, they may start to feel "trapped" in their career and may not even know it.
Sorry, I don't want to be contradictory or make you defend what's obviously a working system for you guys. Just food for thought.
So decisions on who gets promoted are made by people for whom getting promoted chimes with their value system.
One route to that is by getting out of mainline production coding and going meta: designing new systems, programming languages, tools etc. Good work if you can get it, but prone to failure. Another route is to become a manager. There's plenty of demand for managers with technical ability, a clear route for advancement and much lower risk of failure.
The managers your describing sound more like PM's.
I come across mentions of WWII era projects, designs of famous aircraft etc where the designs were incredibly well suited to the needs, and produced in incredibly short time-frames. Some of that is obviously the war-time focus, but I also notice that in many of those projects the lead position is not a non-technical manager, but a technical chief engineer where management functions must have been subservient. Unfortunately, there isn't a huge amount written about the organizational day-to-day project structures during WWII.
I think at least some of the appeal of startups and smaller companies for technically oriented people is that division of responsibility is not as formalized and contributions can flow more easily between different domains. But again, startups are structured where technical founders are at the levers of control, and that's inverted from how many companies end up organized as they grow.
http://www.amazon.com/Skunk-Works-Personal-Memoir-Lockheed/d...
Imagine a carpenter who sets his sights on working hard at carpentry and being a good carpenter so that he can eventually advance his career by becoming a plumber. Makes no sense, right? These are completely unrelated skills. Maybe plumbing pays more (reverse the roles if I got that wrong) but that doesn't make it an "advancement". Certainly one could make the move, but it's a lateral move, not a logical progression.
Management is the same way. It's a different field. Yet somehow we have this idea that if you do well enough in some other field, "advancing" into management is a logical next step. It makes no sense.
It can make sense.
A leader needs the respect of those she/he leads. It is easier to command respect when you are perceived as a technical expert or authority.
Promoting the Dilberts to bosses is an attempt at giving the engineers what they say they want: bosses with Dilbert's engineering skills.
The current way it works in trades like that is to move up to being a foreman or a site supervisor or similar. It's no different really.
Ideally companies with management gaps want to move experienced tradesmen into these roles rather than put "management professionals" who wouldn't know a 2x4 from a band saw into them.
Segregating out "management" as an entirely separate discipline makes a kind of sense, and it's why management is taught as a separate skill set with certificates and MBAs and all that. But I think the results have been mixed (90s Apple is given as a textbook example). For every professional manager who does a good job, I think it's possible to find somebody who moved into management who does as good or better because they understand the problem domain of their reports better.
Wholly agree though that it's a separate set of skills, but I think those skills usually work best when layered on top of practical experience in the field.
Yeah, you're right in that. It obviously doesn't work that way, there are far fewer management positions than people who could be promoted up into those positions. Even in some of the most hilariously top heavy organizations I've seen, this holds true.
I guess we have a cultural model of the "career" that somebody can start "in the mail room" and work their way up to CEO. Any deviation from that as a possible path is seen as veering off course or failing or "career ending" and you just end up in mere "jobs". You're right of course that this is unfair as the vast majority of people will never be on a career path that looks like this.
There's also an old fashioned class-based hold over in organization structure: nobles and commoners. This has been held over as "management" and "workers" or in the military as "officers and enlisted" and it seems that a great deal of our organizational theory, promotional structure and cultural ideas about progression are based on this: the "organizers and communicators" and the "producers".
Is there a better way? Maybe. But I don't think current counter-approaches work well over the long run, e.g. flat org structures. As a species we seem to naturally arrange ourselves into hierarchical structures, and if one isn't imposed on a group, one will emerge.
So I guess the idea of management, and it being a career path, is complex and is simply an optimization on these issues.
I've also been in organizations with very explicit, upward moving career trajectories that didn't end up in management per se. For example, one very large R&D organization I worked in was broken down like this.
You had the "employee". They all theoretically started as a "researcher I". As you progressed into "researcher II" then "research associate" then "research scientist" you were offered two choices go into "people management" and become an "associate manager" where you did normal people management stuff, some HR functions, signed off on time cards and did promotion stuff, but otherwise didn't involve yourself in the day-to-day of an employee's work life. Employees were simply "resources".
Or go into a research track. If you went research track you would then end up as a "research scientist" then a "principle research scientist" and then a "senior research scientist" (with research fellows etc.) Around the time you became a "principle" you were then offered two choices, stay in research, or go into program management. A program manager "owned" a program and requested "resources" from associate managers who were then matrixed under you. You directed their day-to-day, but if there were employee problems, you took it up with their associate manager who then dealt with it.
However, and this was the trick, if you stayed purely research, you'd of course continue doing research, but at that level, you were more valuable assisting the PM or the sales/marketing team (even R&D firms have this) with your expertise in getting research grants, writing proposals, etc. Quite often a PM would assume all the contract management stuff and the Senior Research Scientist would end up running the day-to-day of the lower level researchers. In the aforementioned military model, this ended up looking like an officer and his sergeant major. Or in academia, the principle researcher and his post-doc, with all the grad students. In practice, you'd end up becoming a manager.
However, in the line of "mail clerk" to "ceo", you were out of the game already. Nobody viewed it as a failure, Senior Scientists were revered like high priests there. But a priest cannot become a noble or a king. Only if you were an Associate Manager, or to a lesser extent a PM, would you have a shot at the line. Progressing AMs and PMs ended up assuming other roles, a bit of sales work, and for AMs a bit of PM work. I'll let you decide on the downsides and upsides (there are actually quite a few) of this system.
Some PMs ended up floating back over to Senior Scientist level, because they were already good at doing all the administrivia, but wanted to get their knuckles dirty in research again. Unfortunately, they usually found themselves more mired in more paperwork than pure research, and nobody wanted to take the pay cut and work as a Researcher II again where your time is 100% research. The economics simply don't work out to have a Senior Scientist running lab tests and squeezing pipettes all day. But they're the only ones with enough domain knowledge to write the grant proposal that will bring in a $30m 5 year program.