The Silent Majority of Experts | Hacker News Reader

The Silent Majority of Experts

prog21.dadgum.com

274 pointspmarin115 comments

We call this the 'dark matter' problem - when you're trying to hire great people, for every well-known rockstar, there are tens of people who are better, toiling away in obscurity.

The early JavaScript scene was like this - there were people who published amazing JS work and everyone on DHTMLCentral knew who they were, and it was easy to find them and try to hire them. Yet we felt this huge frustration because occasionally we'd interview someone who was as good or better, but we had only found them by accident.

I'm excited about sites like Dribbble and GitHub, which make it easy to showcase your work without having to go through the pantomime of a large folio piece. Though it can be a pain for introverts to do, I've never met anyone who regretted overpublishing their stuff.

I think that GitHub merely shifts the "dark matter" problem. I think your assertion about "rockstars" applies to GitHub just as well - for every person with an impressive GitHub account, there are tens of people who are better, toiling in obscurity.

The point is that you can't just look for people who are famous or prolific on blogs or twitter or GitHub or whatever else.

Exactly. Github (and every other flavor of the time) is an example of the problem. Not the solution.

Nah, Github is a step in the right direction. They make publishing online easier which gets more people to do it. It's like how twitter grew the blogging world by making it easy to do. Sure, not everyone does it yet, but it's better than before

Also with Github you are publishing actual code, and not just writing about code (I am looking at you, coding horror).

I'm not sure that helps much when it comes to recruitment, though.

When someone starting out in the industry asks for advice on how to get recruited, it's very trendy for wisened old souls on places like HN and Slashdot to mumble something about making sure you contribute to Open Source projects and have a GitHub account.

But on the big OSS projects, where making a contribution wins you significant geek cred, is anyone involved in a hire/no-hire decision really going to take a load of time to look through a bunch of commits you made and figure out whether they were any good and what sort of impression they give of you? IME, you're lucky if everyone making that decision even read your whole CV before the interview.

And on smaller projects, no-one recruiting you knows what they are unless they've shipped and have an obvious presence to see, in which case that is probably a far more effective advertisement than the underlying code anyway.

If you've got your own personal project, which is complete and something a potential recruiter can immediately run for themselves, then sure, include a link to it. Likewise if you've got a blog that's going to give an immediate positive impression and is clearly down to you personally, go ahead and include that. But those are things where you can immediately see the whole picture as a recruiter, with minimal time and effort.

I remain to be convinced that there is much value in linking to things like GitHub accounts or major OSS repositories, because they don't offer the same instant gratification. They do demonstrate that you're a keen programmer generally and not just in it for the money, but that's going to be all they do for most potential employers.

This is true, but not the point (at least until automatic tools run through the commit history and rate the contributions automatically).

Contributing to OSS projects gets you through filters, in the same way being Ex-Google will open a lot of doors.

The fact I'm an Apache committer and have code included in Spring gives me some credibility as a Java programmer (even if I haven't been active in either of those roles for years). Hiring managers are more likely to put you on the "worth looking at" pile rather than the "doesn't have a clue" pile based just on that.

Fair enough, but is the advantage here that you have worked on OSS projects, or that you have worked on big name projects that a recruiter might have heard of?

While I've never seen a hiring manager looking at the code actually committed to open source projects, I've certainly been asked in interviews about the contributions to well known projects I list on my CV.

Other interviewers haven't mentioned the contributions at all. So I guess it depends on the interviewer.

Sure, anything you can list as a contribution is potentially going to come up at interview, and anything you can honestly say you've contributed to is potentially valuable on a CV/resume, particularly early in your career. I'm not disputing either of those things at all.

But if you get to the point where you're discussing your contribution in an interview, you've already got past one of the big hurdles, and the guys interviewing you are now going to be forming opinions based on what you actually say to them personally. In that respect, contributing to an Open Source project or hosting something on GitHub didn't convey any particular advantage just because in theory the recruiters could look you up, and just giving a link to a GitHub account rather than highlighting specific contributions isn't really worth much at all.

My advice to someone wanting to stand out in those early days would simply be "Make stuff, preferably stuff that looks interesting". Whether you contribute to an OSS project or build something else makes little difference IME, and so I think the cookie-cutter "You simply must have a GitHub account and contribute to OSS" wisdom is just plain bad advice.

> I'm not sure that helps much when it comes to recruitment, though.

I've gotten a cold-call email from a recruiter who started the email out with "I found your github account…" And all I have in there are two tiny personal apps I wrote to scratch an itch I had.

As a hiring manager, I do look at github accounts if a candidate includes one. Even from a small project you can often tell if the person is clueless or not.

I question your assumptions. It is every individual person's prerogative to decide whether they wish to publish, and we should avoid setting up systems that exclude those who decide not to.

The majority of the most intelligent and productive people on Chrome don't blog, don't have github accounts, and they only way you'd have a hope of finding them is because Chrome is open source so you could trawl the repository and discussion lists. But there must be similar networks on tons of close-source projects.

I guess the takeaway is that:

- If you're looking for work, be public. It is still a huge differentiator. Edit: Actually you should try and make a practice of being public even before you're looking for work.

- If you're looking for people, personal networks are still probably the most effective approach. But looking through the activity of open source projects (not just on github) is also very effective. You'd have to put in a lot of work though to actually read code and discussions to find the best people.

For context, unless I'm mistaken, aboodman wrote GreaseMonkey, went on to work on Gears at Google, and since then has been on the Chrome team.

Yeah, sorry if the comment was a bit context-lacking. gmurphy is a coworker on Chrome.

This problem is why, as a hiring manager, you should be asking your best people whom they know and would hire. Then, talk to tose people, establish a relationship over years, and be ready to jump during the small interval between projects when they might be looking.

Recruiting top people is a multi-year project. Every time.

The only exception is when there's a meltdown (e.g. Sun is acquired by Oracle). And you can bet in most of those, serious players are literally flying down people to recruit out the best people. When the whole Borland/Inprise debacle happened, I dealt with one full-time recruiter who we'd used to capture the few great people who were poachable after that terrible transition happened.

Github suffers the same issue. What about Bitbucket, Google Code, and Sourceforge?

Blogging also takes confidence, that you have something substantial to say and that what you say is valid. Unfortunately, due to the Dunning-Kruger effect[1], actual smart people have too much self-doubt, which would imply that not-so-smart people are overrepresented in the blogosphere/google search/messageboards. Couple that with the internet's popularity-contest dynamics and you have a larger problem than earlier publishing media (which required a substantial level of peer-review-by-experts)/

Take home message: read everything on the net with a grain of salt.

[1] http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect

Also, writing requires a tremendous amount of effort. Effort I'd personally rather spend on creating my next project. When I get to a point that I feel writing would scratch an itch, I find that simply getting the thoughts out of my head and onto paper or into the computer is enough therapy for me to move on. Organizing that into coherent written English takes lots of effort I'd rather spend elsewhere; sharing it on the 'tubes then necessitates dealing with the things you mention.

In short, being technical (and pretty decent at it at least), public writing just Isn't My Thing.

That said, I find that spending a few moments organizing my thoughts into coherent English for commenting on HN encourages me in the direction of writing and could indeed possibly lead to be blogging. Eventually.

I enjoy writing when I get the chance, but right now, working on my own software project, it's hard to take time away from developing in order to write blog posts. This even though I know that raising awareness about what I'm doing is as important as actually building the product.

I look with awe upon people who manage to create amazing things whilst also writing eloquently about them - I just can't handle both at once.

Exactly. Writing well is hard, and you have to spend a lot of time thinking, writing, editing.

'A writer is a person for whom writing is more difficult than it is for other people.' - Thomas Mann

I don’t write because I can, I write because I can’t not write. Alas, looking at my Github projects, it seems that I underscore the problem described in the OP. I have done a lot of writing, but how good is my code, really? If I published the exact same Github projects under a pseudonym, would anyone who hasn’t read my writing care?

I don’t write because I can, I write because I can’t not write.

That seems a better definition of a writer than the Thomas Mann quote.

If I published the exact same Github projects under a pseudonym, would anyone who hasn’t read my writing care?

You can pull a Stephen King and try it. Start rolling out new code under a new name. See what happens.

I think, however, that you'd run into the problem of getting lost in a sea of code, regardless of the quality of your work.

Raganwald, you are a computer philosopher more than you are a programmer. Blogs by you are always interesting, so don't worry too much about it.

'Read everything on the net with a grain of salt.'. Agreed. A lot of the time the internet is about as representative as graffiti is of the majority of people living in an area you have never been to before.

Since HN is full of smart people, I feel like asking - do you guys realize you're smart? Having never made anything worth talking about, I can't imagine someone who makes some great web property going on and saying "oh it was nothing, 5 million users or some number is insignificant" or "my page to the Kernel runs trillions of times a day on a billion devices worldwide, but that doesn't mean much".

> Since HN is full of smart people, I feel like asking - do you guys realize you're smart?

When I go dancing in nightclubs, both men and women will come up to me when I stop to get a drink and they'll tell me they think I'm a really good dancer.

But when I go to a dance class and beside me is someone who works for a world-famous dance company, I can look in the mirror and see my skills are only the crudest parody of theirs. There are certainly thousands of people in the world who can dance better than I can - probably tens of thousands.

So it is with my programming achievements - one or two linux kernel patches for seldom-seen bugs in seldom-used subsystems? There are certainly thousands, probably tens of thousands of people who could have done the same thing easily - I was just the first to feel the itch I scratched.

TLDR: I don't know if your question is even intended for me.

If tens of thousands of people in the world can dance better than you, then let's assume the maximum number of "tens of thousands" before hitting "hundreds of thousands" -- 99999. Let's assume there are 7 billion people on Earth (not perfect, but good enough for back-of-the-napkin). That means you are (1-99999/7000000000)*100th percentile at dancing: 99.9986th percentile.

If tens of thousands of people in the world are better than you at dancing, you're a really good dancer.

Even as I'm writing this I'm at a hacknight in Bangalore, a over night even on BigData. We are hacking away to learn and make a lot of things.

One thing that I observed today was an old friend and colleague sitting next to me. He sort of wrote a Perl program to analyze a weeks worth of tweets in India and draw real cool visualization out of it. The thing is script was not really big and was full of functional programming stuff.

As I was reading the script with another friend, My friend was like 'Oh! I though SQL is unpopular these days, or Perl was dead, or no one uses functional programming these days'. I soon realized- For each of these cries, some one is still sitting and doing awesome stuff with things and using the most simplest and basic tools.

Vast majority of awesome guys don't blog or tweet or harp about every bit of tiny things they have achieved on the Cyber space. But your everyday guy is vastly influenced by ranting, blogging and tweeting and takes them to be holy! And anything other thing than that- which produces results is generally a 'black swan' event to him.

So when something unpopular is used to get the job done, its often difficult to the everyday guy to understand this.

My experience is that less experienced programmers tend to want to get things done the "right" way, so they tend to jump on the latest-and-greatest bandwagon. After enough experience (and having seen multiple bandwagons come and go, they tend to just want to get stuff done.

This change in focus is one of the most valuable things (IMO, of course) that more experienced developers bring to a project.

If the point of this article is to say that you should not be spending time worrying about and evangelizing your tools and more time using them to create awesome stuff then I agree. However if its to say that there is a large crowd of exceptional programmers and engineers that no one hears about because they spend _too_ much time creating great projects then this is baloney.

There are some people out there who are truly exceptional and might never be heard from. However in general the real exceptional people are the ones pushing the envelope of whats possible forward. Their work _always_ finds the limelight. The author uses an example of a rigid body physics implementation in a game, and while that's great work and the team should congratulate themselves its not that big of a deal BLAS had been around for decades by that point. That does not describe experts doing amazing things. It describes software developers doing a good job and not being cocky about it.

I think why this has received such a warm welcome here is so many of us software developers want to think we are part of that silent majority. That we are doing amazing things too and if we cared to we could write articles and become famous. The truth is we are not experts we are people who have jobs to feed our families and we may be just as skilled and talanted as the vocal minority, but we would never be able to reproduce their success. That's not a bad thing. I have no problem being an unknown developer working on a piece of corporate software and providing a good living for my family. Its ok if others get press time for their contributions I might take from them and learn, but it doesn't change the fact that they at the end of the day have moved our profession further and I haven't.

You're wrong.

There's only a mild correlation between fame and skill set. I can name a few dozen 'famous' internet developers, and have even interviewed a few of them, and most weren't anything special, other than being good bloggers.

Meanwhile, take someone like Jeff Dean (who I just learned about a few days ago). Designed and implemented BigTable, MapReduce, Protocol Buffers, etc. I'm not sure most people outside of Google have even heard of him.

Sometimes your work finds the limelight but you aren't credited for it. We all know Elon Musk but who knows the guy who actually wrote the code for the rockets? I don't remember the name of the Amazon Dynamo guys, and have no idea who designed EC2 and S3. These are awesome systems used by millions of people, but the creators are (relatively) unknown.

Jeff Dean would be exactly who I am talking about. I don't know how you managed to _not_ know who he is. He along with Sanjay Ghemawaty pretty much took google from a research project to the Juggernaut we know about today. he has no blog that I can find no GitHub account and yet most people who work in the field of data processing should know who he is. He has numerous published research papers and has given talks about his work at Google. Someone of his caliber can not stay hidden. Thats the definition of a "Master" at their trade, if your really _that_ good at what you do you can not stay in the shadows people will search you out. This idea that there are a majority of people like Jeff who no one knows about is silly. they may not get profiles in the new york times, but anyone who want to do what they do will know who they are.

Also how can you not know who made Amazon Dynamo they published a research paper on it.(http://www.allthingsdistributed.com/files/amazon-dynamo-sosp...)

When you are considering hiring you don't want to hire the already famous Jeff Dean, you want to hire the 25-year-old that will become Jeff Dean. This person is not in the limelight, and not everyone knows their name. I mean, you want to hire Jeff Dean too, but you'd be happy to find all the 85% as smart as Jeff Dean people out there who will never be famous based solely on their work and without self-promotion.

I would but that 25 year old is not a "Master" as this article represents. I know the context of the discussion on this thread has shifted form the content of the article, and in the context of trying to hire talent yes there are a lot of talented people who are in the shadows, but my argument is they are not the true masters and innovators of our craft.

"An appealing theory that gets frantically upvoted may have well-understood but non-obvious drawbacks."

I'm reminded of the quote, "To every problem there is a solution that is simple, elegant, and wrong." I have fallen into the trap of looking at a long, messy piece of code, thinking "I can do better than this". I would replace a 50-line code with 5 lines, only to have it fail at some random edge case, which the original has been fixed for.

That is why I always remain skeptical of people who are out to "disrupt" an industry, especially when they don't have much experience in that industry.

While often you do get rock solid code that is ugly and just works. there is nothing to say that elegant code can't solve all the problems and do it in a more readable and maintainable way. When I started transition my coding style form a imperative/OO base to a more functional base I found that you can get many elegant solutions that handle edge cases and are maintainable by designing a solution instead of growing one.

Cant remember where I read this, but I realized a while back the folly of "rewriting your code". Your code looks messy because it works. Its the result of weeks of testing and bug fixes. It can handle special cases and is robust.

Many a times, to start of, a clever 5 or 10 lines would suffice. But the moment you start testing it for production, it'd require more and more logic to handle _all_ the cases.

I think this is very common with new simple magical frameworks / languages too. They work really well for most common tasks but can't handle the edge cases.

I'm pretty sure you're referring to a Joel Spolsky article, "Things you should never do". http://www.joelonsoftware.com/articles/fog0000000069.html

Excellent post, thanks for the link.

Don't rewrite, refactor!

I liked this sentence. It's so very true.

"Just because looking down your nose at C++ or Perl is the popular opinion doesn't mean that those languages aren't being used by very smart folks to build amazing, finely crafted software."

Dagdum has been harping on this concept for over a year at this point: "don't waste time talking about your tools, you can do amazing things with everything, go do them".

It's a platitude, sometimes it's even the wrong mindset to have, tools do matter. And worse still it's starting to sound hollow given that all dagdum talks about is how you should do awesome stuff instead of talking on the Internet.

How about he talks about something cool he did recently instead? I guess that doesn't get him as many page views.

At least in this case it is actually correct, unlike previous entries, C++ and Perl are filling two very specific niches perfectly and their obvious shortcomings are the flip side of their strengths.

An ounce of passion is worth a pound of tools. You /can/ do amazing things with limited resources and crappy tools.

You have to be careful about this. Using this as a justification not to have good tools ("What's wrong with you? Suffering worked great last time") is bogus logic; there's no reason not to have good tools. The other side of the coin, NOT doing something because you don't have good tools yet, is equally poisonous.

- The test team that spends all its time writing test frameworks and tools, and never writing any actual tests. (How many times have I seen this?)

- The engineer who spends half his time "optimizing" (finding the perfect language, defragging his hard disks six times a day, tweaking TCP settings so repository access is as fast as possible) and not writing code.

- Going around and around on a class design until it's /perfect/.

- Not moving on something because you lack complete understanding.

It's hard to overstate the benefit of just getting in the ring and doing /something/.

After 25 years of random fiddling with one side project [not going to tell you want it is] I finally just started writing some fucking code. And you know, it feels darned good. The stuff I'd been worrying about being inefficient and so forth turned out to be only a few hours of thinking and coding. It was fun.

A poor craftsman blames his tools.

A poor craftsman blames his tools -- because one upon a time, a craftsman would make his own tools.

If you give a shovel and a screwdriver, don't complain if the house I build is not up to code.

If I show up with a shovel and a screw-driver, you have every right to send me away.

Productivity is productivity, just like "Power is power."

http://www.youtube.com/watch?v=ifaRhL95HUM

http://www.giantitp.com/forums/showthread.php?p=13003072

Using this as a justification not to have good tools ("What's wrong with you? Suffering worked great last time") is bogus logic; there's no reason not to have good tools. The other side of the coin, NOT doing something because you don't have good tools yet, is equally poisonous.

This middle ground approach is probably best.

PG writes about tools and their importance in "Beating the Averages": http://paulgraham.com/avg.html .

>It's a platitude, sometimes it's even the wrong mindset to have, tools do matter.

They help, but they don't matter one iota in most cases. Whatever one is doing in Rails could be done in Django or PHP or whatever. A better tool can help, but not as much as using what you already know, using things that have programmers to hire in your area, using what has community support, etc.

Do people look down their noses at C++ programmers?

If you are using C++ you are probably doing so because you're programming something "closer to the metal" so are probably more likely to know your stuff technically.

I see a lot of derision aimed at C++ devs -- from both higher and lower level language advocates.

On the one side you have the "C was created by God himself, and undefined behavior is expected", and then you have the "languages should be as restrictive as possible because this one time I worked with a developer who was very bad."

It's all very tiring. I do mostly Ruby now, and I love how expressive it is. It reminds me a bit of C++ in that regard. It's just funny how the mindset has shifted.

There is a bit of that, but at least in my circles I more often run across people aiming derision at C++ the language, and the people who are most vituperative are people who have years of C++ dev experience. People who don't program C++ often have mildly negative opinions about it, but something about the language seems to inspire bitter hatred in some percentage of the people who actually have experience in it, in the way one can only hate something you've been intimately familiar with and once may have loved. Yossi Kreinin's scarily thorough "C++ FQA" (http://yosefk.com/c++fqa/), where he goes through the entire C++ FAQ and criticizes every single point based on years of C++ programming experience, is sort of a paradigmatic example. It reads a little bit like the Unix-Haters Handbook in its depths of familiarity/dislike.

A good programmer chooses to use C++ because they're close to the metal. A lot of people just use it because it's their Blub.

I'm currently writing a C program that runs on a router. The router has 4mb of flash. It also runs linux has wifi and all the usual crap. We added VPN support. I don't have the "luxury" of including Perl or any other "run time system" like Python or Ruby because I have about 50kb of flash left for my own code and any customizations. In this case C (maybe even C++, but why if C will work?) is the best choice for the small userspace app I have to write. Over the years I've worked on enough embedded systems of all sorts and C has been the obvious choice. It worked out pretty well.

Whenever I meet someone who extols the virtues of language X I always, sort of cynically, ask them what low-level embedded work they've done and how X could be utilized for that use-case. It's one of those areas where most programmers don't seem to have much background, which if they thought about it would make the utility of C readily apparent.

I guess I should have said, "a good programmer might choose to use C++...". My point was, just because C++ is low-level doesn't mean everyone who uses it knows what they're doing down there, or could have chosen a different language if it wash good idea.

Edit:rephrased last clause.

The same can be said about COBOL.

Or Visual Basic.

Or PHP (shudders), which Facebook (for some reason) relies on.

The "shudder" is against the spirit of the quote you were replying to:

"Just because looking down your nose at C++ or Perl is the popular opinion doesn't mean that those languages aren't being used by very smart folks to build amazing, finely crafted software".

Ie: don't shudder.

I accept that smart people are using C++, Perl, and even PHP, to produce amazing, finely crafted software.

I do not accept that they are making it any easier on themselves by this choice of language.

I work on a project that uses C++. Our runtime is implemented in C++, and our compiler produces C++. For us, performance is a top priority. I am quite frustrated by C++ sometimes, but I think it's the best tool for our job. I'm quite fond of pure C, but I also like having rich abstraction mechanisms like classes and templates, so I don't think it's appropriate for us. And no other language with broad support is as well-suited to writing high performance systems code while also allowing us to have rich abstraction mechanisms.

I am interested in D for this purpose - the most recent version looks quite well designed - but it's not as widely supported.

Isn't it easier to work with the tool that you know thoroughly, which may not be "cool", rather than pick up a new tool in a different ecosystem?

Who said anything about coolness? Seriously, why do people think that just because all languages are Turing complete, any further discussion is just a popularity contest?

The point is that if you know one of the languages cited by the parent ("C++, Perl, and even PHP"), then you do make your life easier by working with that language rather than choosing one which you don't know.

All other things being equal, sure, choose the maximally powerful language in whichever dimension makes sense. But all other things are usually not equal.

If you are working on a project by yourself, here are some things which are more important than the characteristics of the language:

  - your knowledge of the problem domain
  - your knowledge of general software development
  - your knowledge of the limits, capabilities and ecosystem 
    of the language (aka project experience)

Also, if you're working with others, the following factors are more important than choice of language:

  - your customer's expectations (if you're shipping software to them)
  - the ability to recruit experienced individuals for your project
  - the availability of mentors / reviewers within your organization
  - the availability of secondary resources (books, FAQ, etc)

(edit: formatting)

Only in the short term.

Picking up a new tool shouldn't take long, but can give a significant benefit. As I like to say, learning something well enough to use is an O(1) action where the benefit is in O(n) to how much you use it.

In the long run, the quality of the tool dominates your current familiarity with it.

Simply using a language to build interesting software doesn't make it good.

Very true.

While bringing lots of money home, I would add.

I remember, but sadly haven't bookmarked, a blog post by a guy who made a fair share of money building a decent music player for BlackBerry when all cool kids were building iPhone apps. Actual needs that will get people to pay money can be quite unhip.

This amazing, finely crafted software is built despite the handicap of lesser tools, not thanks to them. Which makes the feat of crafting that software even more amazing.

Sure but C++ and Perl don't fall in the category of lesser tools.

I've worked with both and cursed them both to death. But they achieved crazy popularity for a reason. They are extremely powerful and capable tools.

C++ especially can sing in the hands of masters. (And anyone who thinks they mastered C++ in less than 10 years doesn't realize how much more they have to learn. (I mainly curse it because once you have 20 people on a C++ project, there's no hope...)

And you get a lot of attention for bashing something, which aggravates the problem.

What problem? You can bash a language without saying anything negative about its users. Not all programming languages are equally good, for any reasonable definition of good. This is a valid thing to discuss.

There is a more general point here. There seems to be a growing assumption that activity on the internet is some how proportional to activity in real life. Yes this could be a problem for programmers, but IMHO, its becoming more and more of a problem with news gathering. Issues or causes with facebook and twitter accounts get far more coverage than those with out. Basically these sites provide tittle tattle to fill the 24hr news channels. OK, cool if you happen to be in a banana republic dictatorship with internet access, but deep in Africa, your tribe can be slaughtered and no one knows.

Incentives also tend to disfavor "popular" work like blogging, but I think in parts of academia that is somewhat changing. May be changing less slowly with regard to experts in industry, some of whom also have confidentiality issues to worry about.

As one example, law-professor blogs have really taken off. They're good for building reputation and making a legal argument actually have real-world impact (people read them, whereas the average law review article is little read), and sometimes have even been cited in court decisions! So there is a movement towards law professors doing more popularly oriented, but still legally sound, writing; and towards universities taking this into account.

Science has also long had some research/popular-writing crossover, and there are a number of researchers who blog in areas like physics and climate science. And mathematics has an increasingly strong blogging culture, with Terence Tao setting the nearly impossible to top example. So I think there is good stuff out there if you know where to look. Of course, it's also good to experiment on your own, and also to look elsewhere (e.g., in books).

Implicitly in contrast is a loud minority of newbs, with their resume-building blogs that have an about-me page, a picture of themselves, and reposting/paraphrases of old Joel on Software posts.

Hey, we newbs can contribute SOMETHING--namely useful content for other newbs (that rises above the noise). Newbs have to learn somewhere, and if experts aren't contributing, then to whom do we turn?

As one among them, I agree.

As far as I know, most Wikipedia articles were mainly written by students, not experts.

I'm surprised to see the above comment got downvoted. Do people think that excited acolytes contribute nothing?

I agree with you. There are things that the experts don't want to talk about (because it's too easy) and then there are things that a beginner can explain in a simpler way than an expert.

Staying in a beginners mind is hard once you have spent some time with a technology. It also sometimes offers great points for the experts what could change, because it could be easier or is simply unclear.

Of course there are very simple-minded blogs with practically no content, but there are also simple-minded Github accounts that offer practically nothing. There are CVs for the same purpose. I don't think this has anything to do with blogging per se.

I signed so many NDAs in the past 4 years it almost looks like I did nothing.

One thing is being a good programmer and wholly another selling oneself as good programmer. And that's absolutely not sad state of affairs; it is how it is and it is how it should be.

Why is it how it should be ?

The fact that I know how to sell myself makes me better paid, better treated, and better recognized that others who have the same level of programming ability as me but are less good at shouting it to whoever wants to hear it. And yet, I don't provide anything better than them to anyone. Worse, it's overall detrimental, because I spend time making myself known instead of doing actual stuff.

Working with/for someone isn't programming, it's business relations, and thinking otherwise hurts both parties. It is a nuanced difference, though.

Err, yeah. Except at the end of the day, the goal and outcome of the relationship is the programming (or, well, the technical side in general).

The business relationship is supposed to be just a construction between us, in order to be able to agree on the tech work. If there is no need on the other party, they have no reason to do business with me. People don't just hire me because I'm a cool guy and I say funny jokes and blabber insightful-sounding things. They hire me because they believe I can, one way or another, provide them some value. Telling jokes, or writing link-bait-ish blog posts, (or doing mostly shiny and maybe useful software), or whatever you want to put under "developing your personal brand 101" is just a way to make them believe that in order to put myself in a position where they would allow me to provide value to them. It doesn't, in itself, provide value to anyone.

Or, at least, it provides less value than the time I spend doing the real, and boring, and lonely, and hard to justify research groundwork that could (or not) lead me to developing something valuable. And I mean research in the broad sense, be it evaluating new or old software, trying to talk to people and to discern if they have problems that I could solve, or just chasing after a sneaky bug in an obscure backend thing that I happen to maintain.

IMO this is why people who post online think GCC is more popular than it is. VC++ and C# have far more usage than GCC, but most of it is by professionals who don't post online. Ditto for the lack of awareness of the popularity of Perforce in professional environments.

I've been fascinated by this phenomenon for a long time.

The best coders I've found are not the loudest voices. In many cases they do not comment publicly on the web at all. They just submit bug fixes.

That said, the best code I've found is the stuff that's made publicly available via internet, and I'd argue it's precisely because it allows all those "silent experts" to review it and submit their fixes. It's not the best simply because of the contributions of the silent experts, but because their scrutiny raises the bar.

It takes a certain amount of confidence to make code avaiable for all to see. Because it becomes very clear, quite quickly, whether it's any good or not.

I don't see this same type of peer review and vetting within the corporation. The goal there is different: one need only satisfy the company's clients, most of whom cannot themselves review code quality.

In my experience, many of the silent experts who submit fixes work at corporations. They are no strangers to closed source.

So it's all quite interesting. Open source and open discussion in its own strange way effectively raises the bar. But at the same time the web also lowers the bar to spreading bad code. And disseminating much nonsense chatter about programming.

Hence we have lots of chaff. Lots and lots of chaff. The wheat is there, but you have to know where to look. And generally, when you find wheat, you'll also find lots of "silent experts" submitting fixes.

I often wish which they were the ones making the noise on the web. Because then it wouldn't be noise. Many of these guys really know their stuff. I guess over time the web has worsened things. In the early days, many online contributors, e.g., to bulletin boards and newsgroups, really were knowledgeable in their chosen fields. There were just less people online. That seems to have changed somewhat, but maybe it's just my perception.

In any event, they might be silent, but we know they are paying attention. No one is going to completely ignore the continuing growth of open source and open collaboration.

No doubt a lot of expertise is locked away in commercial software, but the open source community has equally (or more) impressive experts working on software. Maybe not blogging about it, but the expertise is definitely there.

The thing is: (blogging|writing)-about-/publishing- code takes time. And time is really what people doing amazing stuff are short on the most. Don't get me wrong, both discussing and publishing are amazing, but often life is just too short for the blah blah blah, unless you have good reasons for doing that.

I think this problem is why academia is so valuable; academics sole purpose is to share their discoveries. What they build is not a product, but rather a set of ideas ( or others' ideas) to share with the public

if only I'd read this article two - maybe five years ago ...

I think this has more to do with age than anything else. But it might also be because you can do stuff faster in Ruby or Python or JavaScript so you have all this time to wax poetic about it. ;)

There's still nothing like C/C++ for getting close to the metal. And until we have an OS written in Ruby or Python, rather than C, it's gonna be like that forever.

It's funny you should mention that, because the language in the blog post was Forth. And you can run Forth directly on a chip, unlike C. I'm pretty sure that's exactly what GreenArrays is doing[1]. Starting this fall, I think I'll be working on something similar as well.

[1]: http://www.greenarraychips.com/

Or what about lisp? While these were well before my time, my understanding is that they could run Lisp directly in hardware.

And, of course, there have been OSes mostly written in higher level languages than C or C++. The one that springs to mind immediately is the Singularity OS[2] project from Microsoft that is mostly implemented in C#. I think there is still some assembly, C and C++, but the kernel and drivers are in C#.

[2]: http://en.wikipedia.org/wiki/Singularity_(operating_system)

People still building software in Perl ought to be arrested.

I know a guy working at JPL doing Perl stuff for internal use still. I give him a hard time about it, but it's a lot cooler than Social_Widget_0003993986 in any other language you care to name.

"but it's a lot cooler"

Are you referring to the work or Perl? Most programmers I know working in a scientific area switched to Python years ago and run away screaming when they hear the work "Perl".

I cut my teeth on perl and don't run screaming when I see to this day. Perl has an undeserved bad rap. I like python as much as the next guy but frankly Perl does some things way better than python. And don't get me started on Pythons rules around scope.

Choice of language is quite marginal when it comes to determining the quality and utility of code.

Actually, it's been known for quite awhile that there's a factor of 2 to 3 times in productivity. That's not really marginal. Talent far outstrips that, though.

Indeed talent does. I make no claims about productivity. Your note is intriguing, references invited.

productivity != quality and utility

It doesn't equal utility, but it does equal quality, at least to the extent that bugs per SLOC matter. This has also been known for awhile, with data to back it up.

I'm not sure that's true. Though it obviously depends more on the programmer themself, the semantics of a language can make it easier, at least, to write higher quality code. For example, bounds checking removes a whole class of bugs (probably the most common in C) from the equation. I disagree that this effect could be described as "marginal".

yeah but other languages have better minima cause they hand-hold so much, like python

Is it a silent Majority? I don't think anyone will doubt that not every interesting thing that gets done gets published and discussed on the internet and on blogs, but I don't know if I would say its a majority. From my experience there are some people in the software world who think they are either not noteworthy enough to write about what their doing or they are too noteworthy ( I am not going to share my amazing super advantage I got by making this really smart and novel thing). I have found however that in general software development, like any technical trade, works best when its subject to peer review and an open community. I think most people also know this, which is why the tech heavyweights have talked about their infrastructure and technique. They do it so others can build on what they have done and as a community we can build on each other.

There are a few who don't care to participate thinking they are either above it or below it , but the truth is most people do and everyone should.

I've worked with far more excellent people who don't blog than with those who do. In many cases, confidentiality agreements prevent the really cool stuff those people work on from seeing the light of day (except of course nestled deeply within the binary of the product in question, if it's even for public release). I don't have data, but anecdotally I'm completely willing to believe it's the majority.

That really cool stuff nestled deep in a propiratary binary, Odds are its not really that neat or cool. Thats not to say the people who made it aren't really smart and good at what they do, but odds are its been done before. By refusing to learn from and give back to the development community all they have done is made sure no one ever learns from something they might have done differently or better. Meanwhile the experts in the field are the ones who have taken whats been done, understood it, and built upon it. My point is a majority of those people _are_ giving back in the form of public discourse.

Are you really trying to argue that the majority of that which is novel is already open source, or has been discussed publicly? That's a really bold claim to make, and I really doubt it. NSA and DOD black projects? The majority of software products that I can think of are more advanced in proprietary form than their open source counterparts. Google's search algorithm?

The NSA will have some really novel and new ting we don't know about. They might be one of a handful of places where you really do have some Silent Experts. However as to your second example google

http://infolab.stanford.edu/~backrub/google.html

This is really common I can pull you up multiple research papers from them on everything from database design to ranking alogrithems. Same thing for Facebook, Twitter, IBM, Cisco, and the numerous computer science research institutions.

For the most part what you see as "Advanced" is just a better implementation of the same idea. Thats not to say that the people making the proprietary software aren't intelligent and aren't building better products because they are, but its just silly to assume that there is this world of computer engineering talent we don't hear from that's doing groundbreaking things in the shadows.

Good point, but that's research, not general software development, which is what you were initially talking about. Software development is about implementation, and less about research. That research paper is not the entirety of what's novel about google's search algorithm. It's more silly to assume the majority of computer engineering talent get on the megaphone when they've done something novel, many of them just aren't interested. Games developers leave it a couple of years before getting public about the nitty gritty of what they did.

You know already which one this is going to be :) http://xkcd.com/664/

I think you're taking too narrow a definition of "expert"? - or maybe my definition is too wide. I just wouldn't expect one to necessarily be pushing the boundaries of knowledge. They just need to be good at what they do.

Two things come to mind...

1. You can learn from a community without giving back to it.

2. It's not really a developers choice if their work is locked in a proprietary binary. That decision was made by their employer, and the only thing the developer can do is get another job (which isn't something most people will consider).

That really cool stuff nestled deep in a propiratary binary, Odds are its not really that neat or cool.

Well, cool could be defined to include popularity. A lot of ideas were great before they were cool. There are a lot of great ideas out there now that aren't cool.

>That really cool stuff nestled deep in a propiratary binary, Odds are its not really that neat or cool.

Really? By what logic exactly did you come to this conclusion?

The irony is that many of this "silent and modest" group, would also not bother to react to the discussion here.

Even today, the web is a minority of programming.

Reply on news.ycombinator.com