Reddit Is Down To One Developer | Hacker News Reader

Reddit Is Down To One Developer

techcrunch.com

197 pointscitizenkeys91 comments

I took a job some time ago (back around when the internet was morphing from uucp to ppp) as a consultant to an "independent agency of the United States government" which seems to enjoy placing big blue boxes all over the landscape.

It was my first real job outside of the university system, so it is safe to say that I was rather green. This job basically involved being an apprentice to someone who oversaw the code supplied by vendors that sorted a tremendous amount of non machine readable mail, i.e. it assisted humans using chord keyboards sorting mail with pneumatic arms and big chains and slots. Back then, a fair amount of mail traveled this way - stuff your grandmother wrote that couldn't be OCR'd, postcards, etc. The whole apparatus was similar in size to a semi trailer [1]. Each station dealt with a piece of mail every second (give or take), with two fifteen minute breaks and a half hour for lunch. This was around the time of the postal shootings [2].

Two weeks into the job, my boss quit. Just never showed up again. Not exactly sure what happened. Apparently, I had become the most knowledgeable person in the organization on this software which was currently being rolled out widely.

Unsurprisingly, there were some serious code quality issues. This was QNX real time sorta unix in C and ASM with RTOS daughter boards. The vendor's code jockeys, apparently, had become unavailable.

Main takeaways: Systems seem to keep working despite everything looking very fragile. Read tons of code. If it's working, don't poke it.

A year later I was able to make some form of industrial dance music by coordinating the openings and closings of the mail slots and the chain with the software. Bad situations can sometimes lead to greater clarity and new opportunities.

[1] http://www.usps.com/postalhistory/images/photogallery/equipm...

[2] https://secure.wikimedia.org/wikipedia/en/wiki/Going_postal

It seems like a common situation in the tech world for those with a little capability to deal with some legacy system that you somehow get lumped with or become the "expert". Maybe it's inevitable, people do get bored with doing the same things all the time.

I'd love to hear some of the music that came out of your experience - anywhere to listen to it?

I wish the TC article had done more than merely mention that keltranis is "joining Alexis". Hipmunk now has both reddit cofounders, their first hire in keysersosa, and their most senior programmer in keltranis. I wouldn't be surprised to see jedberg join them before too long, and then hipmunk would have almost all of the old reddit team on board.

Me too. It is a total dis-service to hipmunk to not mention Chris or Steve.

But for now I think their tech team is full, so they won't be needing my services. :)

Does that mean you'd consider joining them if they asked you to? Also, given the concerns expressed at [1], once you get a healthy dev team would you guys look into maintaining redditOSS a little more thoroughly? I imagine personnel shortage has made that impossible to date, but if you had a sizeable group is that something you'd do?

[1]http://www.deserettechnology.com/journal/reddit-the-open-sou...

> once you get a healthy dev team would you guys look into maintaining redditOSS a little more thoroughly? I imagine personnel shortage has made that impossible to date, but if you had a sizeable group is that something you'd do?

Yes, very much. We would like to make the open source software MUCH easier to run, so that it is easier for others to develop on. It will probably be one of the first projects for a new hire, because it will make it easier for THEM to develop too.

Unless they're working on new features which are needed tomorrow, one dev seems fine in the interim. The number of sysadmins is more important to keeping a site running, but of course that's not what's in TC's headline.

Reddit's sysadmins neither have to maintain DC racking (nor travel), maintain hardware lifecycles, nor maintain vendor relationships. They certainly appear up to their teeth in working around AWS (EBS) deficiencies. They're not going to control a lot of their hardware, and most "fixing" of AWS pitfalls are going to come in the form of software.

Now if we are talking devops, I can agree. Engineering teams this small need to wear many hats; I think jedberg has been said to have helped in development efforts, if my memory of past blogs serves right.

Yes, I do dev too. I just haven't done it in a long time.

A lot of our sysadmin time is spent tracking down bugs in the stuff we use (like Cassandra) and doing tuning and automation.

Also, I do a lot of the other business stuff, so at best I'm a part time sysadmin right now.

We do pretty much do "devops" though. A lot of the work of Neil, our programmer, is tracking down bugs due to data inconsistencies due to hardware problems.

That's why they need someone from the start-up world -- both a systems guy/girl and a hacker guy/girl.

The reddit blog linked to in the TC article says the recently hired a new sys admin, which "doubles the size of our systems administration team."

Both are important. A lot of work of a reddit programmer is finding bugs that come about because of the unreliable disks and also finding scaling inflection points and fixing them before they take the site down.

This makes me shiver in my boots. I couldn't imagine having to run that site myself. Neil is inline for a big pay raise.

He is, but he's only been there a couple months.

He doesn't run it himself. Reddit has two sysops and Amazon AWS staff. I won't pass judgement on Neil's performance or compensation -- we shouldn't speculate, but it'd be surprising to see a developer of only 4-5 months get a big raise.

Supply and demand is an important factor. If CN demands the services of a developer with experience in the reddit codebase at reddit and the worldwide supply of developers able and willing to supply that need is only Neil, it wouldn't be unreasonable business to give him a raise, short tenure being somewhat irrelevant if that's the situation.

That's a bit unfair to jedberg, who actually runs the site with his operations team of two (one of them a new hire). This article and the comments are putting far too much weight on the developer, as if Reddit is a site that's constantly iterating new features. That's not to downplay his contribution, but:

    * He was hired in November.

    * He works on code, and didn't get woken up during
      the last major downtime until the end (from their post
      mortem, presumably due to the DB eating itself). The
      word 'even' in "we *even* woke up spladug" is telling,
      that indicates to me that he doesn't wake up often.

    * Reddit is pretty much a done product, and the last
      features I can think of were meta (Gold comes to mind)

He has a positive outlook[1], anyway, which the TechCrunch piece didn't seem to notice.

Doing operations for a site the size of Reddit with a staff of 2, on AWS, is far more impressive to me. Keeping an operation like Reddit afloat in front of that many uniques must be a pretty significant challenge. There's not a lot of depth for on-call, so I feel for their sleep.

On top of that, jedberg is the sole voice actively fighting for Reddit's reputation in public, since (let's be honest) the exodus looks a little suspect. Watching his comments[2] in the face of a perceivably sinking ship -- not to mention former employees who are saying quite a bit in front of an impressionable user base[3] -- is interesting, to say the least.

They're definitely undergoing a challenge, and I'm interested to see how they come out of it. (Aside: Today has apparently been discuss-Reddit day, looking at my threads.)

[1]: http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...

[2]: http://www.reddit.com/r/reddit.com/comments/g6d7i/three_form...

[3]: http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...

Sorry! I accidentally downvoted while intending to upvote. Tricksy fingers...

This is the second TC article today I see where they claim news site like digg or reddit are in competition with twitter. I don't think they understand that it's two very different type of stuff that is complementary. Or they are paid to say twitter in every article.

Tech journalism has a hard on for Twitter and it's not so surprising why. Twitter is all about self promoters. I see people using reddit all the time...I have never seen Twitter in the wild.

What happens when the last one leaves?

Is Conde just going to run off to Elance or something?

I can't even find a jobs page on Reddit.

At least it's open source.

Ironically, a month ago they were looking for devs and asking them to invest considerable time writing demo apps (front end engineers, for example, were tasked with writing an Javascript Reddit clone using HTML5 storage to keep track of votes, etc.) to prove their worth:

http://www.reddit.com/r/blog/comments/fjgit/reddit_is_doubli...

My guess is this may have repelled some experienced devs that, otherwise, might have applied.

Wow, sometimes companies are just out of their minds with the hoops they put up.

I would definitely not invest that much time into a job application, especially before an initial interview to know if I even want to work there.

This application is the initial interview. You now know that you don't want to work there. Saved both you and Reddit a lot of time.

From the application:

Well, first off, it's supposed to be fun. If you don't see the joy in either of these puzzles, please don't apply.

Seems fair to me.

Set high standards. You can always lower them later.

> You now know that you don't want to work there

If the filter were genuinely indicative of that, that would be great. However, my guess is that working there is quite different from doing makework puzzles for no pay (I hope, at least!).

I don't think you get a good idea of what a place is like until you sit down and talk to the people involved and see the code. At least that much is possible.

> If the filter were genuinely indicative of that, that would be great. However, my guess is that working there is quite different from doing makework puzzles for no pay (I hope, at least!).

The puzzles are in the links provided, why didn't you just go read them?

Because I have more interesting puzzles to work on: my own projects, many of which are available on the internet for potential clients or employers to look at should they wish to hire me.

Let me recap:

* You have the time to post on HN strongly implying that these "puzzles" are likely not that great (they're not even puzzles to start with)

* You have the time to purposefully misread replies (I suggested that you go read them before judging them, not that you go work on them) and reply yourself

* But you don't have the time to click on a link and read 20 lines of text?

I actually had a look before, and they confirmed my impression that they're pretty much a waste of my time.

The point is not whether I clicked through or not, the point is whether I'm going to spend my free time on my own projects, or completely useless projects for some company - they're not even things the company needs!

I'm more than happy to be judged on the merits of what I've produced, both proprietary and open source, rather than fooling around with invented puzzles.

If the tasks were things the company needed directly, they'd be accused of trying to get people to 'work for free'.

This is the wonderful thing about Homininae. I also have lots of my own work and at this very minute I have a ton of outstanding issues to close on the Faux framework alone.

Nevertheless, programming puzzles are like catnip to me, so I don't mind solving them as part of a job interview process (or at least, I didn't mind the last time I was actually looking).

I would NEVER say that I have a superior strategy to you, or that you have a superior strategy, or that programming puzzles are an excellent filter, or a terrible one. Just that some people will like them, and some not.

You don't, that much is obvious. Or at least, you don't in the context of a job application. Fair enough, and nobody should ever suggest that (a) This is a bad idea on your part, or (b) on the basis of your anecdotal evidence, this is a bad idea on Reddit's part.

That's exactly right. I did a similar quiz for my current job, and now I administer something similar when I look for new hires. It's amazing how many people with solid-looking resumes flub basic questions once you ask to see actual code they wrote.

The problem with this approach is that you never see the code of those not interested in jumping through your hoop.

I think its completely fine to have quizzes as part of the process but asking people to spend several hours on one before they´ve even talked to the company is a bit much IMO.

Ah, thanks. A while ago I was interviewed by the SpiderOak team, who asked me to make a FAQ system with commenting and CAPTCHAs in Django to prove I could write code. This would only take one or two hours, but I've already developed countless web apps in Django so I felt rather bad that they wanted me to pass some arbitrary test before even talking to me (as you say).

I thought it was just me being a primadonna, but I'm glad to find someone else thinking the same way. After all, my resume (which is full of all the things they ask) should be enough for at least an initial impression on their part.

They never did bother to reply after I sent them the code. I actually now see that they have deployed a new FAQ system with commenting and CAPTCHAs, although it was simple enough that I don't think they'd just use my code.

That's how you reach out to inexperienced devs. And frankly, if they're posting on Reddit, they know what they're getting into. They have to set up a pretty serious mediocrity filter.

If your resume identifies you as a badass programmer, you just toss them a copy of your resume. You know how it works, and they know that insisting that you do a programming challenge before setting up an interview hurts their chances of landing you, so they'll give you an in-interview test instead.

It's the same with Facebook and all the other places that have programming challenges -- they're there to qualify leads.

>That's how you reach out to inexperienced devs.

Not sure why an org like Reddit, which maintains a small team, would be interested in hiring inexperienced devs.

Nobody from the original team had any experience before they started there, to my knowledge.

...and now, one of the most popular sites on the internet has 6-hour outages.

Coincidence?

Outage, singular. You are talking about the 6 hour outage, not the 6 hour outages. Reddit is prone to go down more than some other high profile sites, but you are talking about a single occurrence of it going down due to a known issue that is explained in the article this thread links to.

No, it is not a coincidence. Please don't sensationalize a comment about an article that clearly explains what went wrong. Go to Reddit for that.

As someone who hangs around in the reddit dev and downtime IRC channels, several-hour outages do happen on a monthly basis, as well as loss of some features more e.g. commenting. Reading between the lines on each outage, it was due to building the site on shaky hardware foundations. It just wasn't explicitly mentioned until now.

http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...

Forgive me for the hyperbole, but I've been getting timeouts and extremely slow page loads from reddit daily for at least six months.

They always seem to have an excuse (usually about something being screwed up with Amazon), but frankly "not my fault" only works so often.

Yes. The outage was due to an Amazon EBS hardware failure.

Developing a brand new startup is a different challenge than maintaining a site that has a billion page views a month.

Also, they really need to weed people out. They were swamped with thousands of applications the first time around.

>My guess is this may have repelled some experienced devs that, otherwise, might have applied.

I told them as much too.

I thought their backend challenge looked reasonable, though: given a timestamped log file, return all entries since a certain time (efficiently!)

With a couple of caveats, iirc. There were a few idiosyncrasies with the log files that needed to be dealt with.

Reddit is definitely not a fun code base to go mucking around in. I wrote about this here: http://www.deserettechnology.com/journal/reddit-the-open-sou... . I think the amount of effort necessary to grok reddit's codebase would be pretty large, and I can't imagine many people doing that without getting paid or having a ton of extra time and a lot of love for reddit. The code needs a lot of work before it's ready for anything besides reddit.com.

My experience is that most people that try to get reddit running in the first place give up because it's such a gargantuan monstrosity. reddit will take any reasonably-sized personal VPS entirely over -- I was running on a 1GB slice and had to login every few days to restart Cassandra and paster (after the Oct. push). Reddit is just not well-suited for anything except reddit.com, and I think new developers would have some difficult maintaining that, and that most people aren't going to do it without getting paid. It's a big, ornery project, certainly not easy to jump into.

I think this is close to the truth. Even from the outside Reddit seems a tangle of functionality.

My guess is that there are leaks that have lingered for quite some time.

It is a fantastic site, but I have the feeling that leaving it has been a relief for those that have. I might be wrong. A bit drunk.

Lesswrong.com is built on reddit's codebase. I guess that's a hurray for Tricycle (http://tricycledevelopments.com/) beating it into submission.

OT but I found this amusing: competitor Digg (which has 42 employees and has raised over _$40_ in funding)

That's as many as four tens!

This is what happens when your primary demographic is people with no jobs and no money -- no ad revenue because nobody clicks the ads, and no possibility of selling subscriptions. Compare Reddit's target demographic with Bloomberg's target demographic. I bet Bloomberg has more than one developer.

(Incidentally, what a scam. Bloomberg terminals cost some insane monthly fee, and they have ads! And all I've ever seen anyone do with them is check sports scores and read celebrity gossip. Google should have put them out of business years ago.)

Bloomberg's annual revenue is about 25% of the entire journalism industry's.

It's not a money problem. We've got plenty of money. We just weren't allowed to spend it on developers until recently. Now we have a bunch of open developer positions that we are in the middle of hiring for, and will probably have more soon.

I'm sorry, but if you have money and you're not allowed to spend it on what you do need the most, then actually you don't have money and you've definitely got a money problem.

If you ever get the chance to sit on the shoulder of a Bloomberg power user I would recommend doing so.

In the right hands they are mind-bendingly powerful.

Also, a large part of the cost of the terminal is the data feeds.

In theory, if they can get by temporarily with just one engineer then the site was engineered well to begin with. There's some good management lessons from "The Mythical Man-Month" being applied here: http://en.wikipedia.org/wiki/The_Mythical_Man-Month

How many developers does hacker news have?

rtm and pg, plus a few mods as far as I can tell.

HN is a much, much simpler program, both in terms of features and internals. Internally, Reddit is a gigantic mess.

It also doesn't have a gazillion pageviews a month and daily downtime.

Isn't HN basically news.arc[1] plus some custom mods and caching with content/stories saved as flatfiles?

Reedit has many more sophisticated load issues to about. The two aren't really comparable it seems.

[1]: https://github.com/nex3/arc/raw/042990a55d8c880a0a88476eb52b...

Yeah, that was my point. Reddit is much more complex and much messier.

my bet is that there is some 'technical debt', so the devs leave when said 'business is over your tech details'! :)

This is not good given that they just had a recent 6 hour downtime. Why would folks at Conde let this happen ?

I think a large company such as conde nast has a problem with engineering talent retention -- hence the move to AWS maybe?

I know that at least one of the motivations for using AWS was that they didn't have to go through purchasing process of Conde Nast which was a huge roadblock.

I think that's actually one of AWS' biggest strengths. Once you get a corporate card in the account it's easy for the devs to buy what they need.

Conde really doesn't know what to do with reddit. They probably acquired it "knowing" they had to have something in "the social media space" (the acquisition predates the salesman's current buzzword of the hour, "the cloud") so they could compete in the "digital age" and all of that, but since they've acquired it my understanding is they've basically starved it and reddit has been running on fumes until they implemented Reddit Gold last summer. They're keeping it because they think they need something on this new-fangled internet stuff, but they don't what to do with it, and it's not generating much revenue, so they're just trying to leave things alone until they figure out something.

Note that this is just speculation based on what I've read, I don't have any inside knowledge.

Apparently, they have enough money but internal politics prevent them from hiring more people (it's a different budget...) Hence AWS, which isn't that cheap but does reduce personnel costs.

See the big reddit thread on this topic, or HN's thread about that reddit thread.

Wouldn't it be surreal if Kevin Rose went to Reddit?

Interesting. This sure is boosting their pageviews/developer ratio.

Which is currently sitting at 1.1B : 1.

Hmm.. Would now be a good time to build a new reddit?

Building a new reddit is easy - building a community, and scaling to support 1.1 billion views a month is not.

Does this mean that the social new bubble has finaly burst with reddit down to one person and digg winding down?

HN seems to be doing fine with just one awesome developer :-)

Yeah, but look who he is.

Not accurate.

HN has been struggling to keep up with load recently. Remember the pagination of comments a few weeks back needed to keep the server from falling over? Yeah. I think news.arc/HN could really do with some extra scalability work; it still slows down significantly at peak times.

Server? There's one server?

Yes one server, but that isn't entirely fair because pg has Robert Morris to tweak it.

And he is a kick ass sysadmin.

Scrap it. Rewrite. It can't be that complicated and if it is, it needs to go.

Rarely a good answer. All of that complexity wasn't added for fun. Making a comment like this really shows ignorance in software development. While we all want to scrap and rewrite some projects, most of us know that it isn't as easy as it looks and you will be spending your time on the wrong thing.

Also, I don't have any idea what to make of your comment "...and if it is, it needs to go.". Are you just trolling or do you really suggest that they shut the site down because it's complex?

totally agree.

I think he's suggesting that if any part of the reddit codebase is too complex, then it needs to be ripped out.

A sizable portion of reddit's recent success is owed to Digg's decision to do just this.

It appears that Reddit's days are numbered. Sure they may be able to get by with one developer for a while, the product is fairly mature and content is driven by the user base. But how long can a website remain stagnant, with no innovation until the user base starts to migrate to the next best thing? Also, being owned by a large, old media company doesn't necessarily give one much confidence that Reddit is backed by visionaries who are willing to push the envelope and take risks.

And what is the next best thing in your opinion? It's one of the few places that does nested comments properly and the lightweight up/down voting always keeps it interesting. The only thing I can see replacing it is an outright clone, but no one's stepped up and done it in all the years Reddit has been around.

>And what is the next best thing in your opinion?

I'll know it when I see it.

I was a daily user of Slashdot from 2001 to 2006 and then I found Reddit. I completely abandoned Slashdot less than a year later.

What could beat reddit? Well, if there was a facebook-y site which allowed me to associate with people who had my interests (rather than family, friends, and people I went to college/high school), I'd be down with that. (I love my mom, but I don't think she cares about Minecraft, old-school RPGs, French pop songs, Asa Akira, and Jules Verne novels.)

Twitter is close, but the character limit, url shorteners, and "twitter-speak" (intentional bad spelling/grammar in order to "get under the limit") just make my teeth itch.

Reply on news.ycombinator.com