Why Reddit switched to Python (2005) | Hacker News Reader

Why Reddit switched to Python (2005)

blog.reddit.com

118 pointsryutlis42 comments

When it comes to programming languages/framework debate, there are a few interesting tidbits:

1) Geeks _love_ to debate languages, especially one they _know better_

2) Geeks also love to trash languages that they don't know that well

3) There's always the claim of N-increase of productivity without any scientific backing (yet we always stress out that we should measure everything...seems like a paradox eh?)

4) This is how we do politics: debating languages, tools, frameworks.

Use whatever you like. That's the bottom line.

Joel used ASP.NET for FogBugz, ASP.NET MVC w/o ORM for SO.

LinkedIN, TripAdvisor, Jive, some Google properties are using Java.

Facebook, Flickr, WikiPedia use PHP.

You are far more productive using the tools you know.

If someone just picked up Node.JS (and a novice JavaScript developer) yesterday and claimed he/she got 10x productivity boost, ask him/her to show how they go about and code the solution a'la Notch (live-stream how you code or from pre-recorded coding session).

Building products require more than just a syntax. Libraries, Tools: build tools, packaging tools, deployment tools, documentation tools, etc are all as important as the programming language itself.

It's about time we ask for proofs, not just a blog post claim.

> Joel used ASP.NET for FogBugz

To be pedantic, that's not true. As I understand it, FogBugz was originally written in VBScript (Classic ASP) and then ported to Wasabi, FogCreek's own compiler. Wasabi then spat out PHP and VBScript.

http://www.joelonsoftware.com/items/2006/09/01b.html

However, I still agree with your point and this probably makes it an even stronger argument.

Doh, you are correct. How did I forget about Wasabi! It created a bit of controversy too back then when he criticized rails.

I came back to this PG essay on identity a couple of days ago. The essay discusses the points you mentioned and a compelling explanation for them: http://www.paulgraham.com/identity.html

> some Google properties are using Java.

Google is mostly a Java shop, IIRC. They even write HTML and JavaScript in Java, a.k.a GWT.

They also have plenty of C++ and Python. Small projects will always be allowed to run whatever language they fancy, but the largest and most critical codebases at Google are definitely in java, c++ and python.

They don't use GWT for many of their big properties though.

> 2) Geeks also love to trash languages that they don't know that well

Usually, fresh (and valid) criticism can only be found when someone who knows a language abandons it in favor of another. Some of my criticism of Java and C# would, probably, refer to features and techniques that are outdated. However, other aspects would remain as they were (tendency to depend on overly complicated tooling, verbose syntax, relatively low productivity - here we are, headcount-wise, mostly a Java shop while delivered-product-wise a much more even split with most Java projects being delivered late) and my criticism should still be valid.

You're right. Some developers who moved away from Java during the heyday of EJB1.0/2.0 and Struts1.0 definitely had valid criticisms back in 2004/2005 (the war between Java ecosystems and Rails).

Fast forward to 2010-2012, the criticisms themselves had become outdated.

Not all of them. Java and C# tooling is still horribly complicated, both languages are overly verbose and building any significant app with either one still requires tons of boilerplate code (usually generated by said overy complicated tools. Productivity is also much lower (here) when compared to teams using JavaScript or Python.

Weird. Maybe it's a matter of taste but I find each tools have the same level of "complication".

Having said that, I do want to ask you to refrain yourself from commenting non-issue such as the IDE + Static language and LoC due to formatting debate because it is heavily a matter of preference in which I think it would definitely be a waste of my and your time to discuss further.

Side note: I don't think Eclipse is any more complex than Vim to master (unless of course your developers use Notepad) and Maven to be... more stable and less of a moving target compare to similar tools in other environments.

I don't write much boilerplate code these days because I use Spring-Data and other modern frameworks (JAX-RS, JAX-WS). So I'm still confused with many people keep saying "tons of boilerplate code". Can you quantify these "tons"? 2 extra lines? 10 extra lines? what is the context/situation? Can you explain with some examples so I can see your pain point?

Gone are the day of boilerplate XMLs as well since these days frameworks are moving toward annotations heavy.

Many situations depend on the context: we built a moderately complicated "portal" that communicates with 4 different data sources (2 Web Services, 2 scheduled DB dumps) using Java in about 6 months. The software has to be deployed in 4 different environments (LOCAL, DEV, UAT, PROD). Bending Maven to meet certain build/packaging requirement was simple.

Maintenance has been a breeze so far: no downtime (we use the latest GlassFish), once the DB locked (but that was the DB), and once JPA/Hibernate bailed on us due to the size of the data we pulled (we skimmed the data down a bit and the issue is gone). DB migration was a walk in the park using Flyway.

After going through Rails, Python, Java and lately C#, my mindset definitely has changed when it comes to the holy-grail of productivity debate: they all suck with different level of problems.

Things are a bit different now. SBCL works great nearly everywhere, Clozure CL does too, and there are more libraries that are easier to install than ever before.

As someone who is really, really new to Lisp (worked through the Little Schemer, now in Seibel's "PCL", just started learning about this QuickLisp thing that you might have heard about... ;)), I've been wondering about the whys and wherefores of the Reddit switch.

Do you think that if Reddit developers were faced with the same decision today, that the progress you mention might indeed push the pendulum the other way?

I don't know. There are still a lot of things for which CL has no well-tested, widely-used, out-of-the-box solution, and someone looking for a reason to use another system can pretty easily find it.

The situation is improving, though, and will continue to improve as long as people scratch CL itches instead of switching to something else.

I agree Zach, it is a different world now. This really hit me in the article: "my choices of threaded Lisp implementations was limited to OpenMCL, and in FreeBSD it's CMUCL". As another poster mentioned, SBCL and Clozure run on many platforms providing multithreading. Obviously, your QuickLisp package manager now helps find and install useful libraries: I used to install all dependencies into each of my Common Lisp projects, which I don't do anymore.

I also wonder if the Reddit team's decision might have been Clojure if Clojure had been mature when they developed Reddit. Clojure is roughly 1/3 as fast as Common Lisp and takes just 2 or 3 times the memory for a typical application (my experience) - a good alternative depending on the application.

Yes, but the Python eco-system didn't remain standing in one place either. Lots more libraries, more stable and feature-full releases, and great support for bindings for well known libraries/frameworks/APIs all across the board, as Python has clearly emerged as the favorite hacker language for general purpose programming.

I love Python, and have noticed that its popularity is trending upwards, but I'm hesitant to upvote any thread that begins an assertion with "clearly".

I wish. As mainstream as it got, it's way less deployed than java (which is used for literally everything, including mobile development), PHP, C/C++, Perl and even Ruby. It's starting to be fairly ubiquitous, thanks to hacks like jython and ironpython, but it's nowhere near being considered the "clear favourite" for anything.

Each of PHP, Perl and Ruby has their niche(s), but excluding those Python is far more popular. And the comparison with Java/C is irrelevant here since this is a different domain of languages (dynamic) we're talking about here.

You didn't say "the favorite hacker dynamic language" though :)

I'm a huge Python fan, but I'm also a realist: the Python ecosystem (in terms of vendors, products, customers and professionals involved, as well as libraries and tools) is nowhere near the mainstream Java/C++/whatever-Microsoft-is-pushing-this-year. Python is not the language of choice on any mobile platform, for example; it barely registers in the humongous "enterprise" space; SDKs for most hardware devices (or anything else, really) will usually list Python as the third or fourth choice, if at all. It's doing reasonably well in scripting, automation, Linux administration, 3D and web, but that's a far cry from being a clear favorite for "general-purpose programming".

"...On my Mac, my choices of threaded Lisp implementations was limited to OpenMCL, and in FreeBSD it's CMUCL. Because of the low-level socket and threading code we had to write, reddit would not run on my Mac, and I was always tethered to our FreeBSD development server. Not being able to program offline is a pain."

Interesting choice; the reddit guy gives up Lisp for python rather than Mac desktop for a BSD/Linux desktop. I would rather do the reverse; would move where Lisp works great on client and server.

> After spending the entire summer working entirely in Lisp, it's nearly impossible to work in another language and not say to myself, "If only this were Lisp..." Lisp has many goodies that make programming in it a joy: the REPL, macros and the lack of syntax are some. I won't go into the details, but rest assured, it's cool. People become Lisp zealots for a reason.

I honestly don't understand this sentiment. For me it's nearly impossible to work in lisp and not think "If only this were some other language..." The baseline functionality I'm used to from other languages, like even type checking, is absent. It's like writing in assembly just for the degree of control you have as compared to managed code. I mean yeah writing something in assembly can be fun in its own way, but the result will almost certainly be awful, and you'll solve problems that have built-in solutions in other languages.

I guess you code in ML/OCaml/Haskell the rest of the time.

Common Lisp has type checking.

Come on, we all know what he meant.

I don't really know what he meant, with the references to managed code and assembly language in relation to Lisp, but what he actually wrote is wrong when it comes to Common Lisp.

Perhaps he's referring to the batteries included feel of other platforms. Every other platform (i.e. .NET/java/ruby/python) have gobs of built-in functionality in the standard library - everything from parsing, to different I/O models (NIO in java), to frameworks (WSGI, WS consumption and production, etc). Lisps diverge fundamentally (e.g., the threading model) if it isn't specified in CLtL or ANSI CL. CL seem hamstrung by those documents.

Concrete Example: 7 years ago I couldn't (using available libraries) reliably parse in lisp a DB dump in XML (~8GB), select some records based on a set of calculations, and convert it to CSV. Most libraries insisted on needing the whole document in memory, some libraries leaked like a sieve and caused heap exhaustion, others couldn't handle UTF8. In the end, I could do it python (couple hours to develop and took 45-60 minutes to run), I could do it in java (several hours to develop and ran in ~8min), but I couldn't do it in LISP without writing a custom XML parser or delving into FFI. Maybe it was an abnormal case, maybe not.

I miss macros and the expressiveness; I miss the community; I don't miss the insanity.

Well, it's not like that any more. There are portable ways to work with threads, there are multiple XML libraries, some of which do not read the whole document in memory, and all implementations support UTF-8.

These things happened because some people decided to improve CL (both implementations and libraries) instead of using something else.

It may be worth retrying as an experiment to see how lisp fairs now.

You should always pick the right language for the job. You could have written the right language in Lisp or you could pick a good enough language to solve the problem.

Good enough is good enough.

Stop thinking about writing programs and start thinking about building a language to write your program in. That's Lisp.

If you approach Lisp the same way you'd approach C or Python, you'll fail.

This is an example of the Lisp Curse.

http://www.winestockwebdesign.com/Essays/Lisp_Curse.html

This is the reason why the Clojure community is taking libraries very, very seriously.

"Translated very easily" suggests they didn't create a domain-specific language for their problem, which to me seems to be missing the point of Lisp. The exotic features of Lisp and Python don't overlap much at all, so the intersection of the two isn't a particularly powerful language. They probably could have ported to any Blub just as readily.

Why, maybe there are problems that do not require a domain-specific language! Doing something like reddit.com probably require a simple and readable codebase that can be extended and optimized easily. It is not likely that the domain is so complex that it needs macros and all the "clever" stuff.

I like how they refer to reddit as a "FreeBSD box." My guess is that it's something much, much larger now.

He also says "the reddit code itself is not incredibly complex," which I imagine is less true now.

There’s no need to leave it to imagination: https://github.com/reddit/reddit

Right, but I don't know how complex it was at that time in the past.

Does EC2 offer FreeBSD?

Not officially.

cperciva (HN username), FreeBSD security guy, maintains some AMIs for it: http://www.daemonology.net/freebsd-on-ec2/

Yes, it is different. At the time he wrote that, there was literally one hosted FreeBSD box running all of reddit. It was called... 1.reddit.com. Eventually the database had to be moved to a separate machine called... 2.reddit.com.

reddit now runs on a few hundred Ubuntu EC2 instances.

If there are any Lispers who lament the Lisp Curse, or some idiosyncracies of your favorite dialect of Lisp, and you have an open mind to give something new a shot, consider checking out Clojure.

I don't want to become one of those diehard __-advocates, but I'm pretty impressed with Clojure. Speaking from experience, I think it solves quite a few of the pitfalls mentioned in the OP. I wrote up an extensive post on how to learn Clojure, and veteran Lispers can jump to the end and catch the links of Clojure for Lispers:

http://www.elangocheran.com/blog/2012/03/the-newbies-guide-l...

Reply on news.ycombinator.com