HNHacker News
TopNewBestAskShowJobs

DavidMcLaughlin

932 karma · joined December 7, 2008

Software developer @ Twitter david @ dmclaughlin.com

blog: http://www.dmclaughlin.com

submissionscomments
DavidMcLaughlin··on Lucene's FuzzyQuery is 100 times faster in 4.0
Are you saying you wrote an alternative to Lucene because users don't like to type in search queries that Lucene understands? Lucene is a low-level library for implementing a search experience. If you don't understand that then I don't think you understand how a sophisticated search engine like Google works.

Say you type in:

"Lucene equivalent in C++ or C#"

This query is not passed as-is to some magic implementation of a Search Engine by Google.

First they need to parse this query. This involves a tokenizer step which will have many rules depending on the context and the language requirements, etc. An industrial strength tokenizer is a significant undertaking in itself.

The next step is turn this user query into something that a search engine understands. Unless your default mode of operation is "match this user query EXACTLY" then that means having some kind of query language. For any search engine, this will end up looking like the syntax you have a problem with in your post.

Next, to have any kind of useful and competitive search experience you will need to take the user query and generates tens, maybe even hundreds or thousands, of different sub-queries that try to explore different estimates of user intent and fuzzy-matching. For example, you might do some analysis and realise that equivalent in this context is not very important and decide to generate another twenty queries with all-known synonyms for the word equivalent. You might decide to try the query with certain words dropped altogether.

And of course, you also need to be able to rank results. Not just within sub-queries but when merging all the results from sub-queries together. This takes sophisticated and highly customised scoring algorithms that will be dependent on context and a whole lot of other things. Essentially you need to guess what is the best match based on what you think the user wants.

Finally, to be an industry leader like Google you will use a data-driven approach to scoring that makes use of machine-learning techniques to keep your advantage more than a purely technology one.

For companies that just want a basic search that will behave intuitively for people used to the Google model, Lucene helps with A LOT of this. Their default query parser, scorer and other vital components are good enough that most websites can just use the default configurations out of the box and get a good search experience.

All that you're left with in the end is to create the basic user interface abstraction - the "one box" experience. Solr helps enormously with this, but in the end all the functionality it exposes is Lucene.

For more complicated search experiences, there isn't much that I described in the above approach that you can't do in Lucene.

So yeah, Lucene is awesome.

DavidMcLaughlin··on The myth of the Lisp genius
> In the context of javascript, "unobtrusive" means it is designed to degrade peacefully

Nitpicking here, but that is actually graceful degradation/progressive enhancement.

Unobtrusive JavaScript is about writing JavaScript code that plays well with others. The example you gave of not having inline JavaScript in your presentation layer (HTML) is one example. Not writing JS code in the global namespace and avoiding monkey-patching built-in objects are other examples.

DavidMcLaughlin··on The Dark Side of Hadoop
I work at Nokia and I'm not aware of any big production uses of it inside Nokia. Most of the teams in my part of the company use Hadoop.

Oh well.

DavidMcLaughlin··on Hacking The Status Game
I guess this could be good advice for manipulating status for short time periods (dating, job interviews, business meetings, etc.), but fortunately once it goes beyond that (relationships, careers, business deals, etc.) the status of everyone involved will settle down to the natural order of things.
DavidMcLaughlin··on [dead]
I've flagged this article, but I just have to post to say - this is one of the worst contributions I've ever seen on the front page of Hacker News. For all of the value of the advice and content, it may as well be named "Top 100 funniest domain name mistakes!" and forwarded on to me by my uncle.
DavidMcLaughlin··on Secret Fears of the Super-Rich
What an interesting thing to take away from the quote, maybe I assume too much because I've read the whole speech but certainly I did not think he meant to say what you are implying.

What I took from it personally is that people tend to find meaning in life from something - a deity, a job, money, fame, looks, their own business - and that it doesn't really which of those you choose you're essentially taking a leap of faith that it will deliver happiness.

DavidMcLaughlin··on Secret Fears of the Super-Rich
I don't know what's more disappointing, the fact that your original comment, which added absolutely nothing of value, was voted up or that this explanation has even more votes.

All it takes is a simple Google search to show that the author of the quote uses worship quite routinely in one of the word's many "widely accepted" meanings:

http://dictionary.reference.com/browse/worship

The only contentious thing he has done is draw parallels to atheism, the definition of which is not so sensitive to context. The merits of doing so I think would have been a good talking point, but instead all you contributed was "a stupid nonsense."

DavidMcLaughlin··on Secret Fears of the Super-Rich
"In the day-to-day trenches of adult life, there is no such thing as atheism. There is no such thing as not worshipping. Everybody worships. The only choice we get is what to worship. And an outstanding reason for choosing some sort of god or spiritual-type thing to worship - be it JC or Allah, be it Yahweh or the Wiccan mother-goddess or the Four Noble Truths or some infrangible set of ethical principles - is that pretty much anything else you worship will eat you alive. If you worship money and things - if they are where you tap real meaning in life - then you will never have enough. Never feel you have enough. It's the truth. Worship your own body and beauty and sexual allure and you will always feel ugly, and when time and age start showing, you will die a million deaths before they finally plant you. On one level, we all know this stuff already - it's been codified as myths, proverbs, clichés, bromides, epigrams, parables: the skeleton of every great story. The trick is keeping the truth up front in daily consciousness. Worship power - you will feel weak and afraid, and you will need ever more power over others to keep the fear at bay. Worship your intellect, being seen as smart - you will end up feeling stupid, a fraud, always on the verge of being found out.

The insidious thing about these forms of worship is not that they're evil or sinful; it is that they are unconscious. They are default settings. They're the kind of worship you just gradually slip into, day after day, getting more and more selective about what you see and how you measure value without ever being fully aware that that's what you're doing."

http://www.guardian.co.uk/books/2008/sep/20/fiction

DavidMcLaughlin··on Engineering management at Facebook
You can say shit here.
DavidMcLaughlin··on Moving to QT Quick, Nokia has won the developers battle?
> That they have abandoned MeeGo was probably a good idea

Nokia didn't abandon MeeGo, they just unified their developer offering because the feedback was that the current 'developer story' was far too complicated. As a Nokia employee, I think it was a step in the right direction but we have not 'won' anything yet.

DavidMcLaughlin··on How to Make Visa Obey Your Every Desire: The Credit Card Concierge Experiment
I'm not really sure what generalisations like yours bring to the table. I mean I've only been to maybe 5 or 6 states in the US, but I know that a city in Maine is so completely different from somewhere like New York or Boston which themselves are completely different from, say, Orlando or Las Vegas. So trying to make generalisations that hold for all 50 states in the US is even more ridiculous.

I guess an analogy is that Turkey is in Europe, and being a European it would be very easy for me to say "the service in the US is way superior to the service I get in Europe" and then I could add "but at least in Europe I don't get someone telling me how much I should tip when I get the bill like I did in America." But the truth is that the tipping culture is nowhere in America that I've visited like it is in New York and the standard attitude is actually very similar to the UK, which itself is way different to somewhere like Italy or Germany. But hey, let's not let the truth get in the way of a good discussion that is rooted in anecdotal evidence.

I guess what I'm trying to say is that the service you receive is based on a lot of things, the least of which is the country you are in. In fact the most important part of how well you judge the service you receive is you and your expectations. That is something I realised from similar 'anecdotal discussions' with my fellow Europeans when I rave about the service I receive in the US - "they would never let your beer glass go empty for an hour like they do here!" - "yet but they also come over to interrupt you every 5 minutes to ask you if everything is okay.. when I need something, I will ask for it!"

DavidMcLaughlin··on Google Maps Instant (inspired by Youtube Instant)
Actually, to me it seems like the real lesson most companies are picking up from Apple is to grossly overstate every little thing they do in the name of hype and buzz.

I mean, revolutionary feature? Seriously? My whole point is this revolution happened five years ago and people learnt their lesson and moved on. I'm sure a lot of developers will be asked to put this back in, but on the plus side if you're not dealing with Google scale it's a trivial thing to implement.

DavidMcLaughlin··on Google Maps Instant (inspired by Youtube Instant)
The funny thing about the "instant" search is around the time of AJAX exploding four or five years ago, this type of "real-time" search was all the rage. Over time it disappeared, most likely because it both wastes resources and a lot of users found it annoying. From that, auto-complete suggestions was the next evolution. Now we seem to have gone full circle. As impressive as it is that they're handling way more searches than before with lightning speed, the whole thing seems really naff. The marketing video for Google Instant in particular is hilarious - "I don't even have to press Enter!"

I work for Nokia and we used to have "real-time search" on Ovi Maps and this was removed around a year ago, again due to a confusing and poor user experience. After all this buzz I saw some people on the internal communication tools ask if we have the technology internally to do instance geosearch.

Marketing is everything.

DavidMcLaughlin··on Finally, Hacker Monthly Subscriptions
The border between Ireland and Nothern Ireland?
DavidMcLaughlin··on Nokia doesn't get it - The Truth about N97 [video]
What's with the Nokia doesn't get it part? The market repositioning from smart-phone to mid-range consumer phone and delayed launch of the N8 would seem to suggest that Nokia definitely does get that the N97 was a disaster.

http://en.wikipedia.org/wiki/Nokia_N8

Disclaimer: I work for Nokia.

DavidMcLaughlin··on Ask HN: Share a gem. Teach me and you.
I picked this one up on Hacker News.

http://news.ycombinator.com/item?id=1398805 http://en.wikipedia.org/wiki/Ego_depletion

The title says it all. Self-control is exhaustible. So when you're trying to make changes in your life, keep it to one simple thing at a time. Over time the one thing you decided to change stops being self-control and starts becoming a habit. Once it becomes a habit? Start training something else. You'll rack up positive changes in no time.

DavidMcLaughlin··on The next wave of programming languages
I found going through "Introduction to Algorithms" and porting the algorithms in the book to <new language> a great way to get used to a new language.
DavidMcLaughlin··on An introduction to ZeroMQ
Why do you need both ZeroMQ and Apache Zookeeper for service discovery and coordination? Why not just Zookeeper?
DavidMcLaughlin··on The Hacker News Experiment
This would be really interesting, especially for those of us who are struggling to meet that special someone (i.e. a co-founder). For me at least, I think being part of a competition and having external interest to keep motivation up would be helpful.
DavidMcLaughlin··on That Sounds Smart
Right. I like the sentiment of the post but it's definitely not a black and white case of:

1) You are stupid if you are hard to understand 2) You are smart if you are easy to understand

They say the best teachers and coaches (i.e. the ones which are easiest to understand) are the ones who had to work the hardest to find success in their craft. Conversely, the most naturally gifted of people sometimes don't understand how amazing what they take for granted is and can't pass on this knowledge. There is also a lot of real world contradictions to this (I'm thinking sports here) so yes, there are a lot of shades of grey.

DavidMcLaughlin··on Quora: "How does a penniless entrepreneur attract gold diggers?"
What you are describing is the concept of "Male gaze" in feminist theory. As common as this phenomenon is in general, it's especially rife in tech/geek circles in my experience.
DavidMcLaughlin··on Beginner's Health and Fitness Guide
It actually takes a significant amount of "functional" strength to do even just basic Yoga poses. The common myth is that Yoga is all about flexibility, but in fact a lot of the time "inflexibility" is actually just weak core strength.

Having a good core is one of the easiest ways to maintain your health as you get older too and if you're active you will feel a great benefit having that balance and strength. Until I tried Snowboarding I had no idea my core was as weak as it was.

The test I always like to give myself is being able to put my socks on in the morning whilst standing up, without wobbling all over the place or leaning on anything :)

DavidMcLaughlin··on The Economics of Software
> Funny how the most upvoted comment here has pretty much nothing to do with the premise of the article. This community is going down hill rapidly...

Firstly, the comment you refer to IS relevant because the article explicitly states that PosgreSQL cannot compete with Oracle, et al in the enterprise. I mean it's right there in the conclusion. Maybe it needs more info to back it up, but then so does the article itself when it makes its point.

Secondly, the reason I would wager on it being the highest rated comment is because the greatest value of HN comments is when you finish reading one of these articles and are quite taken in by the premise, many times (for me personally) right there at the top of the comment page is someone with a different experience reminding you not be so impressionable. In my time here this quality of HN comments has been consistent. In this case I would say patio11's fleshed out response is more useful (edit: and already it has overtaken the parent in votes - I would hold off any emergency quality control measures for the time being) but the cream usually rises to the top.

> I wonder if this type of degradation is inevitable when the masses invade

When I read a reaction like this to someone's comment, I always find it interesting to click on the user profiles and see when both parties joined. In every case I've done it, the person making a comment about "the masses" invading has been here a fraction of the time of the person they're referring to. In this case, your 111 days is barely more than a quarter of the time forinti has been registered.

DavidMcLaughlin··on There is no such thing as good hiring, only good firing
> This works incredibly well. At my last company I fired about a third of the engineers I hired after 3 to 6 months

Wow. I wonder, have you and this smug author ever considered that you are the problem?

Interviewing is a two-way process, it's the candidate's job to back up everything that's on their resume and make sure they present an honest view of their ability but it's also your job to decide if someone is a good fit for your team before making them quit their comfortable job and move to your start-up. By firing a THIRD of all people you convince to join your company, you are both incompetent and irresponsible. Precisely the kind of harmful deadwood people enjoy see getting fired.

DavidMcLaughlin··on Are you one of the 10% of programmers who can write a binary search?
> One of the reasons I’d like to see a copy of the Second Edition is to see whether this passage has changed

For the author: it hasn't changed. Still only 10%.

I actually had to write this binary search algorithm in my technical competency interview for my current job. Rather than hours, I had minutes and the extra gotcha was that it was unsorted data so I had to write quicksort first. I remember feeling excited that I was going to be working for a company that had such low level issues but eight months in it's still the only time I've written my own sort or search methods since University :)

DavidMcLaughlin··on Do people subscribe to blogs less now? My blog's numbers.
In the last two years I have went from having over 500 RSS feeds in my Firefox toolbar from various bloggers to narrowing that down to maybe 50 community hub type blogs like techmeme/DIGG/Reddit and then eventually I had one RSS feed which was HN. I find the comments to be invaluable so even now it's inconvenient to read HN stories through the RSS feed, so now I don't use RSS at all.

I think the general principle behind feeds was always content discovery rather than loyalty to a particular author. Blogs were so effective because the likes of Coding Horror didn't just post some insight into programming but generally linked to lots of other really interesting posts that could educate and entertain for as much time as you had to burn. The only problem was if you wanted to do this every day, then it was hit and miss and your reading list for the day was too dependent on a good writer having something to say when you wanted entertained (hence the volume of my subscriptions).

Aggregation sites like Reddit and Digg were the next stage in content discovery but the noise ratio was far too high to replace my RSS feeds. Then the niche sites like Proggit and Hacker News pretty much eliminated my need for RSS. Whereas previously I could spend an hour reading my personal feeds and come away feeling like I wasted an hour, now I feel like an hour on HN will not only keep me aware of the emerging trends in my profession but also give me lots of interesting anecdotes from the general interest articles that are posted here. It's an invaluable source of information.

Content discovery is a really exciting space, and I'm really excited to see how it will evolve. Twitter is another evolution of this I think, and I think way more people use twitter as content discovery or a news source as opposed to "what I am doing right now." But following one of these loudmouths who retweets and links to articles suffers the same problems as RSS in that there is this tendency for one of these people to make a huge amount of tweets due to some event like a conference (or gets drunk one night) and then some stuff you don't care about pushes a lot of the stuff you do care about off your page. I only follow 25 people on twitter and if I don't check it for a day then I have 200+ tweets to dig through, so I have no idea how people with massive lists make their twitter work for them.

The general problem from all existing solutions I see is that they are too immediate. Aggregation sites work on a day to day basis where time of upvote plays more part than overall quality. Twitter goes out of its way to make anything other than your latest tweets hard to access. And right there at the side you find emerging trends. It's all about now, now, now. But generally I find information on the here and now pretty easy to solve, it's those timeless classics that appeared on HN front page eighteen months ago or were retweeted by 100 developers last week when I was on vacation that I want to be able to always have access to. Especially on the days where the whole HN front page is dominated by some topic that I find mundane (Joe Blogger's latest opinion on the iPad!).

I hope the next big iteration in content discovery will try to solve that problem.

DavidMcLaughlin··on Life at Google
I do not know the author so I could be wrong, but I'm pretty sure this is not bravado or bragging, but rather a critique on a culture that encourages you to tip the scales towards work in a work-life balance. This line says a lot to me:

> But, if you like your work, it’s sure easy to get through a whole lot each day

DavidMcLaughlin··on Issue 3422 - android - Animated GIF not working in browser
Comment #45:

According to the Android developer in comment 42 the explanation is that Android has multitasking while the iPhone doesn't.

DavidMcLaughlin··on Sqlphobia: The Irrational Fear of SQL
> Yet somehow most developer got over "what do you mean you have to learn about algorithms and data structures before you can actually program?"

I respectfully disagree.

DavidMcLaughlin··on Brainwashed: How to Reinvent Yourself (Seth Godin)
So many of his points are patently false. I wonder if the author is aware of the bubble in which he lives.

In fact, when trying to get through this manifesto I was constantly reminded of this Bill Watterson quote in his Kenyon speech:

Creating a life that reflects your values and satisfies your soul is a rare achievement. In a culture that relentlessly promotes avarice and excess as the good life, a person happy doing his own work is usually considered an eccentric, if not a subversive. Ambition is only understood if it's to rise to the top of some imaginary ladder of success. Someone who takes an undemanding job because it affords him the time to pursue other interests and activities is considered a flake. A person who abandons a career in order to stay home and raise children is considered not to be living up to his potential-as if a job title and salary are the sole measure of human worth. You'll be told in a hundred ways, some subtle and some not, to keep climbing, and never be satisfied with where you are, who you are, and what you're doing. There are a million ways to sell yourself out, and I guarantee you'll hear about them.

← PreviousPage 2 of 3Next →