Hacker News Reader

Comment by jhy | Hacker News Reader

I dunno, I think there are plenty of other areas for some competitive CompSci.

As a random idea: Oracle puts up $1MM to increase their query optimiser by 10%. Plenty of test data there, easy to measure performance gain through a decrease in CPU / disk IO. And lots of gnarly edge cases. The upshot to the economy in saved cycles could be huge.

Or a transcontinental shipping company could run a competition to improve their prop design, or engine design, or their hull design. Fluid dynamics is a challenging, computationally intensive field. And any percent point improvement on these makes real savings.

Or a city might put some real smarts behind their traffic light system, and intelligently improve traffic flow. Are they much more than day-parted timers and queue counters today?

I think there's plenty of room for other competitions / grants like this :)

View on HN

Testing an improved Oracle query optimizer would require terabytes of different databases in order to have a representative sample of real customer databases. That's not really practical today.

If a contest participant comes up with a proposed design improvement for a cargo ship prop, how do you actually test that? Testing in a CFD modeling tool is one thing, but to really be sure you have to build it and put it on a ship.

For traffic engineering, testing is the real problem again. Who is going to write an accurate simulation of an entire city's traffic flow so that contestants can use it to test their light timing algorithms?

By contrast the Netflix prize involves a small data set and doesn't require building anything in the real world.

First, I don't think that most companies have even harvested the low-hanging fruit in their data. They need to do it before they worry about optimizing it.

Second, there really are a lot of barriers to this. Your idea about improving prop design is cool, but it requires more resources than a few buddies can assemble: huge computational horsepower, as you note, as well as the ability to actually fabricate and test the props.

Liability concerns will squash much of the innovation. Your traffic flow question is really interesting. But I don't think I'll be entering if there's any chance that some lawyer is going to add me to the list of defendants when someone gets killed by a driver running a red light.

In some areas the government outright forbids private development. (I don't mean to turn this into a gun control debate; for good or bad, this is just an example) The USA used to be the world's innovator in firearms design. People like John Moses Browning are part of the reason that our military became the class of the world. But starting with the 1938 NFR, taxes and regulations made it impractical for private individuals. The result is that now improvements are really only the result of direct government funding (e.g., Armalite and the AR-15), or occur overseas, like in Israel. Similarly, I imagine that any biological or pharma work would wind up with the FBI at your door (assuming that you could get by without million-dollar hardware anway).

So, you definitely have a better imagination than I for possible places to use this development model. That said, the areas you've proposed, as others have mentioned, are bigger problems to solve, sometimes much bigger. They require a lot more capital-intensive pieces to come together, or they require accurate models that don't necessarily exist.

Of course, that didn't stop companies from competing for the X-Prize. So, perhaps in the areas you suggest the prize just needs to be bigger (maybe a lot bigger). So, while an improved query optimiser for Oracle might require hundreds of thousands of dollars worth of hardware to develop, a $10 million prize would still be sufficient to interest some teams. At some point, the line is crossed between what is a good development+marketing investment, and what the company could do with its own in-house developers. I don't know if $10 million is past that line, or if it's enough incentive to bring out the geniuses in the field (the ones that don't already work for Oracle).

Oracle would also have the problem that they'd have to create some sort of stripped version of Oracle that only works on this one data set, or something, because otherwise they'd be giving away the biggest license of their software to any developer that signed up (because to test major improvements to Oracle, in the places that matter to Oracle, you need to be running it on big MP machinery with terabytes of data).

So, maybe that's why more companies haven't done it, despite the apparent cost-effectiveness and marketing value (NetFlix has probably gotten a million bucks worth of tech press out of this contest). The cost of getting people on board might be much higher, and the cost of providing the tools to contestants may also be dramatically higher. There may also be negative media consequences. NetFlix spun it beautifully...such that we think of them as very smart scrappy guys and hard to beat, but also eager for a challenge from upstarts. Maybe a big behemoth like Oracle would come across as, "Oh, now they can't even be bothered to build their own software, and expect a bunch of Open Source experts (since Open Source is where a lot of the expertise in databases currently exists, outside of companies that wouldn't want to help Oracle) to do it for a pittance." So, this kind of thing might turn out to be negative publicity for a big company, where it is great publicity for a small company like NetFlix.

Just some devil's advocate stuff. I think you're probably right that there are other areas where this would work. I don't know that you or I have thought of the best examples of them, though. Maybe someone reading this thread might have a company for which this model fits perfectly, though.

Reply on news.ycombinator.com