Data, data everywhere | Hacker News Reader

Data, data everywhere

economist.com

63 pointshelwr16 comments

This is a huge problem for the intelligence services, and is an excuse frequently made for why the CIA failed to stop 9/11 and MI5 7/7. With the sheer amount of data logged and available to the services, how can one reliably wade through such huge amounts of data?

Being perfectly honest, I wouldn't be surprised if a Google Translate-style statistics package was in use right now. You, sir, fit the profile GeekWithGuitarAndCat72.

In recent years Oracle, IBM, Microsoft and SAP between them have spent more than $15 billion on buying software firms specialising in data management and analytics.

That's huge. Does that sound right?

That doesn't surprise me at all. I don't know if it counts, but the Sun deal was around $9.2 billion, wasn't it?

I wouldn't have thought that counts.

$15bn actually sounds conservative to me (I'm in the business intelligence space).

Yep, check out http://www.bi-verdict.com/fileadmin/FreeAnalyses/consolidati...

EDIT: And that table doesn't even include IBM's billion-dollar SPSS grab.

Wouldn't it be smarter to use a couple million of that to create a seed fund for aspiring technologies?

Why not do both? The major disadvantage with the seed fund approach is that you have no idea what you'll get out of it or how long it will take. If you but SPSS, it might cost you more, but you know exactly what you're getting and when you'll be getting it, which is a huge advantage if you feel you're in a hurry.

By "a couple million out of that [15 billion]," I meant both.

The one thing shows like 24 get perfectly right is that intelligence and warfare is now much more about computer scientists analyzing data than people on the ground.

Huge data sets is the new Oil, processing this new huge data is where the big money will be in the future.

Cue flash-forward to Mel Gibson hauling a datacenter-in-a-box across the desert, escorted by mohawked Aussies with guns

After the money made processing the oil there will be plenty more to be made with the distribution networks and gas stations themselves. While processing data is an interesting problem for geeks, the more mainstream problem will be visualizing data. I don't go down to the refinery and pull my car up to the spigot. I buy my gasoline at a pump where I can also pick up the milk and get a cup of coffee at the same time.

It is very true. Intelligence Agencies are always wondering how to view the data better. I know for one example, we have 150 TB's of video data coming in every day from Iraq and Afghanistan and we are trying to find ways to view it.

One number my boss says a lot is the CIA wishes they had 14,300 Analysts to view through all this information at once. These of course would be 14,300 people looking at video 24/7... Its a huge problem that doesn't have a solution at all yet.

As always with data when it comes to dynamic areas such as economics.

"The map is not the territory"

This has been here a few times before: http://searchyc.com/Data%252C+data+everywhere

(Just for the record, this seems the first time it got traction.)

I'm glad it finally hit the front page. It deserves some attention.

Reply on news.ycombinator.com