Hacker News Reader

Comment by thisone | Hacker News Reader

you should figure out what you plan on doing with the data first. That will help inform what you use and how you store it.

View on HN

Nobody really knows. It's all new. In my view we need something we can play with without having to write lengthy requirement docs to IT.

what skills do you have in the team? also, budget & timescales.

SQL and general software development are the main skill. We are not data experts. The timeline should be as short as possible. I feel IT is making this into a big project with big budget to to justify their existence and charge a lot of money to our cost center and not acting in our interest. But I don't have hard data because I don't know that world well.

1TB sounds big, but if you were to get a server with a couple of SSDs and put a supported RDBMS on it you could do a lot. I think that you have a cast iron case with IT because you don't know what the outcomes are for the business yet - frankly who's going to sign off for a big implementation until you do?

We couldn't justify Hadoop until we were +6TB... that was some years ago, pre ssd and pre big disks - our old rdbs server had got horribly complicated because it hit that scale - when it got to 10TB its raid controller corrupted the control files for the db. We had support and the consultants who came did a bunch of mind blowing work to restore it.

But by then we had everything on Hadoop and we were only maintaining one legacy app on the old box - which we had told the users was dead and buried in any case (but kept to maintain friendships and so on).

Back to my main point - some people think that analytics starts with a question, I don't, I think it starts with the data and instruments to inspect and understand the data. I think that you should get the data on some sensible box, put a good database on there that your guys can use (as they are SQL folks get an SQL one) and put R, R-studio and R-shiny there and start inspecting and understanding it.

If no questions or insight arises then little is lost and it can go on a cheap fileserver until someone needs or understands it. If nuggets of gold start appearing then you can invest further in both skills and kit. I would recommend Hadoop either if other datasets of the 10gb+ scale start appearing or if this one gets >4TB. The other database recommendation would be because of mega joins - which Hadoop does well, and the general need to build a cheap EDW - if you have lots of cash you can build an expensive EDW instead! 4TB because currently disks are ~6TB and there are nasty scaling bumps in not Hadoop world that start kicking in as per my experience.

IT are right to charge once this is established to be valuable; if it's work £250k a year to your business then it makes sense to be spending £50k a year to put SLA's and resiliance on it. If it's worth £2k... well...

I forgot, what kind of data is this? Is it images? rows in a table?

I've no idea of what your doc requirements are, but if you don't have any real plans for the data, stick it in a cost effective format that requires little to no maintenance but that can be read by multiple systems. Say parquet or avro for example.

I assume you don't have someone/some system 24/7 pouring over the data so you could look into hosted notebooks like databricks, qubole (and gleaning from people I met at a recent conference any of the bazillion that are about to launch), or host zeppelin yourself or use AWS EMR.

And don't lose your source data for that moment when you do figure out what you want to do with the data and you realise you need to reformat it.

Remember you don't need all of the data for estimating models.

Do you expect to solve any known business problem with it, or you are just toying around in hopes that something comes out of it?

Right now there is no data at all so a lot of people hope they will find something useful. But nobody knows for sure.

Reply on news.ycombinator.com