I think it will be much cheaper to rent an instance on AWS or Digital Ocean , on an hourly basis, with Hadoop and Apache Spark already deployed.
That way you can play around with much larger datasets.
3 karma · joined July 16, 2014
I'll give you an example: My friend hired two developers to build him an e-commerce site with inventory control in the backend. There are dozen of ways to approach this problem but the main thing to understand is, that he needs a solid working solution.
This was their first job, they decided to build it from scratch in Django, by reasoning that it has a great admin ui out of the box. Django maybe a good solution if he had an IT department or the e-commerce was his main business, but it isn't.
They should've offered him an off-the-shelf solution with minor modifications, a solution that is widely used and can be supported by a lot more developers.
Their main concern was to build some thing new from scratch which is tremendous amount of fun, and should be done when it is right, but it also means that you are getting your leg work on the client's expense, which can cost him to lose money.