karterk

That's kind of the best of both worlds. It's actually pretty tiring travelling AND working at the same time. You will have a large amount of context shifts which will destroy your productivity.

I also wish remote work is more common. While it usually works when you know a company really well (e.g. small company with known, trusted colleagues), not many companies are actively trying to make remote work possible.

That's being a little pedantic. Applying X for Y is definitely being inventive. That's how largely every domain works. We stand on the shoulders of giants etc.

The joke at my company is that everytime Google publishes about one of their technology on a paper, they already have a next-gen version of it running internally.

[1]: http://google-opensource.blogspot.com/2013/08/learning-meani...

> But we’re deadpooling all of ‘em, effective immediately.

I really don't like the tone of many parts of this article.

> Facebook is just too big to fail when it comes to normal people and photos.

The same could have been said about places.

From personal experience, this is the most important thing. There will be days you will keep asking yourself what significance a little thing that you're doing that particular moment has on the grand scheme of things. The reality is that such little things compound over time. The most important thing is to keep persisting. Early on, you will feel like quitting and jumping onto the next big idea/thing, but only with persistence any of them will become fruitful. After you work on a few projects you will be able to interpret the early positive and negative signs better.

HBase-Hadoop integration is great, but Cassandra has caught up significantly on that front. If you want strong consistency, apart from HBase there is no other (non-sql) solution that's really battle tested. If you're okay with eventual consistency, you should take a hard look at Cassandra.

Since you will be mostly iterating over all records during your iterative algorithms, storing them in a separate in-memory DB makes no sense (have to call external process via socket).

You can then use a framework like zookeeper/akka for managing nodes in the event that you have to scale out. Even a simple master/slave set-up using thrift services will do.

However, 5GB of data is literally nothing, and that statement holds till your data size is atleast 50-60 GB. Given that 64 GAM RAM machines are now commodity, I would just load the entire thing in RAM and write a multi-threaded program. Sounds old school, but regardless of how well documented hadoop, spark and storm are, there is nevertheless a learning curve and a maintenance cost. Both of which are well worth only if you see your data rapidly growing to the X TB range. Otherwise, it might be just easier to stick it in a single machine and get stuff done.

You can stick to Scala/Java, and so long you develop good abstractions around your core algorithms, you can always move to spark/hadoop when you need it. Feel free to send me an email if you want to talk more (email in profile).

Having said that, Spark is really great for running iterative algorithms and will definitely fit with what you have described. I suggest staying away from building it on your own using riak/redis (atleast until you have ruled out spark), as you will run into lots of operational issues like handling failures, resource allocation, retries etc.

But once you get up and running, you will realize that this comes with a great cost. EC2 absolutely sucks in terms of raw performance. I also notice variation in performance of machines on different times of the day, and between machines.

Amazon has done a great job at selling "the cloud", and I can see many CXOs buying into that. But the fact is, renting dedicated machines from a good provider isn't exactly that hard. They take care of a lot of things for you.

EC2 is good if you have wildly varying amounts of traffic such that you need to really have that kind of elasticity in your infrastructure. However, for most businesses, especially web apps, that's not the case.

* understanding prototypal inheritance, e.g. knowing what __proto__ means

* knowing how to model asynchronous code in several ways, e.g. using callbacks, events or even using promises

* understanding how an event loop works

* knowing perf related stuff (though one should always benchmark stuff)

There are specific verticals, where one can very well be. For e.g. I know people who have been focussing on a vertical since they are 15 years old (e.g. web design, digital art). You can definitely master some verticals after 13-14 years.

The way "big data" is stored today also poses problems with respect to queryability. NoSQL systems are great at storing huge amounts of data efficiently but don't help us slice and dice the data easily. Having to write map reduce jobs for every "query" is painful and time consuming. Tools like Hive, Pig and Cascading help in writing MR jobs succinctly but are still very slow when someone wants to quickly filter and explore the data.

Having said that, Twitter should really be doing what Prismatic is doing. I find the discover tab to be largely a #fail. With all the data that Twitter has about me, they should be surfacing interesting things, but they don't.

Been there, done that. And, it's terrible. Monolithic things try to do too many things. It's fine if you grow along with it but a few years down the line - a beginner will find it very difficult to get started and manoeuvre around.

Small, composable things are better. If something is popular, it will be forked and maintained. Also, monolithic things tend to slow down over time (in terms of keeping up) as they start accruing so much baggage.