In other words: breadth-first search is better when dealing with hard problems
3,263 karma · joined August 11, 2011
haha, I love this guy
I'm trilingual and I can assure you it doesn't help with the ADD problem
lots of practical exercises, playing with real data and APIs
That particular comment conveyed a valid point, it was done effectively and added to the discussion. I'm rather concerned that this site becomes more and more of an echo chamber as I see dozens of comments from different people that could be easily compressed into one.
HN>> Hadoop is not a new type of database, and it cannot be used to sift through big data streams in real time.
Related>> Why big data is better than small data : http://www.quora.com/Why-the-current-obsession-with-big-data
I used Hadoop/Hbase for EEG time series analysis, looking for certain oscillation patterns (basically classic time-series classification) and it was an embarrassingly parallel problem:
Map:
1. Partition the data into fixed segments (either temporal, say 1hr chunks or location based, say 10x10 blocks of pixels). Alternatively you can use a 'sliding window' and extract features as you go. In some cases you can use symbolic representation/piecewise approximation to reduce dimensionality, as in iSax: http://www.cs.ucr.edu/~eamonn/iSAX/iSAX.html , "sketches" as described here: http://www.amazon.com/High-Performance-Discovery-Time-Techni... or some other time-series segmentation techniques: http://scholar.google.com/scholar?q=time+series+segmentation
2. Extract features for each segment (either linear statistics/moments or non-linear signatures: http://www.nbb.cornell.edu/neurobio/land/PROJECTS/Complexity... ). The most difficult part here has nothing to do with MapReduce but decide which features carry the most information. I found ID3 criterion helpful: http://en.wikipedia.org/wiki/ID3_algorithm, also see http://www.quora.com/Time-Series/What-are-some-time-series-c... and http://scholar.google.com/scholar?hl=en&as_sdt=0,33&...
Reduce:
3. Aggregate the results into a hash-table where the keys are segment' signatures/features/fingerprints, and the values are arrays of pointers to corresponding segments (Based on the size this table can either sit on a single machine, of be distributed on multiple hdfs nodes)
Essentially you do time-series clustering at the Reduce stage with each 'basket' in a hash-table containing a group of similar segments. It can be used as an index for similarity or range searches (for fast in-memory retrieval you can use HBase which sits on top of HDFS). You can also have multiple indices for different feature sets.
-----
The hard part is problem decomposition, i.e. dividing work into independent units, replacing one big nested loop/sigma on the entire dataset with smaller loops that can run in parallel on parts of the dataset, when you've done that, MapReduce is just a natural way to execute the job and aggregate the results.
So even if you use customized, co-located hardware/FPGAs or even ASICs with the fastest interconnects and unlimited bandwidth and your engine responds in nanosec, still the message you get from/send to an exchange can sit there in a queue for something like eternity.
here are some links
http://www.quora.com/Machine-Learning/What-are-some-good-res...
http://www.quora.com/Machine-Learning/What-are-some-good-lea...
http://www.quora.com/What-are-some-good-resources-for-learni...
http://www.quora.com/What-are-some-good-resources-to-learn-a...
Workshops: http://www.quora.com/What-are-some-workshops-on-large-scale-...
Also see the tutorial "Scaling Up Machine Learning" at KDD2011: http://hunch.net/~large_scale_survey/