Hacker News Reader

Comment by LetBinding | Hacker News Reader

I am about to embark on a 18-month long data mining project, where I'll be developing Python NLP software for recognizing patterns in textual data.

Will the MBA be a good machine for this? Ideally I should get something with more horsepower, but if the MBA suffices, then I feel its portability will really increase my productivity.

View on HN

Is 4GB RAM enough for your project? If so, you're probably fine.

I do a lot of statistical ML (feel free to AMA) and chafe at the 32GB limit of the machines I'm using.

I could use all the RAM available. Ideally I would need 8 GB. But my thinking is that with the high I/O throughput of the SSD in the Air, swapping from disk won't lead to a perceptible performance drop.

I will occasionally have access to a server for particularly intensive tasks. But I would like to be able to code from my bed, in a coffee shop, library, bus, etc. Hence my fascination for the Air.

What kind of stuff do you do that uses up 32 GB RAM? Is your data set huge, or are your algorithms RAM intensive?

I'm gathering statistics on words, phrases and a few other things from a medium-sized (about a terabyte) corpus. There are several billion that aren't hapaxes. That's just the initial feature collection pass. Next, I'm looking for correlations among these features. It's a challenge to make it fit in 32GB. A lot of effort goes into bit-twiddling to make things fit and into algorithms that try to be intelligent about what to keep and what to discard.

Reply on news.ycombinator.com