> Big data analytics with Pandas and SQLite
> A Large Data Workflow with Pandas
> Data Analysis of 8.2 Million Rows with Python and SQLite
> This notebook explores a 3.9Gb CSV file
Big data?
> A Large Data Workflow with Pandas
> Data Analysis of 8.2 Million Rows with Python and SQLite
> This notebook explores a 3.9Gb CSV file
Big data?
1) Does it fit in a desktop computer memory? If it does it is "small data".
2) Does it fit on a hard drive in a desktop? Then it is just "data". Or medium data.
3) If you need a cluster or some centralized network storage to fit and manage it, you might have big data.
4) Next level up is streaming data. Your data doesn't even sit anywhere because it is accumulated faster than you could ever processes it.
Methodology-wise - yes. Basically, anything that you cannot confidently put into a spreadsheet, manually investigate a chunk of it and search with regex is (a buzz word) "big data". Also, you need you forget about all O(n^2) scripts.
If you do any kind of analysis (and you are of size < Google), processing is the easy part. The hard part is to get maximum out of it, while disregarding noise. (Also, it is why when you ask data scientists what they do, a common answer is that 80-90% of time is data-cleaning.)
Why just spreadsheets? This is essentially loading data in SQLite and manually investigating it with SELECT queries.
Methodology-wise - yes
Big data "size" is a constantly moving target, as of 2012 ranging from a few dozen terabytes to many petabytes of data.
80-90% of time is data-cleaning
But going from ~MBs to ~GBs change methodology (of course, it's more about number of items and their structure than sheer size). Even at this scale there is some tradeoff of quality/quantity. And feasibility of using some more complex algorithms.
Data cleaning - sure, for all data. But when volume is in MB, it is still possible to open it it a text editor and manually look into abnormalities. Once it's GBs some artifacts will remain undetected.
I'd look strangely at anyone claiming "big data" for anything smaller than double digit terabytes.
For data analyses which I've done, my 8GB computers will complain when analyzing 2-3GB datasets because when processing the data, the computers needs to make a copy in memory, since the analysis isn't in-place, and there are memory leaks (stupid R). So 4GB-6GB of memory usage causes hangs (since the OS is using a lot of memory too) and is why I've had to resort to SQL for data sets that hit that size. (which is perfectly fine)
(Also Big Data is one of those words whose definition sucks to pin down)
Let's say I have an organization and we run into a dataset that is 1,000 times larger than anything we've dealt with. Should we put out a help-wanted ad for a "big data" developer? What if the largest dataset we had previously dealt with was 100 rows? The reason we have terms for things like this is to facilitate communication. If the definition is highly sensitive to context, then the term doesn't facilitate any communication, the whole context must still be spelled out. If the term is to have any meaning at all, it can't be a relative thing. Of course I'm of the opinion that the term is already meaningless, so I guess do whatever you want :)
Please don't call any datasets that is smaller than 100 million rows big data.