Show HN: CSV Explorer (YC F1) - Explore CSVs with Millions of Rows | Hacker News Reader

Show HN: CSV Explorer (YC F1) - Explore CSVs with Millions of Rows

csvexplorer.com

60 pointsjastr34 comments

I usually use csvkit (https://csvkit.readthedocs.io/en/1.0.1/index.html). There are commands to list the columns, filter, browse the data in a somewhat formatted way with less. Typing the commands is a huge pain though, and I would be very interested in a tool that could instantly pop open and let me peruse. Excel can take minutes to load and eagerly does a lot of unhelpful formatting on things like dates and decimals.

That being said I would never want to (and often legally cannot) etl my data to some third party. That would be terribly slow. But I would happily pay for a nice desktop tool to do it for me; command line or GUI.

Csvkit is great, and I use it a lot. The target users for CSV Explorer though are mostly non-engineers ie. people who don't use the command line or code.

Kibana has quite a few rough edges but it takes non-technical users very far in terms of data exploration. Local setup also alleviates privacy concerns. The bummer is import/export: 'Upload CSV' feature has been almost introduced in 5.0 release but removed in the end [1], and CSV export has been asked for many times [2] but is still not there. So a user-friendly fork might be a worthy business idea.

[1]: https://github.com/elastic/kibana/pull/8497 [2]: https://github.com/elastic/kibana/issues/1992

shuf -n10000 foo.csv > foo-sample.csv; open $_

That should give you a 10k-line random sample to play with that should open quite quickly in Excel.

At a former job, our embedded device logs decoded to csv. Some of them were too large for Excel.

Pandas handled them without a burp. Pandas in Jupyter (Ipython Notebook) was a godsend.

There's a minimal amount of variable setup, but once you've done that once or twice it's easy.

Of course, any analysis or manipulation takes a bit of python code, but I see that as a feature, not the least because you can read it right there in the open instead of having to hunt for formulas in cells.

Same here. I load the data using pandas or parsing the rows by hand in go. It's interesting to watch your RAM getting filled up while the data is loaded.

Looks like this tool is for non-programmers, it's interesting to see that there seems to be a market here.

I've been prototyping simple desktop GUI tools on top of dask/pandas and PyQt that let you lazily load large CSVs (and other types supported by pandas) and interactively filter based on smart histograms (the per column histograms are fully interactive and provide crossfiltering across the attributes):

http://imgur.com/a/vfAmV

The idea is to map a lot of the basic functionality of dataframes onto simple GUI interactions (for example, changing column types, stacking and unstacking columns, pivoting) and couple that with an ipython console for more complicated data manipulation. And then maybe even adding adding Tableau like charting functionality:

http://imgur.com/a/z8d1w

For quick throwaway exploration/analysis. It can easily handle about a million rows just using generic pandas and a bit of memory. There's lots of cool database techniques that can also be used on small local data (for example, compressed bitmaps using EWAHBool for interactive filtering).

Do you plan to release something soon? Even just a way to visualize the rows by loading them lazily would be a huge improvement. I personally use pandas but some of my colleagues are not familiar with it, and it pains me when they try to inspect a large dataset by opening it on Excel on our small university-provided desktops instead of spending a couple of minutes writing a few python lines to extract what they need.

It sounds like CSV Explorer might work well for them.

I regularly test applications that generate big CSV reports. As I don't always have influence on the input data (we want to test on real datasets pulled from production servers of our partners), I fall back to defining constraints that must be satisfied. I ensure that they remain satisfied by analysing the csv files with powershell (of all the tools).

And the ability to question live services and parse JSONs, and cross-check with other reports, or merge multiple reports into one... It's indispensable.

Hi HN!

A few months ago, a college friend reached out because his consulting company had just received a 60 Gb spreadsheet, and they didn't know what to do with it! They actually tried opening it in Excel.

I'm excited to Show HN CSV Explorer - a simple web tool for opening really big CSVs! Try it out, and let me know how it goes!

Open it with a programming language like R, Julia or Python that supports the data frames. No reason to upload 60 gigs to the web or Excel.

This project came out of some of my contracting work for consultants, journalists, lawyers - people who don't know what a command line is.

Worked like a charm. This is really cool. I can see a lot of uses with large files. Especially filtering down "faceted" search items.

Cool idea - Having to work with sensitive data, it'd be great to have this functionality without importing/uploading to the internet.

Thanks - having worked with health data for a few years I can relate. Unfortunately, I don't have plans for a desktop app right now.

Think not of a desktop app, but of doing everything client-side in JS. That way, it's still a web app, but you're not schlepping [sensitive|large] data between front and back-ends. Also, by offloading the work onto clients, it scales much better - you could host the app on a CDN and have no real back-end.

I originally wrote it in entirely client side JS, but it didn't scale nicely past a few hundred thousand rows. For the really big datasets, CSV Explorer loads them into Redshift - queries takes a few seconds!

This is really awesome. What sparked this idea?

Not sure if you have this feature, but I think it would be really cool if you could open APIs to your own datasets. This could be really useful for enterprise applications that do a lot of flat file imports/exports to push/pull data.

Some consultant friends asked for help when they got a 60Gb CSV file from their client! Since then, I've also found that lots of tech companies share CSV internally. I've also worked with journalists attempting to look at large public government datasets.

Plenty of Big Data things also work with CSV. Like >1TB datasets big. It's pretty insane, but it works.

This tool would be great were it not for me (rightfully) getting fired and possibly sued by any of my clients if I uploaded even a single file. In fact, what kind of business with millions-of-rows kind of files would entrust said datasets to a service that to me seems strangely vague [1] on how the data is secured, or where it is actually going to be physically stored.

[1] https://www.csvexplorer.com/legal/privacy/

Update: A one minute demo video - https://youtu.be/RBiDL5neWDc

Is there any reason this tool couldn't be 100% client-side?

Perhaps - I tried a client-side implementation, but I had issues scaling past a few hundred thousand rows. I understand hesitations of uploading data to the cloud, but I now have users looking at hundreds of millions of rows in seconds thanks to Postgres!

You can run PostGres natively as well and do the same thing no?

Not really an all-in-browser solution then.

I suggest exposing a config variable in the js client to set the hostname of the Postgres server. This way you can provide a self hosted version of the backend, and customers can self host it either locally on their machine, or within an internal network. Same benefits of the browser, just need to change config.

My solution to this problem, so far, has been ipython and pandas. (and maybe jupyter notebook for visualization and sharing).

What's the value-add here?

It's like a Jupyter notebook for non-engineers, people who don't know what python is, eg. consultants.

What's "YC F1"?

I was in the first batch of YCombinator's Fellowship.

Does this do stuff lnav can't?

you should integrate with commatocolumn.com! ;)

Reply on news.ycombinator.com