dbro

Forgive me for promoting this that I wrote:

csvquote: https://github.com/dbro/csvquote

Especially for use with existing shell text processing tools, eg. cut, sort, wc, etc.

Yes, this is what csvquote does. It does nothing else, just this so that programs like awk, sed, cut, etc. can work properly.

https://github.com/dbro/csvquote

That is correct, the data needs to be simple where the delimiter characters are never embedded inside a quoted field. I wrote a simple (and fast) utility to ensure that CSV files are handled properly by all the standard UNIX command line data tools. If you like using awk, sed, cut, tr, etc. then it may be useful to you.

<https://github.com/dbro/csvquote>

Using it with the first example command from this article would be

  csvquote file.csv | awk -F, '{print $1}' | csvquote -u

By using the "-u" flag in the last step of the pipeline, all of the problematic quoted delimiters get restored.

There is a small program I wrote called csvquote[1] that can be used to sanitize input to awk so it can rely on delimiter characters (commas) to always mean delimiters. The results from awk then get piped through the same program at the end to restore the commas inside the field values.

In principle:

  cat textfile.csv | csvquote | awk -f myprogram.awk | csvquote -u > output.csv

Also works for other text processing tools like cut, sed, sort, etc.

[1] https://github.com/dbro/csvquote

Good idea! Looks similar to something I wrote called csvquote https://github.com/dbro/csvquote , which enables awk and other command line text tools to work with CSV data that contains embedded commas and newlines.

To simplify working with CSV data using command line tools, I wrote csvquote ( https://github.com/dbro/csvquote ). There are some examples on that page that show how it works with awk, cut, sed, etc.

While not exactly what you asked for, I wrote something similar called csvquote ( https://github.com/dbro/csvquote ) which transforms "typical" CSV or TSV data to use the ASCII characters for field separators and record separators, and also allows for a reverse transform back to regular CSV or TSV files.

It is handy for pipelining UNIX commands so that they can handle data that includes commas and newlines inside fields. In this example, csvquote is used twice in the pipeline, first at the beginning to make the transformation to ASCII separators and then at the end to undo the transformation so that the separators are human-readable.

> csvquote foobar.csv | cut -d ',' -f 5 | sort | uniq -c | csvquote -u

It doesn't yet have any built-in awareness of UTF or multi-byte characters, but I'd be happy to receive a pull request if it's something you're able to offer.

You might want to check out https://github.com/dbro/csvquote which helps awl and other text tools handle csv files which have quoted strings as values.

Hi- I'm curious to know what your use case is. Can you explain why substitution is not sufficient?

Here's another suggestion for the criticism section (which is a good idea for any open-minded project to include):

Instead of using a separate set of tools to work with CSV data, use an adapter to allow existing tools to work around CSV's quirky quoting methods.

csvquote (https://github.com/dbro/csvquote) enables the regular UNIX command line text toolset (like cut, wc, awk, etc.) to work properly with CSV data.

yes!

https://github.com/dbro/csvquote

csvquote allows UNIX tools to work properly with quoted fields that contain delimiters inside the data. It is a simple translation tool that temporarily replaces the special characters occurring inside quotes with harmless non-printing characters. You do it as a first step in the pipeline, then do the regular operations using UNIX tools, and the last step of of the pipeline restores those troublesome characters back inside the data fields.

That's correct, and as you illustrate it's the possibility to have newlines and commas inside quoted fields that complicates things for grep/awk/cut/etc.

So instead of making a more complex version of tools like grep, we can make the data simple for these tools to understand. That's what https://github.com/dbro/csvquote does. It can be run in a pipeline before the grep stage, and allow grep/cut/awk/... to work with unambiguous field and record delimiters. Then it can restore the newlines and commas inside the quoted fields at the end of the pipeline.

Usually the person parsing the CSV data doesn't have control over the way the data gets written. If he did, he would probably prefer something like protocol buffers. CSV is the lowest common denominator, so it's a useful format for exchanging data between different organizations that are producing and consuming the data.

https://github.com/dbro/csvquote is a small and fast script that can replace ambiguous separators (commas and newlines, for example) inside quoted fields, so that other text tools can work with a simple grammar. After that work is done, the ambiguous commas inside quoted fields get restored. I wrote it to use unix shell tools like cut, awk, ... with CSV files containing millions of records.

Yes it's possible. I did this a few years ago as a learning project, and the relevant lua scripts can be seen here: https://gist.github.com/dbro/9920666.

... but I wouldn't claim that it's done efficiently. From the comments in the code's tests:

    # some results based on running speed tests on a lightweight netbook:
    # standard error = 0.01, 10kb of registers
    # < 1 ms per init()
    # < 1 ms per update()
    # < 1 ms per count() for single set
    # ~30 ms per set for count() of union of sets

that's exactly what https://github.com/dbro/csvquote does for commas and newlines both.

Thanks for bringing up csvquote. I wrote it last year, and am happy to hear that other people find it useful.

It is indeed a simple state machine (see https://github.com/dbro/csvquote/blob/master/csvquote.c), and it translates CSV/TSV files into files which follow the spirit of what's described in the original article in this thread.

But instead of using control characters as separators, it uses them INSIDE the quoted fields. This makes it easy to work with the standard UNIX text manipulation tools, which expect tabs and newlines to be the field and record separators.

The motivation for writing the tool was to work with CSV files (usually from Excel) that were hundreds of megabytes. These files came from outside my organization, and often from nontechnical people - so it would have been difficult to get them into a more convenient format. That's the killer feature of the CSV/TSV format: it's readable by the large number of nontechnical information workers, in almost every application they use. I can't think of a file format that is more widely recognized (even if it's not always consistently defined in practice).

Thanks very much! You just made my day!

I did, too. Different UI, though. See http://www.domainjig.com

Nicely done. Here's a similar site I made previously: http://www.domainjig.com

Beyond the shameless self-promotion I found that getting people to use it would take a lot of work, with lots of alternatives out there that have dedicated teams promoting them. On the positive side, I learned a lot from it; but I never made any significant money from it. If you're comfortable sharing your next steps, I'd be interested to know what you're aiming for with this project.

Good luck!

For those interested in giving this a try, here are a few additional suggestions from my recent experience.

There are less expensive alternatives to Linode. I switched to Loose Foot Computing (http://www.lfcvps.com) about a year ago and their service has been great. They often have promotional deals listed on lowendbox. See http://www.lowendbox.com/?s=loose+foot&searchsubmit=Find

tmux is amazing and takes care of window management. No need for dwm or xmonad anymore.

Lastpass integration with the browser works great with Chrome OS. This replaced keypassx for me.

USB tethering with my android phone "just works" to get on the internet.

Really happy so far. This is the right mix of easy, fast, cheap, and secure for me.

I wonder what happens in the RethinkDB office on Monday evenings? https://github.com/rethinkdb/rethinkdb/graphs/punch-card

Depending on what you're planning to use it for, there are some less expensive devices that are worth considering, eg. for about $90 shipped to the US

http://www.aliexpress.com/product-fm/546571245-Mele-A2000-TV...

I just received one and installed Debian squeeze on it, following the instructions here http://rhombus-tech.net/allwinner_a10/hacking_the_mele_a1000...

Shipping took about a month, but now that it's here it's as good as I expected.

Nice. I did too: https://github.com/dbro/muss . It's a shell script that coordinates mpc and dmenu. It's got two modes of operation, one for interactive browsing and another for a simple command line search. Some examples:

>muss elvis [queues up all songs that have "elvis" in the artist, album, or track name]

>muss elv pres not jailhouse

>muss lion rich -a -r ["Hello, is it me you're looking for?"]

-a means append to existing playlist, -r means randomize the playlist

>muss

when no arguments are provided, it opens up a very fast search/filter/browse interface using dmenu

Inspiration for this project were plait and the then-new instant search on Google.

not likely. see http://siteanalytics.compete.com/github.com/ and http://www.google.com/trends?q=git%2C+mercurial%2C+svn%2C+gi...

There are two things that need to happen: 1) create a good idea for a name that fits what you're offering to your visitors/customers 2) Check to see if that name is available.

I built this domain name suggestion tool that does both of these instantly: http://domainjig.com

In case you're wondering, it absolutely DOES NOT hijack or steal your ideas and register your domain name behind your back.

For exploring data sets, Google's fusion tables are useful. https://sites.google.com/site/fusiontablestalks/

some support for filtering and grouping

250MB data size limit

nice mapping capabilities

caveat: I created a similar site called domainjig.com, so take these as friendly suggestions.

The expired domain search is cool, well done with this tool.

Make it faster. Right now, loading the page to search for availability is taking longer than 10 seconds. When searching using the different tools, the response comes back after a few seconds delay. If you're comfortable discussing it, what are the major components of the backend? Is your data stored in RAM? Can you make the availability check instant?

eg. the expired domain search for "rabbit" over the last 30 days took a few minutes to return 2 pages of results. maybe the traffic referred from this post on HN is finding some bottlenecks for you?

The name spinner came up with some interesting suggestions. I'd suggest removing or hiding some of the advanced options, in favor of simplifying the default interface.

Hope that helps.

They take an interesting approach to UI design: let the user design the layout himself. I'm sure some people really appreciate this.

Frustrated by what was available, I created a weather app for Android which did what I want from a weather app: smart alerts about tomorrow's weather. In talking with people about it, I learned there are many different ideas about how the perfect weather app would behave for different people. This is not surprising; weather information is only useful in the context of how it would change your behavior. Since everyone has a different schedule, a consistent and simple interface is not likely to be possible.

The name of the app I created is "Anticipates (Weather)". comments are welcome!

What about looking at this from the perspective of the site owner? Facebook comments can do more for the site owners than the alternate comment systems. Increased exposure through facebook news feeds should help introduce their sites to new people.

Just as important is the back-channel that allows comments to be created and shown in different places (this was mentioned as an "incidental" note in Tech Crunch's article describing the comments system http://techcrunch.com/2011/03/01/facebook-rolls-out-overhaul...). I find it particularly appropriate to bring this up here on HN because I expect Tech Crunch to look with envy or exasperation when comments are posted HERE (HN) about stories appearing THERE (TC). Why shouldn't TC want to have all of the relevant, useful comments appear next to the content they create and host? Facebook comments could help that happen (indeed, so could have disqus if it were used by HN and TC both. Alas, disqus, RIP).

Read some good books. start with this one: http://www.amazon.com/Intelligent-Investor-Definitive-Invest...