Hacker News Reader

Comment by geocar | Hacker News Reader

kdb+/q is another really good choice for dsv[1] and json[2]. You can certainly create single-file databases (if you really want to e.g. for exchange), but splayed table[3] is faster so you'd usually do that.

[1]: http://code.kx.com/q/ref/filenumbers/#load-csv

[2]: http://code.kx.com/q/ref/dotj/

[3]: http://code.kx.com/q/cookbook/splayed-tables/

View on HN

The problem with that might be the licensing costs once you use it commercially (eg. at work). IIRC the license prices aren't public, but you're looking at over $10k in any case.

I personally prefer J to K in the APL family of languages. They also have a relatively cheap database, Jd [1]. Individual licenses are $600. Still a bit too much for my data mangling needs. :)

[1] http://code.jsoftware.com/wiki/Jd/Index

$10k isn't a lot (assuming that's right; it could be). I mean, it's a lot if you're used to something like MySQL or Postgres-levels of quality, but I've seen quotes for Oracle being almost $50k per core. MS-SQL is something like $7k per core, and kdb+ is definitely a lot more useful to me than MS-SQL.

There's also a per-core/minute pricing which might be useful.

Sure, kdb+ would probably be worth every penny even at $100k/year when it's the right tool for the job. I gather it's genuinely the best in-memory database for computing arrays of varying rank.

But a lot of the use cases these other tools are good for are small tasks every now and then. I feel kdb+ is in a different category.

Anecdote: I frequently use kdb+ for small tasks. For me, its in the "all-purpose" category. The limitations are only in the ability I have to use it.

For example, removing nonconsecutive, duplicate lines from a file, such as a CSV file:

   exec echo "k).Q.fs[l:0::\`:$1];l:?:l;\`:$1 0:l"|exec q >&2;

where Q.fs is a function in a script thats bundled with the interpreter; the chunk size for reading the file into memory is adjustable by editing the function.

You can make it simpler:

    l:0;.Q.fs[{if[x~l;:];-1 l::x}each]`:input

or if you have memory:

    -1 distinct read0`:input

...or if you want to use k:

    -1@?0:`:input

Stupid question: With -1, how would I suppress the return value? Use a function?

   k)a:{-1@?0:`:input;};a[]

It's not a stupid question.

What this does is return generic-null :: which .Q.s doesn't print.

Reply on news.ycombinator.com