bladecatcher

Are you sure about that? I think it's significantly lower than 200k

This makes sense for e.g, when you want to make your database content available for search (via elasticsearch) - for this you may want to push the raw DB rows.

Hacking on noisy trading floors is great fun

In my experience, Yahoo finance data is not reliable. In one case, I noticed that the stock price is incorrectly adjusted for dividends for all shares trading on a particular exchange. Free correct data is hard to obtain.

Postgres 9.4+ lets you use the write-ahead log and "replication slots" using which you can create streams of database changes that can be replayed to a client in the order they were made on the origin server [1]

EDIT: In case the question was which library did I use, I used BottledWater and it's worked great for me so far [2]

[1] https://www.postgresql.org/docs/9.4/static/logicaldecoding-e...

[2] https://github.com/confluentinc/bottledwater-pg

I recently had to build a product search functionality. I was able to pick up nearly all important features of Elasticsearch in about 2 days and had live-indexing and search up and running within 2 more days. Of course, this wasn't production level scalable code. But it was fairly easy to hook a Logical Decoding Output Plugin on Postgres, which would stream database mutations (in form of JSON) to a Kafka cluster, from where the Elasticsearch layer will ingest the data and index/update it appropriately.

Csv is an ubiquitous data format. Especially when you're receiving data from external sources which you don't have control over, which often happens to be the case in banking and finance (streaming csv dumps are quite common)

Your opinions on employees cannot be truer. Keeping employees in the dark, even when the founders know what's coming is just terrible

Yes, you can start the server with any number of slave processors. As long as you're not writing data to global namespace (i.e, if you can keep your code purely functional), then as pointed out above, one can use the "peach" (parallel each) operator [1] to parallelize execution. Further you also get the "distributed each" to distribute load amongst multiple processes instead of threads [2]

[1] http://code.kx.com/wiki/Reference/peach

[2] http://code.kx.com/wiki/Reference/peach#Peach_using_multiple...

IMHO, I feel Matlab is too clunky and inelegant for data transformation compared to APL-inspired languages, which needs a slightly mathematically inclined user to appreciate and unleash it's true power, with 10 times less code.

The biggest advantage of a language like Q is that it's inherently parallel. Map-reduce is naturally built into the language, so this allows the users to think in terms of breaking problems down into sub-problems and parallelizing them via the "parallel apply" construct[1]

Also another very powerful feature is the braindead simplicity of Inter process communication between KDB+ instances [2]

[1] http://code.kx.com/wiki/Reference/peach [2] http://code.kx.com/wiki/Startingkdbplus/ipc

This makes a real difference in ad-hoc environments where you want to setup and tear down complex trading software in real-time

It is really sad indeed. But I suggest you take a look at the J language. It's free for commercial use, unless you want to integrate the JDB, the equivalent of KDB+ column store. It also provides a rich feature set:

http://code.jsoftware.com/wiki/JDB/Announcement

http://www.jsoftware.com/jdhelp/overview.html

https://scottlocklin.wordpress.com/2012/09/18/a-look-at-the-... comes very close to capturing the ingenuity of J language.

After going through the J primer, KDB+/Q feels like a distillation of J - it basically took the most accessible ideas of J and packaged them for mainstream production use - some of their features such as these make it easy to setup a computing harness for electronic trading in no time:

http://code.kx.com/wiki/Cookbook/LoadingFromLargeFiles

http://code.kx.com/wiki/Startingkdbplus/tick

Also, the footprint of the kdb runtime and installation is unparalleled. I've run systems 24 x 7 x 365

I've been writing q code for more than 6 years now. It's definitely possible to write code which doesn't look like line noise. It's easy to build an automated testing layer on top of your q codebase too (via the c or Java bridges). I feel it's a perfectly legitimate language for high performance computing

Edit: I recently started learning the J language. While it may not necessarily be a language you want to write your production code in, it teaches powerful concepts - it let's you write functions which can deal with radically different shapes of data with ease, via use of the operators to change the behavior of a verb to monad or dyad. It provides great insight into higher level descriptions of your domain

JD looks very promising too for timeseries! I'm going to try it out one of these days.

Thanks for sharing this. I've been a kdb+/q user for ~5 years. Yesterday, after coming across this, I went through the J Primer. It has some really powerful ideas in it (I didn't know Arthur was behind J as well!), and kdb+/q now feels like a distilled version of J.

I've already started prototyping a small simulator using J. Thank you so much.

That's an inspiring project. Thanks for sharing

Reminds me of this video, where Maddox explores similar theme on women in video games

https://youtu.be/MpJGkG1g-Lk

Thank you much for pointing me towards Christ. I'm listening to the John Peel session past couple of days, it's perfect!

I love Nujabes too. I also highly recommend

Boards of Canada: https://www.youtube.com/playlist?list=PL0fSIHcxJhECMV53Llj23...

This is so disappointing! It was made free for commercial use last year, and was discussed on HN - https://news.ycombinator.com/item?id=7522327

I was only responding to the parent question.

Tavaga | Full Time + Interns | Mumbai, India

We're looking for highly skilled Android and Javascript Developers to join our team at http://tavaga.com

Tavaga is here to demystify long term wealth creation in the Indian markets and help investors work towards their life goals. We bring personalised investment plans, portfolio ideas from leading professional money managers and powerful market instruments such as ETFs to retail investors.

See open positions: https://angel.co/tavaga/jobs

We're a funded startup and based in Powai, Mumbai.

Apply: sridhar@tavaga.com

I beg to differ on Death's best album. I would say they realized their true style on Human and peaked with Individual Thought Patterns and Symbolic.

Agreed. I guess, this way it's easier to train people to avoid clicking on links they don't recognize.

This can be done in a variety of ways, but it's unfortunately not usually free. 1) A trading terminal such as Bloomberg 2) get API access to subscribe to the exchange feeds or other data vendor feeds.

Usually, this is out of reach for casual traders. I've seen one exchange publish order book snapshots every 5 minutes on their website, but this data is hardly useful for frequently traded instruments if your strategy is based on the order book dynamic.

I'd received an email on my corporate email address from some courier service claiming that they weren't able to deliver a package to me. The email also contained a link which upon clicking took me to a page which displayed the message "congratulations! You've fallen prey to a phishing attack".

Later we realized that this was a test conducted by the firm (a large investment bank) that I work for!

I remember someone referring to this as "intent driven programming". I once worked for a company that made a business rules engine. The idea was to describe a flowchart to the system and the software would ask what you wanted to do at each step of the flowchart in a top down manner till you fleshed out the whole program ..

I won't say these are flash crashes, but it's very normal to see 6 or 7 sigma events in Treasuries. Such large deviations are very rarely seen in stocks

I guess that depends on how close to the exchange you want to be. If your strategies were extremely latency sensitive, you might want to have not just the execution logic, but the larger algo framework on there too. This is particularly true of strategies that make use of order book dynamics.