How Linux mastered Wall Street | Hacker News Reader

How Linux mastered Wall Street

itworld.com

83 pointsabennett29 comments

"As late as 2007, Wall Street exchanges were still largely run on Unix, such as HP's AIX and Sun Microsystems' Solaris."

Isn't AIX developed by IBM? HP maintains HP-UX if I'm not mistaken.

Frankly, I'm often surprised that how much of wall street is still windows based. A year or two ago I spoke to a co-location vendor who told me that most of their customers run their automated trading strategies on windows machines. A friend is currently working with a very popular market data vendor who only provides windows API. Some of their customers use linux, but they are forced to use WINE! The vendor has no plans to provide linux based api, something more portable like java or even straight tcp/ip message specification.

A lot of this has to do with Excel. Average finance guy lives in Excel and no, libre/open office is not a replacement as they will not run a huge number of macros and DLLs connecting spreadsheets to external systems. Quant guys are often locked in matlab world and again - octave on Linux cannot be used due to incompatibilities.

I forgot about Excel. I think people outside of wall street will be surprised at this. I, myself, used a visual basic app to do some trading (not visual basic.net, VB!) as recently as a couple of years ago :).

Even in the linux world, PERL is used by many quants for their analytics...not just to parse files, but as an alternative to R, matlab, etc.

It's Perl, not PERL. /sigh.

Also, Perl is A Real Language, unlike.. ah, >.>, Excel.

Let me switch sides for a moment and defend some of what happens here. In certain instances, Excel is a very good tool. In fact, if Excel's performance could be improved, it would become the best tool for some scenarios.

For example, you need to manage a stream of data. You have prices coming in and you need to publish another set of prices going out (perhaps you are trying to peg quotes in the market off another set of quotes...details don't matter). Use Excel's RTD to put incoming prices into some cells. Calculate the outgoing price by typing out a simple formula which references the original cells.

All other commonly used languages are more batch oriented, rather than stream oriented. In other languages, the code will implement some sort of call-back method to handle incoming data. This program will need some sort of "main" method. Perhaps the program will need to be compiled. Certainly the user will have to know how to run it.

A non-programmer will obviously pick the excel solution (and many programmers will as well).

The financial industry does lots of things ass backwards -- a few of them really do make sense :)

You sound like you might be interested in a paper about "Improving the world's most popular functional language: user-defined functions in Excel" (http://research.microsoft.com/en-us/um/people/simonpj/papers...).

I came accross this recently, along with some of the other papers guys from microsoft have put out about implementing functional languages within excel.

I have to remind myself that these came out prior to the 2007 & 2010 versions of excel, and they ignored the most important recommendations and went with UI based "improvements" instead.

These guys at Microsoft Cambridge Research are wasted as a talent, Its as if MSFT have "bought" them as a way of keeping ideas that are disruptive to their current streams of income under control.

how is consistently publishing papers and pushing the boundaries of PL further "keeping ideas disruptive to their current streams of income under control"?

I mean the ideas are not hoarded, they are published, F# is open sourced and SPJ hacks on haskell. Where did you draw your conclusion from?

Oh wait, I forgot this is HN.

While they work for Microsoft Research, they won''t be implementing their ideas in commercial products.

I think C# disproves this fairly well. C# has lambda functions, monads, type inference (either now or next version). People like Eric Meijer have moved from pure research to product groups.

And there's also F#.

Our sound designers are using excel with some macros to sort out our sound assets in the game - very useful to them, and very concise representation of data for them.

if Excel's performance could be improved, it would become the best tool for some scenarios

Where have you found Excel's performance to be inadequate?

I believe excel only updates once a second (or whatever the refresh rate is, I remember it being too slow for high frequency trading).

Oh gosh, where to start:)

- on any calculations that are iterative, - the recacluation rate is limited to something like once per second, - sheets containing over 2000 market positions getting real time quotes grind to a halt. - any time you have more than about 20,000 calls out to a COM component per sheet - large vba functions, it seems vba is about an order of magnitude slower than the corresponding Excel built in methods.

TL/DR large sheets, dde, COM, heavy use of iterative solvers, large amounts of vba, all make excel really slow down.

Having said all that Excel is really awesome, it's just not for big data:)

Suppose somebody made a spreadsheet for big data. What might you do with it?

one of the big problems with excel as a tool in these sorts of situations is that it is almost impossible to audit. yes, a non-programmer can hack something up that kinda does what they want but have they stress tested it? there's some statistical evidence (don't have time to google) evidence that virtually all spreadsheets contain errors, add in that these spreadsheets are running large $$ decisions and you have a recipe for... well, pretty much what you see all the time lately ;-)

I completely agree. A few years ago a large bank banned spread sheet based quant models from their desk. There are certainly draw-backs, I wouldn't want to give my programming languages for spread-sheets, as they exist now. However, I think programmers often forget that there are lots of good ideas in Excel as well.

I wonder if a more explicit graphical flow-based programming system would be even easier than the implicit flow-based setup that's going on behind the scenes in Excel. For example, you have an object that represents an incoming stream of quotes. An output pin on that object connects to a "formula" object, which is wired to an output stream object. It's all based on ancient ideas (like Max/MSP, Pure Data, etc.), but maybe it could be useful.

Disclaimer: I created a flow-based system for home and building automation (no doubt one of many), so I tend to see uses for it everywhere ;).

Check out http://www.tickit.com/webinars.html

It reminded me of LabView.

Explicit graphical flow-based systems have their own issues. If the logic is simple, the systems are manageable. However, complex logic very quickly leads to a very complex graph with huge number of lines criss-crossing each other, until they make no sense.

Conal Elliot has done some interesting work in the area of "visual programming." The following is a good paper, but entirely academic: http://lambda-the-ultimate.org/node/2159.

Though there is no drop-in Matlab replacement, at least there are native versions of Matlab for OS X and Linux. The only incompatibility that would arise there is if some underlying third-party .mex files used Windows-only APIs.

To be fair there are a lot of high performance windows tricks. They involve digging through the DDK and other obscure places to find documentation, but it's not impossible to build performant windows solutions. The open nature and documentation of Linux internals makes it easy to hire commodity developers on the cheap (relatively speaking), which I suspect is the real underlying trend.

I'm not convinced that it's really cheaper to hire (good) Linux devs for this kind of work, and even if it were I doubt that's their biggest concern. A bunch of "commodity" developers wouldn't be able to build a complex real-time system like that and meet those targets - look what happened to the London Stock Exchange a couple of years back when they tried that.

It seems much more likely to me that the real underlying driver is the level of customisability they need. They're talking about making scheduler changes to the kernel and building the whole OS themselves - that's a level of control Windows (or any other closed-source option) doesn't offer.

Ahem. I know of at least one HFT in NYC that's killing it with FreeBSD.

is it to do with this : http://info.iet.unipi.it/~luigi/netmap/ ?

If you read the article you'll see it's talking about exchanges, not those trading with them.

Not so:

> "The trading shops saw that the lowest-latency solutions would only be possible with Linux," Lameter said. "The older Unixes couldn't move as fast as Linux did."

Reply on news.ycombinator.com