lrm242

I use HTCondor every day. I find it to be rock solid with top notch documentation. It is cross-platform today as well, so I'm not sure where you're coming from on this.

Link to previous discussion on HN, looks like they've come a long way! https://news.ycombinator.com/item?id=8324567

The morse is subtle but if you listen closely you can hear it. Set the Soundcloud player to 1:25 or so and you'll hear singing as they finish up the chorus. As the chorus completes at about 1:31 the singing stops and instruments play. Listen closely, and try to remember what morse sounds like in the movies. It's there in the background, and once you hear it it's obvious. Don't listen to the drums or tambourine, it's not there in the beat. It's "behind" the beat as if a background instrument was playing.

(edit: it also kicks in again around 2:30 and 3:30. So the start times are at approximately 1:30, 2:30, and 3:30.)

Outstanding.

Oh I know. Cootys Rat Semen is also from Sneakers. It is the first decoding of the anagram that Bishop decoded while playing Scrabble. They quickly discarded that and found "Too Many Secrets" shortly thereafter.

It's also an anagram for Cootys Rat Semen!

Please don't use Adjusted Close for backtesting execution. Adjusted prices are not real. They never happened, and using them will add unknown error to your backtest results.

Back adjusted prices are fine for building a model, but when back testing the model and simulating executions, you should always use real prices.

They are unable to do so. It is a queued system with a fixed transit latency through the queue of 350us. Any remaining un-executed of A is left on the IEX book.

IEX has this transit latency because they have order types that offer pegged execution, i.e. midpoints. A midpoint order is executed at the midpoint of the prevailing NBBO. The data required to build a view of the prevailing NBBO comes in from all other lit venues.

IEX has a 350us delay because they want to slow down the order entry side to give themselves enough time to make sure they are not pricing their pegged orders using stale market data. Simple as that.

It's 350us btw. Micros.... not millis.

Because they suck at technology. Its easier and more justifiable to pay someone else to take on the risk and pension fund gets less variance in their execution costs.

They had a big hand in wiping themselves out when they got caught colluding.

Since Flash Boys keeps coming up in these threads I want to provide an alternate reference for those curious. A recent rebuttal has been published called "Flash Boys Not So Fast". If you're interested in the topic, I highly suggest you pick it up. http://www.amazon.com/Flash-Boys-Insiders-Perspective-High-F...

Sell side traders or block crossing networks (where the other side is taken by another block trader).

Sell siders are compensated because they will build into the price the expected impact. The pension fund gets a known price impact with essentially 0 variance and the sell sider takes on the risk of execution to offset the trade, and essentially earn a spread between what the pension fund paid them for the risk transfer and their 'skill' in getting the trade done in the market.

It obviously can and will be more complicated than that with varying benchmarks and compensation schemes, but that is the jist of it.

IEX only delays the order entry side of the system. Market data (they are dark atm so no quotes/book feed of course) and trade reporting is not delayed. The magic shoe box is there to stop a very specific situation from occuring: IEX pricing pegged orders against stale market data.

Look, all this guy is saying is that he is paying someone else to take on the risk of executing his large orders. Block crossing networks and sell-side traders have existed for a long time. Hell, that's what Brad (IEX CEO and Flash Boys protagonist) did prior to starting IEX. This guy is saying: rather than use an order execution algo to get my trade done, I'm just doing a cross upstairs with some sell-side guy, paying them a known amount, and letting them deal w/ how to handle it on their side.

They aren't outsmarting anyone. They're simply transferring execution risk, and they pay handsomely for it, trust me.

Wrong. Front running has a very specific definition and requires a broker to be acting on behalf of a client.

1. Broker takes client order.

2. broker trades his own account first (with knowledge of client order)

3. Broker executes client order.

If you don't have clients you can't be front running. It is black and white and as simple as that.

autossh + ssh -C + tail -F ?

kafka sounds like an enormous overkill. If you want to store the logs locally while tailing, just add in a tee.

Exablaze also sells NICs with an FPGA as does SolarFlare. You can get an FPGA in an Arista switch as well. Very cool stuff indeed.

Those latencies aren't well defined, and likely representing input to output side processing. Also, they are in the .50ms range, which is 500us, not 50us. What isn't captured here is the significant added transit delay due to the forcing of a centralized processor. They are also averages, which are largely useless when dealing with market data. Let's talk about 95 and 99th percentiles. It's the bursts that kill you.

The perception of fairness as it relates to the SIP will always be an issue (even if it isn't a practical issue) as long as there are "two highways" if you will for market data. There is no reason it needs to be this way. Decomposing the SIP solves all issues and embraces the facts that already exists today: the SIP is a leaky lock on a distributed market that can by passed with ISOs and that there is no such thing as the NBBO because NBBO is entirely relative to point of observation.

Doing so, however, would require the venues to give up a major selling point for their lucrative direct data feeds. Not likely to happen.

Fair points re: the SIP. However, that submission is the benchmark for fairness is unfortunate. There is no reason it needs to be that way, and fixing it and distributing the SIP (i.e., removing the centralized processor) has numerous advantages that put to rest the latency, fairness and SPOF issue.

In that regime, measuring "same" becomes simple. Measure at the source the venue specific SIP feed (which contains the venue's view of NBBO) and the depth of book feed delta. Over the course of a day, that delta should effectively be 0.

You're absolutely right about the SEC website issue. It's silly to get upset about this since the "web" aspect of the distribution has so many layers.

Clock skew in that the clocks used by the participant may or may not have been synchronized. There are, effectively, no details about how the timing is done on the news side and there is allusion to some use of EDGAR posting timestamps which are from an entirely different clock.

Even more concerning, is that they map news events to market data using non-synchronized market data timestamps. It appears to be TAQ data. The timing aspect of the entire article is very poorly described but yet is fundamental to the claim they are making.

Exchanges do send data to both the SIP and direct data feeds at the same time. The problem is that the SIP has an intrinsic disadvantage due to the architecture and technology used that guarantee SIP market data will always be slower than direct market data.

I'm hopeful that, in time, this will be fixed. Not so much because I believe the latency induced by the flawed SIP architecture is material to SIP subscribers, but instead for 2 reasons:

1. The very same architecture that adds latency makes the SIP a SPOF in our market system and as the NASDAQ Tape C outage showed, it can really suck when the SIP doesn't work;

2. The PERCEPTION of unfairness is much more harmful than any actual harm done due to the SIP/direct latency delta. Fixing the SIP can directly correct the source of the perception of unfairness and bring some credibility to the market place and its governance.

Obviously. But the Univ of Michigan data was economic data and not collected from insiders. So it isn't insider trading. It is research. Should we make ALL research available to EVERYONE at EXACTLY the same moment? You can see how ludicrous that sounds when you consider how much research is behind some form of payment.

The SEC website explicitly states: "The subscription price is set annually by Attain, LLC, using a weighted average methodology based on the number of primary feeds for subscribers at the beginning of the each year. Subscriber organizations can be invoiced annually or monthly."

There are two very detailed documents on the linked website that have both business and technical details along with specifications for the service.

There is nothing shady going on here.

The data was also collected by a private institution who has the right to sell if it if they want. I'd love to have a Bloomberg Terminal on my desk, but they are sooooo expensive! The populist rage fermented by Scott Patterson and other journalists is at times quite disgusting.

Ah thanks for the link to the working paper, I had been looking for it. This jumps out from the footnote on page 7:

"As far as we know, this time is not available on any publicly available database. We initially obtained these times using real time “scrapes” of the SEC EDGAR site. We subsequently used a collection of these times made available by the Tier 1 subscriber. Given that this entity’s business model depends, at least in part, on obtaining and disseminating these filings in the most timely manner possible, they have strong incentives to collect accurate information about when filings become available on the SEC website."

Basically, clock skew is not accounted for at all and timings are potentially done from disparate and non-common observation points. It's quite possible that the delays they observe are strictly due to noise.

Uhm, it's not a secret at all. It's called the SEC Edgar Public Dissemination Service. It's right there on their website with instructions on how to subscribe along with specifications and contact information. There is no conspiracy here.

https://www.sec.gov/edgar.shtml

http://www.sec.gov/info/edgar/ednews/dissemin.htm

Before everyone gets all hot and bothered:

(a) The SEC feed can be subscribed to by anyone.

(b) Whether the delay was 0 or 100 seconds, the computer consuming the feed will always be faster than a human.

(c) The details of the study are murky and unknown. In particular, the data collection aspect is suspect given it sounds like the researchers received third party data not directly collected by themselves. How were the timestamps handled? Were various caching issues properly accounted for? (The SEC website is served up via Akamai, for example).

Anyway, it's great a populist topic these days. Evil HFT always beating you to the punch, etc. But in reality, it doesn't matter because they are always faster than you. I suggest you read this post by well known finance/trading blogger Kid Dynamite: http://kiddynamitesworld.com/someone-will-always-have-the-da...

A sharpe of 4 is ok. Most "HFT" type strategies have sharpes so high they don't talk about sharpe anymore as it is meaningless. It is simply a different type of trading. See Virtu's pnl distribution in their S1. As a point of reference, Blair Hull is on record saying they are interested in nothing below a sharpe of 10 (r-finance talk, should be googleable).

People so often overlook the role that execution plays in trading. As time scales shrink the impact of execution increases, and indeed many HFT strategies are not profitable without top tier execution (both in terms of fee schedules and technology). This is also one of the places that most academics fail when analysing a trading strategy. They make typical "assume a frictionless surface" types of assumptions that break down in a real market.

To your point, high-frequency simulation is damn hard and it is highly likely that they failed here and have tainted their results with bad simulations. Most of these papers would be well served to avoid dollars and cents and simply analyse the statistical qualities of their signal to their target over time. That is the first step in this business, anyway. Simulations only happen once the signal has been through the wringer.

First, I never claimed the markets were or were not zero-sum, only that your attempt to explain was wrong as it mixed realized and unrealized profit.

Second, zero sum requires a definition of utility for each participant. If those definitions vary (and they almost assuredly do) then a reasonable argument can be made that to define whether the activity is a zero-sum game or not requires a commonality amongst those utilities (dollars, for example). This is often not the case in trading (for example, the farmer hedging his crop with a speculator who seems to profit short term).