gd1

How do I explain this... it is quite odd talking to someone who thinks your way. I know you exist but I struggle even to wrap my head around it.

Racism/sexism/whatever... a wrong is a wrong. Fight it wherever you find it. My example was an interaction between two people, two individuals, and now you're talking about "discussing social groups". Racism and sexism is caused by considering people as "groups" and not individuals, and your solution is to... do more of the same. Recursively. All I have seen the left do in the last 40 years is carve up society into ever smaller and more specific groups with associated victim-hood points and it doesn't get us anywhere. All you are doing is entrenching division.

I'm sorry, this is ridiculous. So if a Japanese person discriminates against me on the basis of race in Japan, it is racist, but do it in the US and it is "racial discrimination"?

Are you going to let the dictionary makers know?

If the series don't really correlate to temperature (and the red noise doesn't) even though they appear to during the calibration period, then how it moves before the calibration period will be random. So when you take your weighted average of this randomness they will tend to cancel out (or at least tend to be flatter than the reconstruction during the calibration period). The greater the number of series, the more pronounced the effect. The point being that a hockey stick shape is an artefact of the method used, whether or not the proxies actually are good indicators of past temperature. I am trying to keep this somewhat simplified.

Of course you don't have to take my word for all this, thanks to Climategate we have all the behind-the-scenes dirty laundry:

http://foia2011.org/index.php?id=4191

Here Rob Wilson does just the test I'm describing and seems a little concerned about the results.

Does it sound by my post that I am ignorant of these "multiple" reconstructions? They all use the same proxies and similar techniques, Esper, Briffa, bristle cones, etc. I have read a good number of them. They are all awful.

Paleoclimatology isn't any worse than some other soft sciences I guess - the difference is that there isn't so much riding on them being right.

Lies, damned lies and charts eh? The CO2 can be put in a better context here:

http://www.geocraft.com/WVFossils/PageMill_Images/image277.g...

Age of the Earth: 4.5 Billion years. Time we've had weather satellites up: 30 years. Geologic time, it's cool, wrap your head around it.

Fun fact, C3 plant life dies at 220 ppmv or lower CO2. Wheat, rice, soybean, cotton, tomato, barley, potatoes - all gone. No ifs and buts about that one, photosynthesis ceases. That's how dangerously close we were teetering to extinction, with the world's carbon entrapped underground. Luckily, along comes man to dig up fossilized carbon and free it back into the carbon cycle where it came from...

The flaws are numerous, but the easiest to explain is that they employ a post-hoc screening bias. Imagine that you gather a few dozen time-series together. They go back two thousand years. Then you regress them against the (very short) instrumental temperature record in the period where they overlap. You keep the ones that pass and throw the rest away. You form some sort of linear weighted average of the selected proxies (i.e. PCA for example).

The thing about this process is, if you feed in completely random red-noise time series you will get a hockey stick. The reason is simple, the selection period will mirror the instrumental temperature (mental check: it has to, after all we only selected the proxies that matched the instrumental temperature). You have your blade. But none of those red noise proxies actually reflect past temperature do they? They just happened to be selected by random chance. So in the time before the selection period, the noise cancels out and you have a flat line. There is your stick.

This is the core of how these "proxy reconstructions" work, and it is nonsense.

Outside of that, it gets even better. The stories are comical and numerous. Mann famously fed a contaminated Finnish lake sediment series into his algo, which happily regressed it against temperature and found that it passed significance if it was used upside down (in the opposite orientation to that suggested by the author in the original source paper). So in it went. GIGO. Or how about the question of which temperature series to regress against? You'd think you'd regress against local temperature right? Or maybe at least the same continent? Oh ye optimistic fool, go and google "teleconnection" and "non-local statistical relationship". This is what passes as "climate science".

I'm sick of the drama. Once upon a time it would have been possible to just watch that video and marvel at the majesty of nature, now we can't have a single natural event these days without you and your cargo-cult mates attributing it to 'climate change'. Add it the list: http://www.numberwatch.co.uk/warmlist.htm

Why? It's a glacier. A "river of ice". They flow into the ocean. Calving is what they do ffs.

starter pack is 25k for 2 cores

What a terrible article. That line of APL is sheer beauty (http://www.youtube.com/watch?v=a9xAKttWgP4) - being ignorant of a language's syntax and not willing to invest the time to learn it is no reason to claim it is 'impenetrable'. I'm sure sheet music, replete with quavers and semi-breves and treble clefs is impenetrable the first time you see it, but once learned is far superior to playNote(double hertz, double duration).

Worse, it goes against his point just a paragraph earlier. The whole advantage of Q/K/J/Haskell style languages is that once skilled in them, the code is much (as in 10-20 times) denser than the equivalent OO-style code, and the entire flow can be grok'ed with ease. There is much less danger of over-engineering in those languages.

I think we are getting closer to understanding here. You're kind of quibbling over the definition of value in your other post - someone can provide a "valuable" service even if it isn't currently being used (by your 'real transactors'). I might sleep better at night knowing that if I want to liquefy my investment in Apple tomorrow, I can do so without having to worry that nobody will be there to take the other side of the trade, that I'll have to pay an enormous spread or be left holding a position I want to be out of. An international corporation can be more confident of planning their cashflows knowing that the FX market will always be liquid, and will be quoting at most 2 ticks wide in the majors 24 hours a day every day. This is the oil of an economy, the freedom to allocate capital when you want, where you want and pay the smallest possible friction costs to do so.

Many things in our economy can be described as a zero-sum game. Society only needs so much of any given good, and at the retailer level you aren't involved in increasing demand. Two corner stores might get into a price/advertising war, but let's argue that there is still only a fixed amount of product they can sell to a small community. So advertiser's get fat on fees (in our case, exchanges do). So what is the point you ask? Evolution, competition, it can only be good for the consumer. You still haven't disputed that by the way, the consumer wins don't they? How could they possibly have lost? It takes a tin foil hat to think that it isn't better and cheaper to invest now.

" they are not selected by the quality of the service, just ability to rip each other off" - people have a bit of a naive view that there are some sort of magical harlem globetrotter moves you can pull on an order book. The mechanics are dead simple I'm afraid. It isn't chess, not even checkers. You can't even see the other participants, everything is anonymous on the exchanges I've dealt with. Spoofing is illegal, and that is about as clever as it gets. Very little you can do to make your algo better will put it at odds with providing a better quality service.

To prove that, remember the two types of algo. If I write a liquidity provider that doesn't offer a tighter spread than the competition, I will miss out on trade flow and make little or nothing. If I write a liquidity taker that has an opinion that is wrong about the correct value of a security, someone else who is right will smash me. So effectively we are selecting for more liquidity, tighter spreads, and more accurate prices reflecting available information.

You're still missing it. Every single additional limit order in the market creates real value. If I want to buy 500 shares of company A, and one market participant is offering 250 at $10. Another is offering 250 at $11. It is going to cost me 250 x 10 + 250 x 11 = $5250. Now, the "short term speculator" you are complaining about shows up and puts his tiny little order in, he is offering 1 share at $10. Now the equation if you want to execute is 251 x 10 + 249 x 11 = $5249. He/she just saved you $1. One real dollar. As someone who wants to get a transaction done on a market, more people posting bids and offers cannot possibly hurt you. Only help you. You have the complete freedom to decide to meet their offered price, to cross the spread, to pick up that carton of milk at the 7-11 and walk to the till and make the transaction.

Now maybe that little speculator who sold you the share at $10, he goes and works the bid, he advertises that he wants to buy a share at $9. Someone fills him. It take 2 minutes of trading to work to the front of the queue (easily possible with an equity). During that time, he is exposed to the risk that the stock might spike up in price, a risk you are no longer exposed to since you got your desired trade done $1 cheaper and 2 minutes ago. Then if he succeeds he makes his $1 profit. It doesn't always work - maybe 60% of the time the price ticks up and he just breaks even. He pays exchange fees and clearing on both the in and out trades too.

In theory, you could get involved in all of this with your 500 share purchase. Try to work the bid, get a better price, etc... or if you cross you are effectively paying for a service. Every order posted in the market is a service.

How wide are bid/offer spreads these days? How can that possibly be a bad thing?

It doesn't really matter why anyone is in the market. The fact is that they are posting prices to the limit order book, and in doing so providing liquidity and accepting risk and providing a service. The only way the OP could get executed passively is by offering a price equal to or better than the market price, yes? So someone got a better price (or more liquidity at the prevailing price) by him being there.

There are really only two actions in a market, providing liquidity (i.e. market-making) or taking liquidity (including arbitrageurs). Stretching the shopkeeper analogy, any time you put a price tag on an item you are taking a risk and exposing yourself. You slap a price on some bananas and then there is a tropical cyclone (it happened in Australia) and the price of bananas doubles. Someone smart swoops in (call them an arbitrageur), grabs your bananas and goes to the till and you legally have to sell them to him even though they're worth double now. You also have inventory risk, if you're holding a lot of bananas out back then there is the chance that they go rotten.

Every limit order (an offer to sell, or bid to buy) has an inherent risk that you should expect to be compensated for (or you wouldn't do it), and enhances the market. The more competition amongst people posting prices the better, it just means more liquidity (you can buy or sell as many bananas as you want) and tighter prices.

The liquidity taker on the other hand is all about exploiting a misprice. Say, Intel release earnings and are down 10%, but the price of the highly correlated ARM hasn't moved yet, if you are the fastest you can take advantage of some poor person who still wants to buy ARM at the same price as before this information was known. Keep in mind here, a price can move without any trades happening (or goods changing hands). The shopkeepers can whip around and change the price tag on their bananas if they are fast enough before anyone buys them.

It is harder to justify the liquidity taker - what marginal advantage is there to having the price of ARM react to that news in 2 milliseconds rather than 20? They force the transmission of information yes, but why is it better to have it happen a bit marginally quicker? On the other hand, how can you have a realtime market without them? It keeps the providers on their toes, and there will always be someone fastest to react.

Incidentally, providing liquidity must have been the chief function of the OP's algo - 4000 trades a day doesn't work for a liquidity taker that has to cross the spread. He isn't getting 2000 mis-price signals a day to swing at, good enough to justify crossing the spread. DAX traded 120k contracts yesterday, he would have been averaging 2-3% of daily flow.

A strange position to take, I'm guessing you've absorbed it from the media somehow. His machine was in the market enough to trade 4000 times a day (I would suggest passively, or he would have been eaten by the cost of crossing the spread), so he was basically continuously offering a service to the market - an offer to sell and a bid to buy at the price he thought fair. Do you demand to know what value your local 7-11 provides by selling you milk? The value is that you can buy milk 24 hours a day, so you don't begrudge them the 15c a carton they are making. And they probably charge more than your local supermarket for the convenience too. The dirty thieves.

Of course it was silly. Conservation of energy... There is no net gain, they would put energy (food) in to their 'human batteries' but it is impossible to get back any more than they put in. Complete nonsense.

Because the human brains can only communicate and acquire knowledge at a limited rate. This causes the "exponential decrease of discovery impact of each succeeding researcher" that the original post refers to, but he misses the point completely. We can't 'stand on the shoulders of giants' until we've climbed up there, slowly and painfully. An AI can just download the lot. And share any new discoveries with the hive mind.

In Q/KDB:

/1 million random floats between 0 and 10

a:1000000?10.0; b:1000000?10.0; c:1000000?10.0;

sum each flip (a;prev b;next c)

/To do it in parallel

sum peach flip (a;prev b;next c)

Sometimes it is like living in a world where people refuse to use sheet music/clefs/bars and instead they read and write music by listing the notes, their frequencies and how long to play them for. And you can't tell them otherwise. People who recoil at APL because there are a few unusual symbols involved will gladly play a guitar and deal with all sorts of arcane symbols. If we really wanted to do a good job we'd go beyond ASCII.