Why the Dow is a useless market index | Hacker News Reader

Why the Dow is a useless market index

erikrood.com

129 pointsqwerty202063 comments

Relevant episode of Planet Money:

http://www.npr.org/sections/money/2017/01/04/508261371/episo...

"It's no secret that we here at Planet Money think the Dow is a terrible economic indicator. We don't like that it only looks at thirty companies. We don't like the way it does its math. We think it does a bad job reflecting the overall economy. Honestly, we're not sure why everyone is still talking about it."

> Honestly, we're not sure why everyone is still talking about it.

It's like telling a compulsive gambler their favourite “system” won't predict luck. They just don't want to hear that.

Not really, it's not predicting the future like a gambling system. It's just a measurement people are familiar with because they have reference points in mind.

The phases of the moon are also a familiar measurement system for which people have reference points in mind.

The phases of the moon are pretty good for predicting moon rise, moon set, time of month, days of the year, days, and the tides.

Terrible? Hyperbole. The 30 companies are pretty big and relevant, so still it works out well. Also, the DJIA pays a sightly larger dividend than the S&P 500. Also it's not an economic indicator.

> it's not an economic indicator

Can you explain this statement? It's almost synonymous with the stock market, and is one of the most common headline numbers that people mention in business and economy news reports.

The stock market is not the economy.

Capital markets are crucial indicators of economic activity

Only because we use them as indicators of economic activity.

When stock markets plunge people panic and that panic is what causes economic instability. That's what causes issues. Economics is purely a product of human minds acting in unison or opposition.

I think it's the other way around, no? People panic, and then sell their equity.

I think it's both. That is, it's a feedback loop. Some people sell their stock, so the market declines, so more people sell their stock, so the market declines more, so more people sell their stock...

For this to happen, though, enough people must be thinking that the market is poised for a big fall, so they're on a hair trigger, ready to sell right at the start of the drop.

It basically is. It represents the state of capital investment, which as it is turns out is important in a capitalist economy.

The stock market would not be the economy in say, a Marxist system.

For a counterexample see Brexit. The FTSE 100 went up after Brexit, when other economic indicators were negative, because it primarily contains large multinationals which are insulated from the local economy.

So a stock market index is only correlated with the local economy if there are few multinationals.

Well, arguably the FTSE went up because the pound plummeted.

The DJIA is just the sum of 30 stock prices, divided by a "divisor" which is adjusted to keep the value constant when a company is added or removed. The amazing thing is that it's quite useful. But why?

One reason is a long-standing tradition that stocks should be priced in the range 10 to 100. This tradition stems from old ticker system limitations, and it's weaker than it used to be, but it still has some force. Stocks which get well above 100 usually split (yes, there's BRK), and stocks which get below 10 are usually considered to be in trouble. Below 1, they're called "penny stocks" and get delisted from the big exchanges.

As a result, successful companies tend to be near the top of the 10-100 range. This weights them higher in the DJIA. Right now, here's the list.[1] Goldman Sachs is the only one above 200, and they really ought to split. They might get replaced in the DJIA if they don't. Nobody is below 10; GE, at 23, is at the bottom.

[1] http://money.cnn.com/data/dow30/

Fascinating. Thinking about how GOOG is $824 right now, and how it's not in the Dow, I found:

"This quirk kept Apple from joining the index until recently. The world's biggest tech company, despite its massive influence on the economy and the markets, was excluded from the Dow Jones Industrial Average because its share price of about $600 would have given it outsize influence on the index. However, when the company split its stock 7-for-1, it became a viable Dow addition." [1]

[1] https://www.fool.com/investing/general/2015/05/18/the-30-dow...

>So, next time you hear someone talk about the Dow in any serious sense, kindly point them in the direction of an index weighted by market capitalization (e.g. S&P 500 or Wilshire 5000),

The most commonly used index - the one that is shown in the news or discussed with your Uber driver. should be index that provides essential information for the average person.

I would advocate pointing people towards total return indexes of S&P 500 or Wilshire 5000. Index's total return displays a more accurate representation of the index's performance than price index and it can be used directly to benchmark stocks to other investments.

Surprising amount of people are not aware that stock price index is not accurate measurement of stock market performance over longer periods of time.

http://imgur.com/a/uHRZ9

> The most commonly used index - the one that is shown in the news or discussed with your Uber driver. should be index that provides essential information for the average person.

Seriously, why? Those discussions are no different from discussion of sports scores, and equally as relevant to the decisions people make. So who cares if they are talking baseball, the dow, sunspots, or astrology? Or Apple vs android?

(BTW I realized the Dow indices are nonsensical as a teen ager, so am not defending them in the slightest)

The big difference is that you can buy a low-cost S&P 500 fund, and this is a perfectly reasonable thing to have in your retirement account. You will know, therefore, if the SP500 index goes up 1%, your retirement fund has done nearly the same.

It's very important when people save for retirement and think about different investment options.

There are all kinds of invest in gold snake oil salesmen around.

But they shouldn't be managing their investments on a day-to-day bases much less basing decisions on conversations with random uber drivers.

I wouldn't go so far as to call it useless - anachronistic perhaps, but useless no.

When the Dow was first calculated, real time market capitalization for individual companies wasn't a thing. Prevailing market price was a decent enough proxy that Charles Dow could make an index of leading industrial firms out of prices (and price changes) alone.

As others have pointed out, over a long enough time period the Dow Jones has a high correlation with market cap weighted indices. Its annual volatility has been about 1.5 percentage points more a year, but average returns over any reasonable holding period are barely different than, say, the S&P 500.

There's an even better argument against the Dow than the price weighting though - the somewhat arbitrary company inclusions. One of the more interesting pieces of history is IBM's 40-year 'vacation' from the Dow. If IBM had stayed in for the 40 years after 1939 you could tack on another 5 figure number to today's index price.

> I wouldn't go so far as to call it useless

At this point, it's popular art. For any purpose one might use the Dow, the S&P 500 is better.

Any purpose? What about curmudgeonly criticism?

> When the Dow was first calculated, real time market capitalization for individual companies wasn't a thing

How is this possible? People were buying shares without knowing what fraction of the company they represented ??

Probably he's referring the "real time" part. Electronic ticker tapes were invented in 1867, and were relatively prevalent by the 1870s. (The Dow was first calculated on May 26, 1896.) But even still, for average investors, most information back then would have traveled at horse, train, or steamboat speeds.

Okay, but wouldn't that also apply to prices?

Certainly it would. I'm not exactly sure how a longer-term investor would have gone about buying stock then, but probably there was a broker in their city that had a big ticker board showing the current prices (as quickly as some kid could write them up on it as they came in from the tape). And you'd walk up to a window and place your order, and just like today, you'd actually pay whatever price your order was filled at back on the exchange's floor (well, today it's an ECN, but you get my point). Eventually, your broker would deliver physical stock certificates to you.

For shorter-term or smaller transactions, bucket shops were very common. https://en.wikipedia.org/wiki/Bucket_shop_(stock_market) There's all sorts of shenanigans that went on with those - they were more like bookies; you never actually owned the underlying securities you were speculating on.

It's also worth remembering that things were vastly less regulated and insider trading/outright manipulation were very common (the SEC didn't come into existence until 1934).

Good reading on this time period if you're interested: https://en.wikipedia.org/wiki/Reminiscences_of_a_Stock_Opera...

More in the sense that we don't realize how spoiled we are (and how the standards have improved). I don't even have to leave this table to see the effects of share issuance or buybacks from a public company, and it's all updated in close to real time for me.

It's not that you couldn't make a _pretty-reasonable_ estimate, it's just that the ecosystem is much improved and easier to roll up for the indices we follow today. The Wilshire is from the 70s and the first flavor of the S&P came 30 years after the Dow (and was 'only' 90 firms). If you read Security Analysis (first edition: 1934) you can still see some of this in action; it mentions how only some statistical services (paid!) would calculate/estimate the current number of shares outstanding.

I don't know what is "updated in close to real time" for you, but normally companies disclose the number of shares outstanding and details about buybacks programs once per quarter.

They had a good estimate, but not in real time. All of the information was delayed.

(As I said on the other comment) wouldn't that also apply to prices?

Yes.

It still does, actually, but on a much shorter delay. If you make a market buy order with an online broker, it will execute pretty much instantly, but the price might have fluctuated between the time you pressed the button and the time the order was filled. It usually makes little difference, but people have ended up paying a lot more than they expected on some rare occasions.

This is common enough with ETFs that some brokers advise their clients to always use limit orders.

Do you think people buying shares now know what fraction of the company they represent?

It's not ideal but to call it useless is arrogant and immature. It's a decent indicator of how the blue caps are doing in the markets. SP500 is arguable better but the DJI isn't useless.

No Google, No Amazon. Its an arbitrary choice of 30 companies out of the top 100 market caps based mostly on stock price and whose friends are on the board.

I think Hacker News could come up with a better list of 30 stocks with market cap over $30B.

You're not disagreeing with what I wrote.

That's the reason why Berkshire Hathaway is out of the index, while based on the market cap, it should clearly be in (~400B).

That being said, the correlation between DIJ and S&P is close to 100% so for all practical reasons (from a long term investor point of view) it should be OK.

I see a couple of problems with these arguments. First, the assumption that market capitalization is more important for weighting a market index than share price. I'd argue that if your goal is to measure the health of the overall stock market, companies that control the most money should be weighted not much more than less valuable companies. Just because Wal-Mart, Amazon, and Costco are doing great, doesn't mean that JCPenny, Macy's and smaller retailers that employ hundreds of thousands of people are doing great. Giving more weight to smaller companies introduces more companies than just those at the top of the value spectrum. A company with massive resources can probably weather certain negative economic variations better than smaller companies, so to focus solely on those to the exclusion of all others seems to be willfully ignoring a significant part of what comprises the overall market.

The other issue is that even though the S&P is cited as being a much better index to track the market, the fact is that the Dow and the S&P correlate almost identically historically. So even if the Dow is reasoned to be much worse than the S&P, history shows that they both reflect basically the same thing.

I guess all the Dow futures and options traders should just pack up and go home? Dow trading is as useful as any index with volume in analyzing herd behavior. It's a far better prospect than analyzing any of it's individual components in isolation.

Not so useless: you can construct hadge fund-like trading strategies with the DOW and SP500.

Both DOW and SP500 have some tendency to go up and down at the same time in bull and bear market times.

When this happens you can (for instance) short DOW and long SP500 and your losses in the sort side will be covered by the gains in the logn side...

...but they will be uncorrelated for short periods of time. So, if your short possition goes down while your long is constant or goes down much slower than your short, then you're making money.

Probably you can make this kind of hedging with very little money (or almost free) using the fee you get from going short to buy the long leg of the spread.

So all you need to do is predict the future to know which way to go long/short, and... profit!

Ive heard similar complaints for years and it makes sense to me - what doesn't is why alternatives arent used or created? I don't understand everything about stock markets or certainly their politics but it seems on the surface that if you had a better indicator of the economy itd be pretty easy to show its usefulness. And with historical data you should be able to replicate a new index and measure its effectiveness.

The S&P500 is a fine index. People just use that instead.

tldr: The Dow 30 index is a bit absurd, however it's not as bad as it seems because of portfolio theory.

The Dow 30 correlates highly with the S&P500 index over medium-long periods of times (approx more than a month) [1]. This correlation is due to the fact that the 30 stocks that are selected are arbitrary but not random. They are selected by a human process which considers which companies contribute an important factor to the overall market [2]. Of course this process will be imperfect because it's vague, however it gets pretty close.

Don't forget, according to findings from portfolio theory, it's possible to replicate an index with only a small subset of the components of that index [3]. There's a whole field of work describing how to do this [search google for 'replicating index'],

In fact, replicating an index with a small number of stocks is exactly how 'smart trading' companies like wealthfront and bettermint make money. They find multiple non-overlapping subsets of the S&P500 which correlate strongly with S&P500 performance. Then they buy subset A, and if a loss is incurred in any month, they sell subset A and buy subset B. Since A and B are highly correlated with each other (ideally r>.99) and with the S&P500, the investor should expect the same return in the future, however can book a capital loss which creates a tax refund. Shifting from subset A to subset B avoids a wash-sale allowing the capital loss to be recognized [4].

The Dow 30 is certainly not an optimal index, however it's a case study in showing how selecting the "top" companies for some reasonable definition of "top" will earn a return approx. equal to the market (represented by S&P500) no matter the strategy (in this case price-weighting instead of mkt cap weighting can be considered a strategy).

The reason it's cited so widely is that there's a lot of historical data on the Dow 30, whereas every other index is less reported.

[1] Yahoo Finance comparison of Dow 30 vs. S&P500 chart http://finance.yahoo.com/chart/%5EDJI#eyJjb21wYXJpc29ucyI6Il...

[2] Dow 30 index is built and maintained by a subsidiary of News Corp. https://en.wikipedia.org/wiki/S%26P_Dow_Jones_Indices.

[3] http://www.etf.com/etf-education-center/21038-how-to-run-an-...

[4] http://www.investopedia.com/terms/w/washsalerule.asp

That isn't how the robo-advisors actually implement tax-loss harvesting. They just buy slightly different index funds.[0]

E.g., VTI vs SCHB. VTI tracks the "CRSP US Total Market" index, while SCHB tracks the "DJ Broad US Market" index (note, this is not the same thing as the DJIA). Those indices are not comprised of disjoint subsets of the total stock market.

Additionally, large index funds have enough capital to just buy all of the components (100s-1000s) without sampling. Thirty funds doesn't really do it justice. It can also miss out on big winners, which is where lots of growth comes from.[1]

[0]: https://research.wealthfront.com/whitepapers/tax-loss-harves... [1]: http://www.efficientfrontier.com/ef/900/15st.htm

I believe this is not allowed by the IRS because ETFs tracking similar indices would be considered 'substantially identical', but I'm not sure [1]. If it is allowed, then you're right - this would be the most efficient way to implement tax loss harvesting.

  A wash sale occurs when you sell or trade stock or securities at a loss and within 30 days before or after the sale you:
 1. Buy substantially identical stock or securities,
 2. Acquire substantially identical stock or securities in a fully taxable trade,

 3. Acquire a contract or option to buy substantially identical stock or securities, or

 4. Acquire substantially identical stock for your individual retirement account (IRA) or Roth IRA.

[1] https://www.irs.gov/publications/p550/ch04.html#en_US_2016_p...

Different indices are not substantially identical.

Different ETFs tracking the same index (e.g., SWPPX and VOO) are likely to be determined substantially identical. (I don't know that the IRS issued guidance on this yet.)

Yes, they choose different indices precisely because of wash sale rules.

Agree that the Dow 30 correlate reasonably well to the larger market, but it's the price-weighting piece I have a major problem with. You have to admit that the index would be better off weighing its components based on actual market value rather than share price. Fortunately, as the scatter plot shows, the under/over representation isn't that bad, but still ridiculous for the most widely cited market index nonetheless.

That is what the S&P 500 does. No one stopping anyone from buying the individual stocks of the Dow in a market-cap weighted manner

This is completely incorrect. Betterment, for example, adds nothing that individuals can't do themselves, simply by rebalancing and tax-loss harvesting into similar but not substantially identical ETFs. This way of tax loss harvesting is better because you retain the high diversification of 100s of stocks.

loeg points this out. Wealthfront doesn't do this either.

This is only true for small amounts of money.

  Instead of using a single ETF or Index Fund to invest in 
  U.S. stocks, Wealthfront's Tax-Optimized Direct Indexing 
  directly purchases up to 1,001 individual securities on 
  your behalf — up to 1,000 stocks from the S&P 500® and S&P 
  1500® Indices and an ETF of much smaller companies.

From Wealthfront's website: https://blog.wealthfront.com/introducing-direct-indexing/

Analysis from bogleheads: https://www.bogleheads.org/forum/viewtopic.php?t=152903

Thanks, good point.

It is nice to see someone dig deeply into what the numbers actually mean, but it is surprising to see them come away with the wrong conclusions.

The DOW isn't useless, look at this graph : https://www.google.com/finance?chdnp=1&chfdeh=0&chdet=149177... which is a comparison of the DOW, S&P and NASDAQ indices. Note how closely the DOW and S&P 500 track each other? If the DOW was 'useless' as the author surmises, and the S&P 500 is 'better' then why are they so closely correlated?

The answer is of course that membership in the DOW (and S&P 500) changes over time. The selection of companies and their weighting is done by people who look at the impact a particular company has and how its stock price 'leads' or 'trails' the market. You pick 30 of those and they become your 'signal' for the health of the market.

For short distances, the pre-Newtonian "ball flies straight and then drops vertically when it runs out of oomph" math approximates Newton. It's close enough, except when it matters. Unlike Einstein, those conditions arise regularly. It's as easy to calculate Newton as its predecessors. The predecessor is useless.

The Dow and S&P 500 are as easy to access. (They're similarly intensive, with a computer, to calculate.) When they diverge, the S&P tells one more. (The Dow only talks to industry, for which the relevant SPDR is better.)

It's uselsss. (We loved it on the trading floor, though. Retail would buy based on the Dow. Free profits.)

   > It's uselsss. (We loved it on the trading floor,
   > though. Retail would buy based on the Dow. Free
   > profits.)

This is almost like the Yogi-ism "Nobody goes there any more, its too crowded." Because in the same sentence you say "its useless" and "it gives you free profits" (which is useful), much like "nobody goes there" and "its too crowded" (hence lots of people do go there).

I don't disagree with the assertion that as a trading signal it is useless, there is much better information around where the market is going right now, or perhaps in the short term future. But as an investing signal it serves the purpose well, with fewer stocks than the S&P 500.

My parents for example received an inheritance and wanted to invest it, so I suggested an S&P 500 ETF fund. Doesn't cost anything for them to sit on it and when they hear the radio say "The S&P is up 10% for the year so far" they know that their investments are also up about 10% for the year. For a while I had a DOW basket, actually holding shares in the 30 companies in the index. When the DOW hit an all time high I knew my basket of stocks was worth more (as a group) than they had been in the past.

Changes in the index did not cause my parents to make trades, instead they just reflected changes in value of that portion of our investment portfolio. And for that it continues to be useful.

It is useless in the fact that you can just use the S&P numbers, with no extra cost of downside.

If you had a number that almost represented the temperature now, and one that represented it a lot closer, and it was free to use either one, then why use the slightly more error prone one?

Your graph tracks changes over a 3-day period. Look over a 5 or 10 year period and the separation between the DJIA and S&P 500 would be far greater.

Interesting, the link doesn't go to where I had it set. I had selected 'all' on the top line. It might take a refresh to see it, and it takes the indices from 1970 to the present (approximately) the distance between the S&P and DOW indexes fall on top of each other in that view nearly everywhere.

S&P 500 and the Dow are highly correlated (95% over the past 50 years) and so as an overall temperature of the market it's typically sufficient.

Yup. Completely useless index.

Not sure how this got to the front page. This is not new information or even that interesting.

Literally, hundreds of articles about this exact same thing

https://www.google.com/webhp?sourceid=chrome-instant&ion=1&e...

another brand new site up-voted to the font page...created in 2017 and only 8 very sparse articles

erikrood.com

Reply on news.ycombinator.com