A much-maligned engine of innovation | Hacker News Reader

A much-maligned engine of innovation

ft.com

57 pointszt34 comments

It is quite interesting how much of the really foundational technology that exists was either funded by the government, by the research labs of monopolies like AT&T or NTT, or by the research labs of companies that had near-monopolies in a particular sector (Xerox, at the time, Microsoft, Google). The things that people hate: governments and monopolies.

In retrospect it's obvious why: R&D takes money, and there is no money in competitive industries, since competition drives profits towards zero.

General (non-patentable) R&D payoff goes to the whole industry but only costs one player; if one player has a monopoly on the industry, they get the full payoff of their work.

In the US the private sector spends several times what the government spends on R&D and that gap has been growing continuously since the 1950s when the US government did spend more than the private sector. Now government funded R&D is a small piece of the pie. Academic R&D has been shrinking in many areas. The biggest growth area in the US is non-profit R&D foundations that focus on areas that all three of the other sources tend to ignore.

Where R&D money is focused is a matter of goals and incentives. Private companies invest in R&D that can add long-term value to their core business and are unlikely to invest in R&D that in an area where no business currently exists (perhaps due to a lack of foundational R&D). Governments invest in R&D to achieve political objectives. In some cases this means investing in foundational R&D that private R&D can build upon. In many cases though, government R&D follows fashion because it has little incentive to return value. It will over-invest in areas with low ROI but great PR value while studiously ignoring high ROI R&D that does not seem as sexy. The growth of R&D by non-profit foundations has been in part to address those areas of R&D that businesses have no incentive to invest in and which are also politically unfashionable such that the government will never invest in them.

Academic R&D has been faltering for some time partly because its role is being better served and better funded by the growing investment in private R&D labs by companies. Consequently, there has been a selection effect where many of the most productive pure researchers have been absorbed by companies that can offer compelling research environments and high pay but it is reducing the average quality of what remains in academia. Importantly, the R&D that is done in private labs is often treated as trade secrets and rarely published, which in many domains has created a growing gap where the work being done in private labs is years ahead of what is being done in academia. I see this all the time for computer science.

The R&D ecosystem in the US is quite complex. Right now, there is a slow, large-scale shift toward private R&D. Ironically, this is enabled in part by how inexpensive and widely available many kinds of high technology have become.

Some things are just so big that it is unlikely a company or startup can start and finish. Society really cannot function without both public and private contributions. The proportions of each determined by your flavour of politics.

Cases in point are things like the British railway, roads, the NHS or NASA. In our own industry it is open source software or things like CERN and the donation of the WWW we enjoy today.

You need both. They tend to flip flop too, we nationalise something because we think that would be better. Then we privatise it and maybe at some point in the future we nationalise it again. We cannot seem to escape that both are necessary for healthy society.

If you want to have expensive ground breaking research done in the private sector, you basically want superprofitable corporations that have at least regional monopolies and power to guarantee their long term security via direct or indirect law-influencing/making/changing powers. If you get to this, then they'll also need corporate police, corporate armies, corporate intelligence services etc., that will also have powers over and provide protections to civilians depending on their corporate affiliation.

You basically want the dissolution of traditional states and their replacement by corporation states - a form of corporate semi-anarcho capitalism.

...and it doesn't sound like such a bad idea actually. We just need to choose between gross incompetence (what governments worldwide have shown us they are capable of) and pure for-profit maliciousness (what corporations showed they are capable of). We just need a 3rd magic ingredient, maybe something like a "global karma system" that could help keep the "profit driven maliciousness" in check. If this magic 3rd ingredient is ever realized, and corporations agree to become more democratic, a switch to the new system becomes possible if desirable.

Over all this, the basic truth is that in order to take on the risks of costly research you need to have either:

1. guaranteed long term and real power (governments and monopolist corporations can have this, less for corporations since the landscape is too dynamic now to guarantee the "long term" part, nor more big Bell and AT&T :( )

2. a reliable way to turn a significant percent of any kind of research "outputs" into guaranteed profit (this is actually an area I'm very interested in and I think I have some ideas, but I'm not sure if I should just"open source" them or wait until I'll be able to "sell" them...)

It doesn't look like most of HN is in the R&D field, but from the inside it's obvious why we need government funded blue sky research. Once you get into the research and science field you realize how many old ideas you are using, ideas which were just developed because of scientists were advancing the state of the art their chosen fields.

The fact is, it takes decades to realize the implications of fundamental breakthroughs in math, physics and chemistry. The long time horizon and inability to judge the worth of these endeavors at the time means a patronage model is basically the best bet. It's kind of sucks but we just have to pour money into research hoping for a better future.

Medicine is a good example of a field that has benefited enormously from publically funded research. In countries with nationally-funded health services, it is often the doctors and healthcare professionals who treat patients who carry out clinical trials and medical research. Sometimes this research is done in partnership with private companies. But without Government support, a huge amount of valuable medical research would never be undertaken.

Erm, and how is that really different from the USA, which doesn't have a nationally funded health service? Well, one could argue it might as well be given that many hospitals are publicly supported, or are even attached to medical schools, and a large amount of health care is funded through medicare?

And the USA does tend to do much more medical research than anyone else (most of it publicly funded).

Government doesn't have to care about risk or tangible profit, and they have a longer time horizon than most business can afford. Look at the space race, for example; the potential gains were purely political and the project certainly stood no chance of ever recouping its costs directly. But the technological by-products are things we now take for granted and which have spawned whole new industries.

I'm replying to a review, so this is perhaps addressed in the book.

1. Couldn't public research simply be crowding out private research?

If you're a company and you can use research someone else paid for ... why wouldn't you? Once that model is established, what incentive do you have to spend above your peers?

2. Is the finding stable across time and different countries?

Public funding of research grew enormously in the 20th century alongside the growth of the state and in reaction to various wars (hot and cold).

What did the mix look like in the 19th century? The 18th? What did it look like in Qing China or in late Republican Rome? What differs between Venice at its peak and the glory days of Islamic civilisation?

3. Is much research simply more expensive now?

Requiring funds that only super-profitable companies and governments can muster?

4. How does research vary between industries and problems?

Some kinds of research are more difficult or expensive than others. Is that why there's a surfeit of software companies appearing in the private sector while almost all biomedical companies grow out of government and university laboratories? How does a small business invent a supercollider?

I guess if I combine all these questions, my underlying question is:

5. Is the current configuration causal or just a surface feature of the current background environment, from which it is problematic to extract generalist statements meant to hold true for all time?

My own suspicion is that some kinds of intellectual output are just not very highly demanded in a general market. Patronage, whether by medieval nobility or modern states, is a form of demand that has given us a lot of public goods.

1. Couldn't public research simply be crowding out private research?

It depends. There are plenty of examples of research that private companies don't want to do that the government is prepared to do.

Governments have the benefit of being able to take on lots of risk and have a very long term perspective; maybe we'll get to Mars in 2040, a private company couldn't wait that long and spend that much.

The only companies who have been successful in long term research have had some kind of market dominance that was eventually tempered. E.g. AT&T Bell Labs or Microsoft. There are arguments out there that companies shouldn't become successful enough to do long term research, it doesn't really fit into a healthy dynamic market. Then you have companies like Google who are very successful in leveraging publicly funded research (self-driving cars...).

spacex?

oh right, government funded ...

Yes, and, it's an "attention economy."

I call a field "fully subscribed" when it has enough money and researchers to make full progress. It's over subscribed when you are far into diminishing returns. And of course, under subscribed fields lack everything.

What is under subscribed today? Electric cars less than they were, but perhaps zero energy homes ...

The UK government has a requirement that all new homes will be carbon neutral by 2016, so I imagine there will be some money appearing for research in this area.

There are a couple of caveats:

1) You can build homes which have worse energy efficiency than the requirements if you offset it somehow, e.g. by improving existing homes.

2) The definition of 'carbon-neutral' has been diluted for these regulations. For instance, it doesn't include appliances that you plug in.

I was speaking from a US perspective. I think we lag. We probably could import that tech of course ... and I was really grasping for under subscribed fields. Any other suggestions?

The author is more of an economist than a historian, so the book probably won't take a satisfying angle in answering your questions, except the non-temporal part of #2 and perhaps #4. Some of her previous work that she cites in the book has looked at the relationship between industries, countries, sizes/stages of companies and R&D, and different kinds of state funding structures, both in terms of what factors seem to influence money being put into R&D, and whether there's empirical evidence of growth coming out of it.

The main thrust of the book seems to be to defend some current thinking in economics to business and public-policy people (most of the book's points are summaries of existing studies, some by the author and some by others). For example she has a section summarizing a set of views on why the U.S. outpaces Europe in innovation, and criticizing what she sees as some incorrect popular views on that question.

The book is less aimed at first-principles questions and more at questions like, "does public-private technology transfer work differently in the U.S. and Europe?" and "how are R&D risks currently distributed among different funders in the U.S.?" To the extent there's a headline thesis it's that the public-sector role in modern innovation provides some critical features that are deeply enmeshed into successful networks of innovation.

note: This is based on browsing some of the author's papers and skimming through the book, not a deep read.

What is emerging, then, is not a truly symbiotic ecosystem of innovation, but a parasitic one, in which the most lossmaking elements are socialised, while the profitmaking ones are largely privatised.

Yes, this is a real problem, especially in biochemistry right now. It's not that the intellectual property moves over (because government can't own a copyright) but that the people do. There are a number of cases where a researcher in the public sector discovered something and, instead of putting it into the public domain (where it belongs, since the government paid for it) he went to work for a pharma company on a 500k+ salary, which is essentially a finder's fee for IP worth $100M++ to the company.

I don't know that this can be solved completely, but I think that (yes, I am going to advocate for more government spending) salaries should be raised to be comparable with, at least, the industry average in the private sector. (Obviously, government can't match the private-sector high-end.)

I agree with the general insight of the article.

Now, there's a second-order effect that is poorly understood but extremely important. Startups were more innovative in the 1970s than now. Why? Bell Labs and Xerox PARC were in their heyday. The public sector paid only slightly less (10-20%) than private. Academia wasn't the jobless shithole it is today. To get people to leave their jobs in basic research, companies had to compete on autonomy and interesting work. The result was that, even to get in the game, people had to come up with more interesting and better companies.

The general lack of autonomy for technical people in 2013 is why the Zyngas exist today. There really isn't that competition. Sure, the large tech companies offer free lunches and one massage per year, but the work is usually so managed and uninspiring that they aren't putting up much of a fight.

Doubling down on high-autonomy, basic research work, no matter who does it, government or private, is good for society. It means that the startups that do claw their way into existence will have to be better.

The state doesn't actually spend that much on R&D, most of it goes to various welfare programs and the military. Getting into the whole internet, GPS stuff, we should just spend lots of money on the military because they fund a whole tonne of innovation, oddly enough because they receive a whole tonne of money.

R&D is a byproduct of massively wasteful state spending, putting more money into government will not result in more R&D spending...

I'd hope that an entity that spends 20 to 30 percent of a nations GDP was responsible for at least a little bit of innovation.

Whilst I agree that the state is often wasteful of resources, it is often more efficient and able to fund riskier ventures that result in longer term results.

As for a "little bit of innovation": the Internet alone is not what I consider to be a small innovation.

"The internet" was hardly created by government funding, yes, DARPA funded TCP/IP and CERN funded Tim Berners Lee, but if you're going to ascribe the entire internet to the government because they funded the initial R&D that led to fundamental pieces of it being developed then you might as well ascribe the internet to Bell Labs because they invented the transistor.

Why not say that ordinary people are the funders of all this tech because if they didn't pay taxes, the state would not exist...

As with all things all these achievements were possible because of the giants who came before.

That's what's so stupid about the entire article. You could just as easily make a case that everything we have today is the result of gentlemen scientists from the renaissance. Or that Alexander the Great is the inventor of all of western civilization.

The government role in the internet was considerably deeper than just funding the early protocol work, though it was definitely not the only player. It's a fairly good example of the public/private innovation networks thesis the book being reviewed here is arguing for. Abbate's book Inventing the Internet (MIT Press, 1999) has a pretty good account of all the various players and how they related.

Not to mention, it's trying to put paid to all those who think that th government shouldn't be involved in research. There are a hell of a lot of those sorts out there.

> That's what's so stupid about the entire article. You could just as easily make a case that everything we have today is the result of gentlemen scientists from the renaissance Or that Alexander the Great is the inventor of all of western civilization..

No, the article does not overgeneralise, your comment does.

The proposition is that a substantial proportion of particular quite well-defined 'innovation' was created through state systems as opposed to private commerce.

If, in order to argue against that, you have to generalise so much that one can almost no longer say anything causes anything, because everything causes everything, you end up saying nothing. You have not refuted something, you have just made up a more meaningless way of measuring anything.

Be careful of reading claims into the article that it doesn't actually make. You don't have to believe 'all government spending is productive' or 'government should spend more money across the board' - claims the article does not make, and with which I, like you, would disagree - to agree with the claim 'government R&D spending is productive and we should be doing more of that', for which there is strong evidence.

Governments are most effective when they are competing, just like private companies; e.g. USA vs. Germany/Japan in WII, USA vs. Soviet Union during the cold war. These days, we are kind of at peace with countries...even China is just a big finicky panda bear right now.

We've replaced are bogey men from a well-defined Soviet Union to a bunch of people in caves who blow things up randomly with IEDs. Well, this war has given us great drone technology at least, but competing with people in caves doesn't give us as much.

It would be nice if we could replace our obsession with competing with each other to achieving hugely ambitious goals like colonizing the moon or Mars. I'm doubtful this is part of our collective nature, though.

This is the Neil Degrasse Tyson argument. Though I like it, I kind of really prefer the Bill Gates argument on healthcare and education, and think its the more important one, that everyone can come together on...not just the nerds.

And I am not as doubtful. Our collective nature resulted in the internet. It is only as imperfect as we are. But it also represents collaboration between people on a scale never see before. And that is a cause for hope.

Virality on our social networks can be used, and I am sure it will be used, for more than just getting Psy to the top of the charts or Obama elected. Its just a matter of time.

>Virality on our social networks can be used, and I am sure it will be used, for more than just getting Psy to the top of the charts or Obama elected. Its just a matter of time.

Greater errors have been made in the understanding of human nature; think communism and libertarianism. The technocracy is a bit more reasonable, but we are still far away from it. We need some crisis to drive radical social change, and for all the crying right now, we are pretty comfortable.

Veto sign-in-required articles on hn.

Of course the article is correct about US Federal Government funding of DARPA and NIH (and NSF) being the major source of research and innovation.

One question is why, that is, why don't private industry, VC funded startups, etc. do more with applied research and innovation?

My view of the main reason is that private industry, especially nearly all of finance for the funding (YC is an exception), are not qualified to evaluate such projects. So, such organizations don't. Likely also hurts is that the VC limited partners are likely much less well qualified to do such project evaluations than the VCs and, thus, insist on criteria for VC investment that rule out evaluating applied research or anything non-trivial in technology (or making significant use of such evaluations in funding decisions). Instead, the LPs, and, thus, the VCs, mostly want to see 'traction' in the market.

In the past, that approach worked because a VC could be fairly sure that a good project that did get some good traction would still knock on the doors of the VCs for 'go to market' or 'expansion' capital (forgetting that insisting on only such late funding would reduce the number of projects that did get to such traction). But now, as in PG's latest essay, can do some information technology (IT) startups for less than the cost of renovating a bathroom. So, with such low costs, and likely low burn rate, we should, in time, see some major IT startup successes that never took equity funding and were just 'bootstrapped' and have 100% ownership by one person.

Likely another reason is that people with the backgrounds to get funding by DARPA, NIH, or NSF are quite reluctant to pursue startups.

So, net, it looks like there are some especially good chances for Ph.D. degree holders with money enough to renovate a bathroom to do a startup and become a billionaire! We will see.

VCs are overwhelmingly finance people, not people with technical backgrounds. Meanwhile, almost every DARPA program manager, the people who make funding decisions, has a PhD.

DARPA does a lot of things they won't generate a return, often because much of what they do can't be productized. They flail around in a field and reveal to industry what works and what doesn't. Finance guys would never do that--the near term returns aren't there.

My take is that overwhelmingly VCs finance projects with 'traction' where the traction can be measured quantitatively. Thus this measure is a surrogate for traditional financing criteria in terms of revenue and earnings.

The problem sponsors at DARPA, NIH, and NSF can evaluate technical projects and fund based on such an evaluation. Funding is not easy to get. For US DoD projects intended to result in something practical, the batting average is quite high, much higher than in VC.

Yes, a major difference is that DARPA, NIH, and NSF fund projects intended to have a technical accomplishment, maybe with an actual product. VCs want projects that make a big splash in the market, and that is an additional difficulty.

One solution for VCs is to fund technical projects aimed at creating new or much better solutions for what is clearly a "big ass" problem in the market so that if such a solution can be found then there is essentially no doubt the solution will be a "must have" in the market and make a big splash. VCs don't like to think this way.

As reported by Fred Wilson in his blog, over the past 10 years, on average VC returns have been poor, e.g., less good than the S&P. There is other strong evidence of poor VC financial returns. VCs aren't doing well so that whatever they are doing should not be well respected.

A problem with the VCs, again, is that they can't/won't evaluate technical aspects of project proposals, stay with 'traction', and won't consider technical accomplishments, even if done, even if as peer-reviewed original research, in funding decisions. Net, they won't fund technology, even working and ready for market; instead, they fund 'traction'. Did I mention that they want to fund 'traction' -- that's no joke.

For funding "people", sure, people with a project with traction if the people are not just totally wacko.

The other reason that finance people typically won't go for blue sky stuff is that they need to cash out within 10 years; a few years to select and kick start investments, 5 years of hard work by investees, then a few years for cash out, IPO or sale (preferably cash). There isn't the timeframe in there for R&D.

I never said "blue sky".

VCs won't fund research if it's done, written up, in a peer-reviewed journal of original research, and implemented in software written, tested, timed, and documented. Period. No research, of any form, any time, in any case, period. Stop. End.

Instead, VCs, and their limited partners, want to see 'traction' in the market.

If the traction is from some research, then that's a net negative because (1) it would be something it might be important to evaluate, more difficult to evaluate than just routine software, (2) might make the CEO more difficult to replace, (3) would let the CEO have an easier time telling the Board that they don't understand the business.

Instead, VCs want a nice, simple, little business, with traction significant and growing like a weed, in a gigantic market, that can become a company worth $100 billion, that anyone, even a VC, could understand, run by a CEO who tends to drool a little and can easily be talked into signing a bad business deal.

"Blue sky" has nothing to do with the situation.

Reply on news.ycombinator.com