aaronwall

Those of us who are in marketing, publishing, & start ups are not generally an accurate reflection of the general market. We tend to be far more aware of advertising than a typical web user.

Most searchers are unable to distinguish the difference between search ads and content. From a number of surveys we did here http://www.seobook.com/consumer-ad-awareness-search-results

"Even directly after viewing a search result with 3 ads in it, most users are uncertain of where ads may appear, what color the ads are, and if the search result even had any ads in it!"

The ads are just another cost to the business, so that flows on through to the buyer, either through:

- higher prices

- worse customer service

- fewer features

- the company eventually going under as they get priced out of the ecosystem

You won't find many subjective comparisons & reviews of players in the ads (it will all be companies selling themselves) and you probably won't typically find any free or open source software either.

I am with gcarswell on this one. +1 to her!

I don't think search is an easy problem. And (where it is not conflicted by self promotional interests) Google does an amazing job of indexing, filtering & scoring.

It is nearly unbelievable how good Google is in many areas. But that is also the problem...Google set the bar for itself rather high through its own performance.

A gold medal sprinter doesn't get a pat on the back for running a 23 second 100-meter dash.

When most companies put out a product they have to make it better than existing products to win marketshare. They can't put out something sort of average and then just arbitrarily promote it in the results through bundling (the way Google has with things like Checkout, places, product search, flight search, and +)

In the past Google did a much better job at search when it wasn't actively subverting itself.

If I searched on Google 3 or 5 years ago I didn't see Avis-rent-a-car ranking near the top of the search results for "Las Vegas hotels" ... it was only after Google decided to displace & monetize the organic results that Avis started ranking on hotel searches.

http://www.seobook.com/images/hotel-price-ads.jpg

And who's fault is that?

Google's.

If the monetary incentive wasn't there, I am sure Google wouldn't be polluting their own search results, especially as they have put thousands of man-years in trying to do the opposite.

~~~

I also think the suggestion that a person should switch their default search engine is a bit inauthentic for 2 reasons:

1.) Google is spending BILLIONS of Dollars per year buying search & browser distribution + buying default search rights on 3rd party browsers

2.) even if I change my default search engine that personal choice wouldn't impact how other users search & for those other users the copy of copyright content hosted on Google.com will often still outrank the original source.

>> checking for stolen content during the ranking process<<

>Google should check for stolen content during the ranking process, huh? That's a cool idea. How exactly would they do that? They find 5 web pages that all have the same content. How would they figure out who actually wrote it?<

Duplicate content filtering is nothing new.

Scraper sites (mostly Google funded) have been around even before Google+ started behaving like a glorified Mahalo-esq scraper site.

Document first crawl date is a great data point. And if a document links to another document with nearly identical content on it, then the document that is being referenced should probably rank higher, especially if the document was created earlier.

>Perhaps if they had some form of trusted identity service, where you knew what a person's real name was. And if you could mark up your web page with attribution tags of some sort, so that they knew it was you who had said it.<

You don't need real names to allow the link graph to work.

If the proposed service was NOT SELF-SERVING GARBAGE it could use the signal from that network to act as a signal to help rank the rest of the web, rather than sucking content into that network and OUTRANKING THE ACTUAL LEGITIMATE ORIGINAL CONTENT SOURCE.

>But what if multiple people all do this, with the same content? Well, then maybe they would have to invent some kind of social graph, so that they could see which sources you trust most. Is it Linus Torvalds or Deeke McSlayton that you think is the most likely author of this post on the Linux Kernel?<

And, once again, if the service wasn't SELF-SERVING GARBAGE it would create a signal that would apply to the rest of the web so that the ACTUAL LEGITIMATE ORIGINAL CONTENT SOURCE ranked before the house-hosted copy of it.

If I pull a chunk of content from a document and post it to Google+ it is pretty easy for you to put that citation in the SERPs (sorta like what is done with the Google+ votes on AdWords, http://explicitly.me/how-to-get-a-celebrity-to-endorse-all-y... but maybe in a less sleazy way).

>And maybe when you search, they should rank results by how close someone is to you in your social graph<

I have seen Google+ scraper pages that were not in any of my social circles outranking original content sources. That is part of what makes Google's behavior so outrageous.

>Oh wait... That's what Google+ already is.<

Except it's not.

Google was already one of the most heavily linked to websites before launching a social network & now Google+ is a PageRank funneling scheme http://www.thegooglecache.com/white-hat-seo/how-to-buy-11788... built around coming up with a cheesy excuse for Google to outrank original content sources for their content. If it were legitimate then the original content source would outrank the scraped version hosted on Google+

I realize that things are "not spam" when they are done by Google, but if the mechanics & impact are the exact same as what a spam scraper site does then the quacking animal is a duck.

The core issue isn't just how one free Google service interacts with another free Google service, but how when combined they undermine the copyright & revenue potential for a content creator by displacing that person's work with Google's copy of that person's work.

If you are not in business (and don't care if your favorite non-Google sites eventually disappear) then it is no big deal.

For everyone else it has the potential to be a big deal.

If publishers do not complain about it now then it will get worse. How bad does it have to get before the complaints are valid to you?

"If people legitimately like Google+ posts more than anything else on the internet, how would you have them behave?"

So are you suggesting that might makes right or that if a site is popular it should be able to outrank the original content source just because it is popular?

Using this theory shouldn't Google check to see if Facebook has a similar post & rank that above the Google+ post whenever possible?

~~~

More seriously though, Google allows Google to create "relevancy" signals that are unique to certain pieces of content hosted on Google.com (or, at a minimum, over-represented among nepotistic sources), thus Google should aggressively discount that self-serving bias to error on the side of safety rather than setting up platforms that make it really easy to "accidentally" plagiarize content.

Obviously Google biz dev people realize the above self-serving bias & will do nothing to correct it, but it isn't a fact that should be ignored in public debate. At least not if we want to have an honest debate.

>it's childish of you to blame Google+ for it<

It is childish to run a search company & create platforms for stealing content without checking for stolen content during the ranking process.

All the hard work on scraper sites doesn't apply so long as the content is hosted on Google.com???

As a search company your primary job is to deliver relevant search results to user. But as a secondary goal (every bit as important as wrapping everything in ads is) you need to ensure that the right people who are putting in human and financial capital get rewarded for their efforts. If you don't then the ecosystem you create becomes a ghetto.

Sites like Mahalo requiring the Panda update were funded by Google. Sure Google may have (eventually) solved that problem, but they created it too.

And in terms of "Google as scraper site" this isn't Google's first time around with this "accidental problem" either, as I distinctly remember them doing the same thing with Google Knol. http://www.seobook.com/google-knol

Worse yet, when they did it with Knol they even had a check for how related the article was to other content that they posted right on the article, but still chose to outrank the original source. http://www.seobook.com/images/knol-similar.png

And I publicly posted about the above Google+ scraper site issue last September. http://www.seobook.com/the-doors I know Google engineers read my blog & read that post, so PR spin that tries to blame another party is simply unacceptable over 3 months later.

this issue isn't something new either. I mentioned Google+'s looted copyright content problem back in September http://www.seobook.com/the-doors

Google creates the Panda algorithm against thin content & then they shag the ad impressions that quality content deserves...it's pretty hard to have a sustainable quality-based content business model when Google can just displace you with their copy of your own content!

Absolutely right on the money, as usual.

According to Compete.com, the % of downstream traffic from Google.com to YouTube has roughly tripled since the first Panda algorithm roll out 7 months ago.

Based on the referrals the site sends (often 4,000+) I would have thought it was much higher than that. My guess is that there are a lot of lurkers & only something like 2% of site visitors even create an account.

Maybe to attract employees? Or to try to appear likable as waves of austerity crash ashore & more people grow to hate the likes of Goldman Sachs & such?

The big content farming companies understand that they have to stuff the "brand" ballot box. Which is why Demand Media is doing all sorts of stuff with promotional sponsors & Aol is closing down many of their niche sites at the same time they are adding a celebrities section to the Huffington Post. Huffington has a green light to spam with authority after the Panda update.

This presumes that the stuff that is ranking higher is more useful than the stuff that disappeared. That, unfortunately, is not always the case. A couple counter examples:

http://www.seobook.com/huffington-post http://www.seobook.com/doorway-pages-ranking-google-2011

And I hope you stub your typing finger so bad they have to amputate both of your arms. :D

You sound much more clever than you are. Or was that backwards?

look in your web analytics. the dates that a lot of people saw sharp traffic declines are:

    Panda Update 1.0: Feb. 24, 2011
    Panda Update 2.0: April 11, 2011 (about 7 weeks later)
    Panda Update 2.1: May 10, 2011 (about  4 weeks later)
    Panda Update 2.2: June 16, 2011 (about 5 weeks later)

source: http://searchengineland.com/why-google-panda-is-more-a-ranki...

'Some people must have benefited from Panda'

Largely 2 groups of sites did - big brands & big media sites / sites with plenty of "brand" signal - smallish sites that were too small to get clipped when a lot of the larger (but not BIG BRAND large) sites in their vertical got hit.

'I wonder what the logical conclusion of search engines/SEO and all this jazz will be in 10 years. If history is any indicator, we'll all swing back to some AOL-esque, walled garden, ad-driven existence where "Search" means nothing more than "find applicable ads"... I suppose that is somewhat true now.'

That is a great point...and in Tim Wu's book I mentioned above, he highlighted how this was the big fundamental strategic difference between Google & Apple.

The answer to how it all turns out is still a bit unknown.

AT&T made the same argument with their network as well. And the problem was that you couldn't see all the hidden costs baked into that while it was happening.

You could only see those sorts of hidden costs after the fact, when waves of innovation appeared quickly out of nowhere. Held back for decades, a half-century of innovation appeared in little more than a decade.

How do you calculate the cost of that? In Tim Wu's book titled The Master Switch http://www.amazon.com/Master-Switch-Information-Empires-Borz... he highlighted that AT&T example & asked a person to imagine some company deciding to hold back email for 10 years & trying to guess the incalculable astronomical costs.

Now, to be fair, I wouldn't say Google is anywhere near as anti-competitive as AT&T has been historically, but they have certainly grown more adversarial over the past 3 or 4 years. Compete.com shows over 5% of Google's downstream traffic going to Youtube & that traffic flow was up 18% in a single month...after lots of other user generated content sites got whacked. Think Youtube is different & better than the others? In some ways it may be, but if you read the comments below the video they are about as low quality as they come anywhere across the web. Something like 30% of Youtube pages have "Bieber" in them.

site:youtube.com search on Google = About 621,000,000 results

site:youtube.com bieber search on Google = About 203,000,000 results

The other big issue (at least inside the United States) is the large companies are not hiring that many people domestically...so most of the job grow must come from small businesses.

A lot of the companies that were wiped off the search results were not "evil SEO spammers" of some sort, but rather small ecommerce shops & such. The drastic shifts in business do increase risks (perceived & real). How much innovation & economic prosperity that destroys isn't something that is easy to calculate.

I am doing better now than I was before Panda, however a lot of people who are not SEO experts who run small businesses just lost their business. As search keeps raising the bar on the number of steps required to be good enough to be in the game a lot of small businesses get torched, because who has time to know their craft & jump through 30 hoops for Google? Of course that is no big deal to the SEO pro who focuses on SEO...but to the small business who has to do everything from accounting to taking out the trash such a large shift in traffic & increase in complexity in search is devastating.

And, if we go back to why the update was even needed, it was primarily driven by Google funding MFA content farms/factories like Demand Media's eHow & "answer" scrapers like Ask.com.

I can show you one HUGE example of the search results looking nastier...check out some of Google's product search integration in the search results.

I just searched Google for "necklaces" and the products they showed cost $60, $8, $19, & $5. They are not widely representative of the diversity of the category, look ugly, and are generally of far lower utility than the affiliate sites & smaller specialty ecommerce sites that got torched & are now considered spam by "the algorithm".

There are also cases where some original content creators had their sites whacked, whereas the file hosting warez sites violating their copyright outrank the original source. Search can't create a much more damaging ecosystem then penalizing the creative sources & directly funding their parasites.

A parting thought here...we are generally the most unique where we are the most refined & we are generally the most alike in how we are vulgar.

As the web moves from specialty shops to big box stores (aka: big brands) a lot of that diversity & beauty becomes less accessible. Google is/was one of the few counter-weights to the big business walled garden interests online. As they place more weight on branding, they ultimately act much more like that they were trying to differentiate themselves from.

What is the long-term cost of less diversity & less economic specialization? That is how one needs to frame the above "what actual search results became worse for users as a result of these changes" question.

We are in the middle of a trend rather than at the end of it, so the answer is currently unknown.

"The author took the lazy way out, appealing to emotion and outrage, rather than actually doing the experiments to see how google differentiates between subdomains and subdirectories."

That presumes that the algorithm behaves consistently across sites & that the author of the article has dozens of sites that were hit by Panda to do a variety of scientific subdomain tests on it.

Of course, neither of which are actually true.

A lot of the biggest brands see more "search" referrals that are just their brand name going through search engines.

As web browsers replace address bars with search boxes the search engines in a very real sense act like a utility the redirects the traffic flow through a page with ads on it. If you rank fine then no big deal, but if you don't show up for your own brand then that is brutal.

Of course most the sites hit by Panda were not hit so hard that they don't show up for their site name (unless their site was named after a generic commercial keyword where there were lots of other competitors already) but in the cases where it was a lot of those businesses were just hosed. It is yet another way to boost big brands...not only put weight on brand related signals, but make the generics more likely to get clipped by the algorithm at some point.

That forces non-brands to build brand assets & re-enforce them display ads and such. Google is even buying a ton of display ads that advertise display ads. And a lot of those ads highlight how display ads have increased branded search queries for the brands that are buying them.

I think a lot of people inside the field of SEO are trying to go broader so that if & when a particular search channel is a bit unreliable or unpredictable (look at how many sites were torched in Google's Panda update) they can still keep the business afloat through other channels (paid search, social stuff, email marketing, and so on).

I don't think you can ever really get rid of the fog in search completely though. If and when you disclose something publicly that reveals an algorithm weakness, ultimately what happens is people abuse the relevancy signal and then Google evolves their algorithm to place less weight on that relevancy signal. The other thing that happens is the cost of trying to influence that relevancy signal goes up. Some people have highlighted that in some instances domain names can be part of an SEO's strategy. Eventually that led to poker.org selling for a million Dollars!

Over time relevancy signals that are shared ultimately either become more expensive to create, less trusted, or both.

The other thing I would add is that you can know a good bit about what some of the signals are & it can still be a royal pain in the rear trying to create those signals. It is easy to write "build links" but search is a winner take most market & you can lose a lot of money marketing before you hit an inflection point. The same is true with brand building. And some brand building efforts are not entirely logical, but some are focused on exploiting the irrational parts of human nature.

If irrational humans create the relevancy signals you need and irrational idealistic humans write the algorithms that grade those signals, then just rationally looking at the numbers does not guarantee you will be able to compete. You can be 100% right and still be plenty unprofitable in many business ventures.

The big catch is that some of the signals are abstract and built on non-public data. And as Google puts more weight on things aligned with "brand" you can know that they are doing so, but that still doesn't solve the problem of needing to build brand.

IMHO we are starting to move to the point of SEO where we are moving away from knowing the data to understanding deeper marketing functions that create the relevancy signals as a side effect. If you have a higher visitor value than your competition and can afford to buy more ads than any competitor can then reinvesting those profits at some point creates a winner take most market segment where you get a lot of the relevancy signals built as a free side effect of buying tons of ads.

It is worth noting that both Twitter and Facebook have worked to turn their "relevancy" signals into what amounts to ad units. The signal available there basically represents things along the lines of either viral humor pieces, large brand ads, or both.

It is further worth noting that when HP researchers looked at viral Tweets that they generally represented an echo of content from sources which were already popular and already had strong distribution channels outside of Twitter. http://h30507.www3.hp.com/t5/Data-Central/HP-research-shows-... So rather than finding new relevancy signals, it is mostly just re-highlighting/hyping/amplifying that which is already popular. Add into the pay-to-play models where the relevancy signals are basically also sold as ad units and I am not very convinced that these aid true relevancy.

Rather they might just highlight the business operations which already have plenty of relevancy signals, those doing off topic viral humor pieces, and those who have large ad budgets to blow through while trying to claim how innovative their marketing approach is.

All the above said, the ability to monetize the "relevancy" signal will cause some search engines to want to lean on it, because ultimately if they can monetize the relevancy signals that allows them to monetize the organic search results as well, while claiming the results are "organic." I don't expect Google to want to put a lot of weight on Facebook likes (doing so would subsidize a directly competing ad network) but I would expect Google to put significant weight on +1 clicks at some point.

IMHO the big picture issue here is not just the increasing complexity to develop across, but that more and more layers of media are being commoditized and owned by the search providers.

- Search engines let others build communities and filter out the noise, then scrape the reviews & replace the sources with place pages. - Expanded snippets (and review scraping) allows the search result to act as the individual product level page. Some custom vertical filters (like the Google recipes search) now allow Google to eat the category level pageview as well. Microformats encourage even more structuring of data, and as it is structured the independent sources of data become commoditized when the data is repeatedly structured in an easy way that allows search engines to displace the end data source. - Ecommerce sites submit feeds & in AdWords they can bid on a CPA basis...there are a lot of 1-word searches where Google shows me some random product ad & they are collecting all that aggregate feedback. They are not the only ones monetizing down the tail though. Search Bing for "mother's day gifts" and they have some ugly shopping promotion above their organic results. Search Yahoo! for something like "mortgage rates" and check out the ad in the organic results. - Then there are tons of other sorta attacks on the organic results in terms of verticals (books, videos, local, etc.) as well vertical ads (mortgage, credit cards, hotels prices, etc.) as well larger ad units that push the organic search results down, and finally even new search result formats which use way more whitespace to push the organic results further off the screen. - Then there are mobile applications and mobile search interfaces that further put themselves between searchers and anything "algorithmic/organic/unpaid" and the end user, replacing it with higher ad density.

SEO was a growth industry for at least a decade, but in the US now I think it is more a game of increasing competition fighting for a smaller pie.

What makes it doubly hard for smaller businesses is that with the Panda update Google directly stepped away from the pure algorithmic approach and decided to subsidize brands. Great if you are brand, but not so good if you are not. Of course, a lot of businesses do not have the scale needed to justify large investments in brand advertising. Geico can spend nearly a billion Dollars a year on ads, but whoever reads this probably can't. ;)

There are all sorts of corporate SEO issues that just "appear" out of nowhere if a person doesn't consider it in advance.

A few examples from our site upgrade to drupal 7... - comments permalinks were generating new identical pages under different URLs - paginated pages were getting indexed by category, by author, by month, and sitewide ... creating tons of duplicate content issues - the rel=canonical set up wasn't set up correctly either - some of the modules for things like related posts & page titles needed redone

And that is just 1 small blog, whereas when you view the NYT it is likely spread across multiple submdomains, multiple CMS tools, and they likely have different directly competing business objectives for different sections. They also have a variety of inbound and outbound syndication partnerships which likely add yet more layers of complexity & yet more competing business objectives.

Upgrading our site cost 10's of thousands, and that was while valuing my time at nothing. If my time was valued at market rate it likely could have been into 6 figures...and all this was doing was upgrading a few platforms to newer versions, changing the site design, and adding a new payment system. Add in the layers of complexity and bureaucracy you would see at a big slow moving corporation (and tons of money into conducting tests and research) and I am not surprised it was in the 10's of millions.

I know one of our clients had a million Dollar CMS that they ended up having to scrap because of major issues with it...am not surprised that integration of a semi-porous paywall at the NYT would be pretty expensive.

Here is another great article for understanding the Google / Wikipedia ecosystem http://www.johnon.com/399/google-las-vegas.html

The big issue here is that if you don't get other people to link to your stuff then Google has little way to distinguish quality vs non-quality, source vs copy, lower quality rewrite, etc.

While many people find push marketing tasteless and annoying (and perhaps a signal of poor product quality), the truth is that most successful sites have relied on the person launching them already having some combination of status + influence + distribution + connections, or they used push marketing for a while to build that following and awareness.

The easiest SEO answer to obscurity or lack of awareness (assuming the on-page SEO & site structure are solid) is to think territorially & dominate a niche where you can own the idea. This is a great article on that front. http://www.copyblogger.com/how-to-dominate-your-niche/

I think there will always be some amount of great content accessible because brand & popularity are required to be able to charge, and there will always be someone who is either really hungry and willing to put the extra work in, or they are just doing what they love & delighted to share it.

Pieces may go behind paywall (like our site or iTulip) but they both still share lotes & sites like KhanAcademy offer tons of great accessible content.

The big issue though is that in most markets people have to get a bit shook down before they find what they need. I remember thinking that search and ads were going to be huge when I bought Inktomi & DoubleClick stock during the last stock bubble. I of course got my head served on a platter on that, but it was cool to get into search a few years later & be right that time. The Google IPO gains made up for the losses on the earlier bets, but everything comes down to timing & then just sticking with something you believe in.

The first site you find in any category won't likely be the best one, just the one which is the most heavily marketed. But the same was true before their was a web.

The hard part with paid content is that the more common it becomes the harder it is for Google or other ad networks to take a big slice of the value chain. For that reason I see the move to paid online content being a slow one (outside of niche b2b sort of environments).

The tricky part was that many of the things were happening in parallel...so it is somewhat hard to connect it all together. I sorta tried to push it as best I could into a cohesive organization with themes like "link spam, adsense, content mills & domain authority" but search is pretty complex and I am sure we could have done a bit better...one problem with anything like this is determining how deep or nuanced to go with it. The more information you put on it the harder it is to keep it organized.

The point wasn't just that Google has to handle spam, but also that they are stuck dealing with it even as they create/fund much of it ... and that their "solutions" for round 1 leave unforeseen exploit in round 2 or 3.

To out it as an equation...

increased weight on domain authority + rel nofollow + premium adsense feeds = Content farm problem

Their latest update left a couple other big exploits open as well.

As far as faulting using an infographic as a format goes, people are more receptive to them than textual articles.

Sure some people do exploitative crap about total junk & have spammy storyboards put together by total strangers for $20, but the above was a storyboard that came from someone who has watched how search has evolved over the past ~ decade, with literally 20,000+ hours of experience in the SEO game.

I could write a 9000+ word article like http://www.seobook.com/relevancy/ but generally the market is more receptive toward infographics. As a marketer it is generally easier to go with human nature than to try to fight it. That is marketing 101 ;)

And since people do exploit infographics for links, sure it can seem cutting edge to label anything in an infographic as 'spam' (or some such), but if you could find me another online document that has described everything on that page with better clarity & in a way that is easier to consumer faster I would be quite surprised.

How else would you suggest visually laying out the patterns? Would you like it in a flash file instead? :D