Google no longer automatically indexes websites?
natehoffelder.com
natehoffelder.com
Another "bug" that seems to manifest quite often: if I search for a specific phrase or unique word on a page that I found in a SERP, so I know it's crawled that page, it often doesn't return that page either.
Add to that the automatic CAPTCHA-hellban you get if you use "site:" in anything more than a tiny amount (and the one you still get if you search "too much"), and I realise that there's increasingly huge amounts of information out there on sites that Google may have crawled before and knows about, but doesn't want to show me for some reason. I remember it used to be much easier to find information about obscure topics even if it meant wading through dozens of pages of SEO spam; now it's nearly impossible for anything but the most vapid of queries.
Perverse incentives at work!
I strongly agree with this bill hicks bit on advertising:
https://www.quantamagazine.org/sci-fi-writer-greg-egan-and-a...
I don't know if 4chan is included in the google index, but I've never gotten a 4chan result in any search I can recall.
Nothing to see here.
This is super annoying because it doesn't appear to inform you of this anywhere in the UI, until you click through to page 2 and see what it modified your query to be.
From what Google's hinted at and probably your own experiences (which I reckon are like mine), it's pretty clear that most folks aren't great at Google searches. This might be why Google has leaned on AI to "guess" the best results. They figure their AI can predict what you want better than you are able to specify via your search query.
No results found for "spill clean big search stain".
Results for spill clean big search stain (without quotes):
This isn't new, I've always seen Google doing something like this. It hasn't always been large on the page but it's always been there.
They don't need to waste the eng resources or infrastructure on rock solid search anymore, they own the market and got all the users into their funnel of products, most locked in for life.
Search results still show sponsored listings, they still have all the users, and all the profit, and a lot less of the profit sucking operational costs it took to be good at what made them a household name, search
I wish there was a search engine that ran like mid 2000s google but with a social media component so you can down vote SEO spammer blogs into oblivion.
> I wish there was a search engine that ran like mid 2000s google but with a social media component so you can down vote SEO spammer blogs into oblivion.
There's no way this won't get abused, but the SEO stuff is out of control. Not even spammer blogs, but if you have a quick question like "how do I check tire pressure" you will only get articles that start with a treatise on the entire history of car tires and the answer is deeply buried somewhere in the article. My guess is that Google sees that we're on the page for a longer time than we would spend on pages that just return the answer, and they assume that "more time on page" == "better content" or something.
This is sort of what I've been trying to do with Marginalia Search, except I don't really believe a voting system would work. It's far too easy to manipulate. Been playing with the thought of having something like an adblock-list style system where domain shitlists can be collaborated on and shared without being authoritative for the entire search engine.
My search engine is still pretty rough around the edges and limited but I think it works well enough to demonstrate the idea has some merit.
I want this too, but I think an often understated aspect of this issue is that by this point Google has absolutely trashed the web of that era. In these threads people will say "the content you want isn't out there, it's all on social media now" -- and they're largely right, but I think Google is the party most responsible for mutilating the web to the state it is in now, and users fled to social media partly because it seemed like a safe haven.
What we need is a concentrated effort to rebuild the web. Take the best parts of what we've learned and combine with the best parts of what we've left behind and try to build something better, for humans, not for advertisers and hyper-capitalists.
That will take time, energy, and people who remember what we lost and believe we can build something better. A better search engine alone is not enough.
We know it's not, but expect them to try to tell you you're imagining things.
I've noticed this as well. I have a crappy website for my app I need to do better marketing for (not my priority just now), but I've noticed that, for however crap it is, I have received ZERO incoming hits from Google, apart from a couple people that have literally just googled my domain name.
I do not believe for a second there's not a single query done in the 2 months the page has been up, globally, for which my website wasn't a bit relevant. Either that, or the spam problem Google has is much bigger than anyone thinks.
Yet another data point in favour of the Dead Internet theory.
edit: Hah. I notice it suggests using it at the top of the page if you use 'site:..." - and I only get 5 results for my site when the console claims to have indexed 10 times that many!
edit2: Also duckduckgo returns more like 15 hits ...
"The dead Internet theory is an online conspiracy theory that asserts that the Internet now consists almost entirely of bot activity and automatically generated content, marginalizing human activity. The date given for this "death" is generally around 2016 or 2017."
Not sure I'd call it a conspiracy theory.
Thoughty2: The internet has died
In this video they rename it from 'theory' to 'prophecy'. As in the internet isn't quite dead yet, but its rot filled bloated body is near its dying breath.
Pretty much any advanced operators seem to do it for me, notably "intitle:" and "inurl:". I'd wager that there are a lot of automated searches using these to look for exposed admin interfaces, but I find them extremely useful for filtering out the crap that clogs up results when a ton of news sites all regurgitate the same viral press release or wire article.
So you may see a certain number of pages using the site: command but not or less pages may be indexed.
If you want pages indexed, out then in an xml sitemap file, make sure there are internal links to them on your site, and external links from other sites really helps. Third party indexer tools help as well.
It's concerning that they're even nerfing site search, which seems like a core feature for a search engine. You could argue that Google isn't really a search engine any more, but rather a general knowledge engine and advertising platform. I hope somebody can build an alternative to Google that does what a search engine is supposed to do, i.e. index the web without all the extra garbage. But maybe SEO has killed that dream at this point.
this is some machine learning stuff they are doing, instead of indexing all the specific keywords they are creating vector embeddings and basically summarizing what's on the page and going on similarity to your query rather than specific keywords. Good for casual searches, but extremely annoying for power users
Source? This would be worrisome.
Appalling
I am not sure why I get them but it might be due to using anti-fingerprinting tools.
I've noticed that myself, looking for very precise content, which I know is out there but failed to bookmark. (Most recently, for amateur astronomy and roleplaying.) The solution to finding niche stuff now seems to be digging through relevant reddit or forum threads, hoping someone posted a link to it.
Is it more expensive, or do they just wish to prevent people from being able to cache their own results locally?
E.g. I can build a searchbox on mywebsite.com, and if you type "hamster" I'll just query google for "site:mywebsite.com hamster" and return the results to you. That way, my site can be static but still have a search box, and google has all the work but gets no money.
If you want "misinformation," or to just search the web like it's the mid 2000s, you can use http://Yandex.com. They do a pretty good job on controversial queries. Google has gotten so political that they even have this "results are changing rapidly" page they return when there's been some new political hot topic that they haven't gotten the commissars at headquarters to weigh in on yet as to what's going to be the official narrative.[1]
[1]https://www.theverge.com/2021/6/25/22550430/google-search-re...
Yandex image search also has an infinitely better interface, linking your directly to image sources and not being full of links to sites that want you to sign up before showing anything like Google is. It's still not perfect and IME often groups images to agressively which effectively hides "similar" results.
The site is probably 20 years old and has no SSL, but still... giving me no results is worse then giving me the one correct result.
I wonder what that reason could be.
Why provide the best product when you only have to have a product slightly better than your competition. After that everything is profit.
Couple that with a huge portion of new sites seem to be bot generated shit that's copied from other places on the internet it seems Google has given up on the open web.
How would that make them money? Here, instead have a few links to irrelevant videos that bring in ad revenue!
This is your misunderstanding. The fact that a thing was in the index does not ensure it will always be there. Things disappear from the web all the time. Serving fresh docs means not only crawling the new stuff but also deleting the unreachable stuff promptly.
Please explain this point.
> Please explain this point.
If Google thinks your searches are unusual, it will force you to answer captchas to see the results. They assume anyone using advanced features must be trying to abuse their service.
In terms of breadth and depth, the quality of google search has declined noticeably. They don't have any real competitors in search so they can do whatever they want.
> now it's nearly impossible for anything but the most vapid of queries.
Rather than getting us want we want, they want to gives us what they want. A narrow band of approved results. Youtube is like this as well, but then again, youtube and google are both part of Alphabet. It's like google news was a test run and they slowly exported it to search, youtube, etc.
"SERP stands for Search Engine Results Page. It refers to the page displayed by a search engine in response to a user's query. When a user enters a search term or keyword, the search engine generates a list of relevant web pages and presents them in the form of a SERP. The SERP typically includes a combination of organic search results, which are the regular listings based on relevance to the query, and paid advertisements, which are sponsored listings that advertisers pay for to appear prominently on the page. SERPs often contain additional elements such as featured snippets, knowledge graphs, image or video results, local map results, and other specialized features, depending on the specific search engine and query."
I know that many of you would say that it doesn't really matter what "hackers" think about Google Search, that all that matters is that the majority of the non-tech-savvy users still think Internet == Google. Well, let's talk again in 5 years.
Today I avoid Google search at all costs, and use it mainly to find Wikipedia pages or to search Reddit.
The Googlers blame SEO and there is some truth to that; but Google has on retainer a huge stable of the world’s best-paid engineers, and still couldn’t be bothered to invest in their flagship service.
Google used to be amazing. If you remembered a set of words that were on a page, you could enter those words and it would find matching pages. It’s been broken for years and keeps getting more broken.
https://www.google.com/search?q=%22may+have+been+less+attrac...
I just took a random snippet from Gergely's blog and using quotes I found it.
Look there's many issues with Google Search in 2023, but these nitpicks that aren't neccesarily true aren't productive to the conversation.
They were obviously wrong.
Then Chrome happened. If Google doesn't change some fundamental things, they'll be the next one to fall for similar reasons.
I'd be less concerned for them if I didn't see signs of decline with Android as well. It is a key component for them to control.
https://gs.statcounter.com/browser-market-share/all/worldwid...
There were "tiny upticks" but, going slowly and steadily, in August 2008 Firefox had gained about 30-33% and IE was down to about 60%. And then (September 2008) Chrome happened.
> “How did you go bankrupt?” Bill asked.
> “Two ways,” Mike said. “Gradually and then suddenly.”
It felt like that’s how IE lost its dominance.
The most recent game being played is language, a game that Google has played alone, until now.
Doom & gloom talk about Google 5 years ago was probably unjustified and rightfully dismissed. Today, I'm not so sure about that.
I needed some somewhat obscure API information recently, and ChatGPT had it -- a testament to how much ChatGPT really is just a compression of "everything ever digitised"
We’ve had it come up with solutions using entirely made up functions. They had names that sounded like something Microsoft would’ve put into .Net, but they were entirely made up. As in, they had never existed in any version of .Net ever.
So as much as I like it, I’m treating it with more caution than google results. Honestly though, most of the time it’s frankly faster to just read the damn manual and figure out things for yourself. I don’t say that as some sort of anti-prompt-programming purist, but wading through GPT responses is as about as hopeless as wading through the gazillion medium, dev.to, stack overflow and whatever else people post their useless stuff on. 10 years ago if I needed to do some obscure Sharepoint programming (waaaaaay out of my field) I could realistically make something work with the help of Google, today the same thing is frankly completely impossible.
Admittedly that was on GPT3, I haven't tried GPT4 as I can't afford it at the moment. No doubt its better, but I'm not sure how much so.
Those correct-sounding answers are often actually correct, but this is more coincidence than design. Anything it says is suspect, and should be fact-checked.
Insofar as it isn't just a regurgitation of "the better ebooks, blogs and docs", the worse it is.
This is why, when prompting ChatGPT, i'm more often aiming to have it use examples (etc.) that have a high likelyhood of exact data in its training set.
Consider eg., the prompt, "write tic-tac-toe in javascript using html and canvas" to "write duck hunt in javascript using html and canvas"
The latter is extremely hard to get out of it, with many prompts -- the former, immediate and perfect.
Why? because there's many examples of tic-tac-toe.
I don't get this problem. Google search doesn't give you answers, it's not reliable, and you still have to double check the answers your.
Remember that? Any information found on the internet used to be assumed to phony too.
ChatGPT is just a frontend UI and refers to two different models (and different tunings of these models over time I guess).
GPT3.5 is kind of useful but it makes up stuff just enough that you can't really trust it and spend so much time verifying it's hard to say whether you saved time vs the old way. But it still produces mostly not made up stuff.
GPT-4, still not perfect, is a game changer though. It's what people generally mean when they talk about ChatGPT. There's far far less hallucination going on (not zero though). There's several ways to access it: phind, bing, ChatGPT. But I still give ChatGPT is the best.
I never really liked DDG or Bing. They both suffer from the same SEO gaming that Google does.
There are also a number of smaller search engines that have been posted to HN that are kind of interesting for certain niches.
I think what will happen over the next five years is that instead of Google being the one-stop shop for search, it will be a number of smaller players + the different chatbot engines like ChatGPT and Bard and others.
Every time new fad appears, I say this. And I say the same about chatgpt. Never failed so far.
So yea, let's talk again in 5 years.
If the way to make it work is to use "site:", to me that doesn't count as "searching the web" anymore.
And these have started to become > 50% of my search results.
For the query "discord overlay not working wow", I get the following as THE SECOND RESULT: https://freeholidaywifi.com/discord-overlay-not-working-wow/
"This structured data helps Google differentiate paywalled content from the practice of cloaking, which violates spam policies."
Which is just odd to me. Why present a paywalled search result, when the market is so fragmented that the odds the user has a subscription are so small?
[1] https://developers.google.com/search/docs/appearance/structu...
Which is just nonsense. A paywalled site showing different content to GoogleBot than to ordinary users is cloaking.
Quora and Pinterest were the first to go for me.
https://web.archive.org/web/20230531062024/https://natehoffe...
The page isn't being cached.
Cloudflare doesn't cache pages by default as that would create issues (logins, comments, admin sections, etc). They cache static files (images, css, js, etc), but that's it. You need to set it up yourself and have ways to bypass and purge the cache.
Or they could just rely on Cache-Crontrol headers (and friends) instead of requiring CF-specific configuration.
When I moved bitecode.substack.com to https://bitecode.dev, I submitted the sitemap to google and was surprised that it took almost a week to even show a single page as indexed.
This used to be a couple of hours to a coupled of days.
So something definitely changed, but as usual we can only speculate until a googler come in this thread.
Their own fault for monetizing every imaginable search query.
It has provided the incentive to ever spammer and scammer on the planet to mass produce copious amount fake interlinking pages massively larger than actual Internet.
Pretty soon the only way to live will be an Encyclopaedia, text books, and cookbooks from before 2010.
A massive ton of bot generated scam sites that scrape content like SO and splatter it on 50 other domain names eating up all the bot time?
If you want to be indexed, submit your pages to Google explicitly through a GA tag, a Google Search Console account, and a sitemap.
If you're publishing many, many pages, you might see a benefit with the indexing API: https://developers.google.com/search/apis/indexing-api/v3/qu...
> I have been a web designer since 2016, and before that I was a blogger for 6 plus years. I have been deeply interested in Google SEO since 2010, and in all that time I’ve never heard of Google not indexing a site.
Google crawls sites it assumes will have useful content. If it doesn't have that assumption for your site, you need to feed it.
The convention site in question had a GA tag, and it was in search console, and I had submitted a sitemap. Search Console knee the site had 9 pages in the sitemap, and had only indexed the homepage.
The other pages weren't rejected, declined or 404ed - they were simply ignored.
What if I don't want to put any Google-related stuff on my site?
I have a GitHub pages site with a very unique topic and it's still not indexed by Google (the site is available at least for 8 month now). I've checked Bing and my site is on the 4-th place on the results page.
Then you're choosing to make it harder to be platformed by Google.
Regardless, a GSC account and submitting a sitemap do not require you to serve any Google-related "stuff".
Currently, the Indexing API can only be used to crawl pages with either JobPosting or BroadcastEvent embedded in a VideoObject.Good, then let's get rid of this ad network dystopia which never worked as advertised (they show me special offers for a frying pan, 2 weeks after I ordered one. Laughable ya folks). Replace it with great authentic inspirational content and people will throw their 2 bucks at it.
Additionally, Mullvad showed how to use cash via regular post to pay 100% anonymously for service X.
https://webcache.googleusercontent.com/search?q=cache:https%...
Unfortunately "[being] a web designer since 2016" is generally not helpful for SEO
But certainly setting up Google Search Console, submitting a sitemap, and 'fetching' your homepage on day 1 is good practice, especially for a brand new website
I did that with the convention site, yes. It still wasn't getting indexed.
On a side note on the SERP (search engine results page) of the foreseeable future, I don't accept the doom and gloom other SEOs have. Google has shown it will give credit to the top 3 links, and to be honest pretty visible credit. This will shift the target from "front page" to "top 3" sort of like Local Seo Map Packs.
Will make difficult keywords more difficult to get clicks for, but long tail keywords could be easier to rank now as a #3 ranking could garner as many clicks as a #1 ranking.
Perhaps the idea of creating a master index of the content of multiple pages on nearly every site is the thing that is not sustainable. Instead, "search this site" needs to be made great again. Individual websites could manage their own search in a way that complies with a standard API that can be consumed by meta-search engines. Rather than indexing pages in the traditional way, meta-search engines instead use a heuristic or AI model to decide what sites are going to have the kind of information you are searching for, perform your query on theirs, and return the aggregated results to the user. The less the algorithm understands the significance or meaning of the query, the more generalized its approach can be. For instance, if it thinks that you're searching specifically for opinion-based content that will appear on blogs and forums, then it will target a federated search engine that indexes those things specifically. But if you are searching for information on making beer at home, it will know to target and weight the search engine on brewersfriend.com and homebrew.stackexchange.com. Although this sounds not that different to how search currently works today, remember that this idea is about having search become more federated and more standardized, and for meta-search to select federated indexes rather than own a god-index. A user of a meta-search can pick and choose what indexes they want available in their searches in case they find any of them to be either superior or particularly problematic, and the meta-search can optionally adjust its understanding of a particular user's queries.
The way I see it, traditional search will continue to decline in part because it's not sustainable, but also in response to AI allowing them to become "answer engines". Although a lot of people do want an answer engine, this isn't for everyone. I think there will always be a market for people looking for content on specific webpages. Whatever that thing is that someday snatches that market away from The Google, if it's going to be successful, won't survive on the current concept of what "search" is.
Over the years, as all of the ways that people who hosted a "search engine", exploited the "digital exhaust" of the people who used it, and sold that information for profit, have been revealed, well regulations and laws have been enacted to reduce or eliminate the egregious (and usually more profitable) ones. That and advertisers tired of paying for "engagement" and seeing few sales or conversions, the economics of running a search engine as this high margin, make money hand over fist, days are slowly winding down.
That Bing has (for now at least), continued to index these sites, will just accelerate Google's decline.
When I was helping to run Blekko, people would look at our curated results and be super impressed at how much better they were than Bing or Google results, but then they would search for their cousin's minecraft blog and wouldn't find it and lament that they couldn't possibly use it as their "daily driver" because if it didn't have their cousin's website in it, how could they know what they weren't seeing? Blekko tried really hard to make the argument that if you made a list of the minecraft blogs you followed as a Blekko user we'd index all of them, and if everyone did that for their favorite stuff then the index would fill up with good web sites and not be riddled with junk. But sadly we couldn't get them to internalize that and they weren't willing to create an account to have a better web experience. Perhaps we were just too early but still it is a weird thing.
Although I don't think any search engine has ever indexed or will ever index everything. That doesn't really make sense. Realistically I think maybe 1% of the documents online are ever going to be a good search result for any query ever.
Internet search is all about being judicious about what you index.
I don't disagree with this, I was simply pointing out that Google makes no money by sending people to a web site unless the owner of that web site has bought an advertisement and the person involved has clicked on that ad. Organic search, like "good explanation for approximation theory" should have something like this: https://xn--2-umb.com/22/approximation/ as one of its top results but it doesn't even make it to the first page.
As for actual indexing, we found the Indexing API to be the most reliable. https://console.cloud.google.com/apis/library/indexing.googl...
The caveat is that you can only request Google up to 200 URLs to index. It's also technically not the most 'correct' way to have your pages indexed. (supposedly used for job page updates, etc)
You could be up 20% one day and down 40% the next and you are none the wiser as to why.
If you are a small time publisher with no budget or no craftiness to attract links from other sites - you are pretty much doomed.
I have a few sites that are in the 200k monthly visits range (from Google) and Google only fetches the homepage/feed every four hours or so, sometimes it takes longer than that. It’s a lot different from what it used to be.
https://www.google.com/ping?sitemap="url"
Site:"my url"
yielded all pages. I'm terribly rusty with websites, hence my use of WordPress and willingness to taint my root directory with Google files. I do notice that exact, relevant queries in quotes still show no results for some content. Much to relearn.
https://www.google.com/ping?sitemap=
only works if there is a previous root reference (i.e. via robots.txt or via GSC submit) of the sitemap.xml
right issue, as from the original spec a sitemap can only submit URLs of the same or lower URL hierachy to Google.
or there is a root reference to the sitemap.xml
I've done everything right according to Google, set up the search console, uploaded a site map, addressed all mobile usability issues. And yet only a tiny fraction of my content is being indexed.
I'm a bit at the end of my rope here as I've poured a year into this project and getting a historically normal amount of search traffic may be the difference between this project being viable or not. The most frustrating part here is having zero visibility into what's going on.
I’d add a paragraph, hero image image, cta/etc at the very top explaining what your site is. Additionally you need a menu at the top and footer at the bottom with links to additional content-only (not products) pages - IE an about us, where do we source data, etc. Even 2 blog posts would help a ton. Do not stuff them with keywords but be sure to use the words that are common in your niche of the industry so your site gets associated with industry sites.
From googles eyes the difference between a scammy online ecom site and your site is hard to see! (Even if your site provides legitimate value to users).
You can try posting a link to it on various digital Marketing subreddits (not the like “rate my website” ones) to see if you can get more feedback - I haven’t done that in years though.
Edit - also didn’t realize clicking product links takes you to external sites. That’s a tough site for google to ever understand correctly since you have so little content and the best possible outcome of a user visiting your site is that they leave it. Maybe set it up to have each product link to your own page for it, maybe with price history from camelcamelcamel, links to the product in tiger sites, generic info about what site it’s listed on, or just crawl the description at the vendors site.
Backlinks also matter, for both domain and page authority. You are competing with 20+year old domains from large companies-- why should you (or any new site) get ranked before dickssportinggoods.com who have top tier backlinks (graph network, implies trustworthiness) from sites like Espn.com? Google likely uses CrUX data for ranking (because 2023 backlinking is vastly different than 2012), so high engagement from users is likely a KPI to focus on, in addition to backlinks (both branded and inclusive of terms/pages you want to rank for).
Have its ranking algorithms do the opposite of all the things you just said!
You might argue that your discount price is "unique content" but good luck getting Google to understand that. Plus, I imagine those shoe names are competitive queries, which means you're up against paid ads. Further, I assume your discounts are time-sensitive, which means unreliable indexing is not a good solution for up-to-date pricing.
I assumed that meant it had been deleted from the web; the publication that had disappeared 10 years ago. Then saw I'd linked to it from my own site, and the link worked just fine :facepalm:
It's weird to see Google forget the kind of content I used to go to them for in the first place.
I wonder if it may be related to to the wayback machene "https://archive.org/web/" ?
My site is on it has has been there for a while. I also have these on my WEB pages:
<head> <META NAME="DESCRIPTION" CONTENT="MYTITLE"> <META NAME="KEYWORDS" CONTENT="COMMA SEPARATED LIST OF SIBJECTS"> <META NAME="AUTHOR" CONTENT="MYNAME"> <META NAME="RATING" CONTENT="General"> <META NAME="ROBOTS" CONTENT="index, follow"> <META NAME="REVISIT-AFTER" CONTENT="90 days"> </head>
Maybe that will help out ? But I do notice google searches are not as good as years ago, and duckduckgo is close or maybe better than what google was.
If I manually request that they index a page, it always succeeds and shows up on Google within a few days, but the page gets pruned from the index within a month or two.
The weird thing is that within the past couple years I've also seen unwanted indexing, e.g. low-spec staging servers getting wrecked by Google crawler traffic. I don't think the "no automatic indexing" treatment is standard for every site, so there must be something that triggers it, but I've spent a long time unsuccessfully trying to pin down the cause.
now it takes weeks to month for the noindex to vanish. even after google has crawled the pages again and again as visible in the logfiles, the pages stay noindex even though the information is long gone.
google, the lazy monopoly strikes again.
They lost the battle, at least for now.
This also lines up with Google Search results becoming increasingly worse over the last few years, but the last months in particular.
If they aren't crawling or indexing your site, then link to it from a high reputation place.
So if someone registers a new domain, puts a site on it, doesn't link to it from anywhere else, doesn't submit to to Google... and yet expects the site to be found by Google, that is just not a reasonable expectation.
For those who missed it: https://gizmodo.com/backup-data-on-youtube-hack-white-noise-...
Set up the sitemap, then link the sitemap in from Google's Search Console Tools, and install Google Analytics. This will help Google pick up that your site exists.
Make sure your robots.txt file is configured to allow crawlers. Make sure your pages aren't inadvertently NOINDEX'd.
SEO isn't as relevant as it used to be, but all this stuff should be part of your QA and pre-launch checklist.
Get 90+ Lighthouse scores.
https://developer.chrome.com/docs/lighthouse/overview/
Run your pages through WAVE or some other accessibility checker.
Run your home page through Security Headers or some other front-end security audit.
Set up uptime monitors on every page that gets more than 1% of your traffic... check page load speeds and HTTP response codes -- you never know when a WAF or some other system will get mucked up.
So... the Google indexer doesn't work? SHOCKED! I AM SHOCKED TO FIND GOOGLE DOESN'T WORK!
Speculation: Crawling the web is expensive. So it makes sense that they decide for each crawled page if it's profitable to put that into the index. In the long run they might not crawl the web at all. People will just pay them to get into the index.
Sidenote: I wish Google would lower the ranking on websites who push for a mobile app and degrade your web browsing experience. There's no point in even getting them as a search result. If I wanted something from an app, I would have just gone on the app store.
I recently put up a new site, and...nothing. As the author said, I finally had to force each individual page to index.
Harder to quantify, but it seems that search results are also worse than they used to be.
It's possible that Google is doing some internal cost cutting...
Recently, I discovered that the top results are now some university pages that are not maintained, or even have never been set up. Which really undermines the purpose of the site. (Notably, the site has quite a high number of reported Google search hits, has rather perfect Lighthouse scores, and SEO ratings are high enough that I get SEO related requests – which are happily ignored – on a rather regular basis. It scores about rank 60K in the "Majestic Million". And the site enjoys a few updates/fresh content a few times a year. So this is not a case of obscurity.)
It took a few weeks until google noticed that it's a page with relevant content, but that's kinda expected. But once it did I feel Google is indexing my pages extremely fast, so fast that I have been wondering how they're actually doing this. I post new content on various social media sites, and my best guess is that google gets some of them as direct feeds that they check for interesting links. Google does not support indexnow, and as far as I know also no similar feature (except manually via the search console), so I'm not in any way directly submitting my content to Google.
It is a link database, at first glance resembles Reddit clone, but my focus is on creating link database, not on providing social media experience cancer.
Links:
[1] https://github.com/rumca-js/Django-link-archive
I made a self hosted spider that only index sites that I've visited and search engine with "and" "or" logics.
Thinking to make it a federated search engine by allow each user whitelist domains that are not sensitive to search with the peer. And users can follow/block other users explicitly to avoid spam.
Google can only index what they know of directly, or can indirectly discover e.g., by links. I have unindexed sites because, well, they're not linked to on the public Internet.
I can pretty easily see a 2 month old website being outside that; the question is does the rest of the Internet know about it? (And, the public Internet. A shared link in a Discord channel also isn't visible to search engines…) Some of the examples contain statements that indicate Google could or should know of them, but not all of them.
Users of Google search want the best results, not all results. Customers of Google want their ad or site visible, not all sites.
So to me it seems they are missing good content, not the bad.
Its mostly due to, its a money saving thing for consumers, and that isnt really profitable. Its low hanging fruit, and I have the best website for it. Nothing really comes close, most alternative websites make mistakes in their advice because they are using feelings rather than millions of entries of data.
Anyway, if you search specifically for my most popular metric, you will always get my website. If you google 'cheap X', you will get inferior websites.
Even with SEO optimized, there are just bigger websites that are friendlier with google, linked by other websites, or it could be better SEO. Whatever the case, it makes me wonder what kind of websites I miss because I use google.
The ironic thing is Google is violating their earlier principles to provide "best results." IIRC, one of their big early differentiators (which they made a big deal about), was making the default query operator AND and not OR. A lot of early search engines used OR to pump up their "total hits" numbers, now Google essentially does the same thing by dropping terms from your query if the number of hits are "too low."
In other words, this was already an issue before ChatGPT and the algo seems to be severely broken. High time for a thorough system check.
Adsense reviews are just as messy. Another department in need of a big shakeup.
I don't know what's going on there, but there's definitely lots of chaos behind the scenes.
Who is to blame? Google. For years they have pushed people to have slabs of text. Now I need to read through somebodies life story and how they stubbed their toe against a rock while hiking in the Sonoran foothills followed by a recipe for pulled pork tacos.
I am not affiliated with them.
No thanks.
I have seen the future. Crap is free. Endless amounts of crap. Soon all this crap will be AI generated and designed to addict you. If you want anything that is not crap you will have to pay for it.
I'm a huge fan of self-hosted and decentralized stuff but a search engine is an area where I can't think of a way to do such a thing. The bandwidth requirements for continuous spidering and the data storage requirements are too high, and if you tried to distribute it you'd end up with an absurdly chatty protocol that couldn't be used by anyone with anything short of an unmetered full duplex fiber link.
The next best thing is a company that I pay to be a search engine for me and does that without trying to shove ads at me.
I get that part of the issue is that I keep a weird privacy focused configuration in my browsers and delete cookies when I close the browser, still it results in poor user experience, for a service that I pay for. I don't have any good suggestion on how to fix it, but it's is valid to complain about having to sign in.
There used to be some sample searches on their blog but it looks like those are also paywalled. wonder if that was intentional.
- Log into webmaster tools and find out what is going on? Maybe there are issues with the page. (I see Google doesn't call it webmaster tools any more.)
- You can explicitly submit a page to be indexed.
- To use the tools, you have to prove to Google that you own that domain: follow their instructions.
The long term answer will be some kind of “Google Blue” where you have to pay for any placement, not just premium placement.
That raises major antitrust issues.
So many people were clicking on the link from here that it inadvertently DDOS-ed my server. I had to turn on the "under attack" feature in Cloudflare just to get the other sites running again.
Ugh, talk about frustration!.. :-/
it has already been hard to discover websites hosted on their own domains, unless you find them organically (such as in HN).
search.brave.com
It's working on this idea called Brave Goggles where you can click on a goggle and it tweaks the algorithm to prioritize that type of content or specific sites get higher ranking
It's pretty cool.
All their results have been fantastic as well for me and it's supposed to be private as well.
It's pretty much replaced google for me without even noticing.
And I left. Irony all exhausted