The web starts on page four
christianheilmann.com
christianheilmann.com
Right now the big problem as I see it is that there is great content being created but its audience just isn't finding it. There's so much noise, the signal is almost completely drowned out.
Out of largely the same frustration permeating the link, I built search.marginalia.nu alone in less than a single year in my spare time without prior knowledge of search, paying for the hardware out of my own pocket.
I don't say that to promote how great I am. The point is specifically that I'm just some guy on the Internet. Just imagine what could be possible if actual resources were put toward finding solutions to this problem.
TF-IDF, an approximation of BM-25. Only twist is that I do Personalized PageRank biased toward a secret segment of the internet to promote the right kinds of search results.
Yeah, they're called people.
We can curate better results.
I really enjoyed the Halt And Catch Fire take on curated vs algo search.
Over the next few years, things will change. When people don't like the product, it gives us an opportunity to make products people do like. Right now, a lot of the internet is just not very likable.
It's not that good things aren't being made, it's just that it's incredibly hard to find them among all crap. There doesn't seem to be a good mechanism that lets the actually good products float to the top.
If you are spending money, possibly just having the worlds info at your fingertips doesn't make up for a lack of experience.
We can still "vote" with our wallets but just like in real elections sometimes it's really hard to even get some worthwhile candidates on the ballot.
It's a curation problem. We have billions of pieces of content being created and only a handful of curators to bring that info to us. All the big search engine players want to be "the one" but it's a fools errand IMO.
No one search engine will be able to become "the one" because there is simply too much information. Google with all of their cash in the bank and talented engineers can't even crack the problem. The problem though is even though each search engine offers something different, they all think they offer the same thing and compete to muscle out any smaller players.
So you get a situation where you want a metal roof for your house, metal roofer gives you a quote. Big roofer guy in town hears about this and runs the metal roofer guy out of town, then tells you he cant offer you a metal roof anyways.
Has the web ever been much beyond that? Have people ever really been just hitting "give me a random web page about X"?
Of course search engines are very useful, just that these conversations about content and "the web", I feel like there's not much focus on how so much traffic is guided through link aggregation.
I use search for information, but I often have to dig quite deep to find something worth clicking on or reading. And that's true for Google and DDG (which I mostly use).
When it's raw, basic knowledge, then I'm using !wiki or !mdn etc. in DDG to search those sites directly. Note: Wikipedia and Mozilla Developer Network are two of my favorite sites.
But when it's something more subtle and specialized that I'm not very familiar with then often my only hope of finding something useful is to search via aggregators and forums like HN, Reddit etc. because then I can gauge the quality of the links via the discussion around it and the people who post them. Or in some cases I know about a specialist or journalist and can use their name to filter out spam.
Discovering these kind of things via search is really hard, especially if their authors are not known. It's the spammers and low effort creators that get rewarded the most, partly because of search engines, partly because of visitors who are content with that kind of quality.
I wish I knew some decent curators or magazines, but most of the stuff I come across is just too far from what I‘m looking for. Search engines don’t help much here either.
Some of these raw looking indie games have turned out to be gems. Thinking of Faster than Light, Loop Hero etc. It's hard to gauge the game-play from a quick look. Not even "similar to" searches help too much. It's not really about the money spent, but rather about the time both finding and then playing. I wish I had a list of curators that I really dig. So far, hit and miss.
Link aggregators are also very much part of the problem. Besides the astroturfing, they tend to favor the lowest common denominator in content. Popular vote just doesn't seem to be a good way of surfacing quality content, especially not with the social incentive of winning internet acclaim for submitting a good result.
HN does surface the odd gem, but why can't I pick one of those and say "Hey, this topic interests me, show me more like this"? I'm currently working on build something to that effect, and so far it seems at least reasonably possible.
RSS was good in one way, but it also sort of turns everything it comes into contact with into a blog.
Why not start it? One of the beauties of reddit subgroups is that not only do you find random very specific ones, but if you don't, you can start your own.
Oh, wait
"Can't they" or "won't they" because it's simply not in their business interest?
Small problem: There are plenty of folks, foundations, companies who are not resource limited in attacking something like this, yet it still doesn't happen. So maybe it's not as simple as you think?
Just look at the recent (as of yesterday) stories of the changes that Googs is pushing to re-enable tracking that has been previously manually/intentionally disabled by the users. Look at how Maps is forcing users to enable wifi in order to provide routing directions. This one happened to me personally as well.
The wifi thing is just a silly example. The wifi is required to do fast start GPS. 99.99999999% of their users probably don't want slow start GPS. If you do, sure, it's no longer the app for you. Yes, they are optimizing for the customers they want and giving them a good experience . I'm not sure exactly how this is something abnormal. Like, you are basically complaining they won't add knobs for you. That's pretty standard in good products - but also means if the knobs they aren't what you want, it's not a good fit for you.
I got the email in the latter, and that is a super weird take. They had one control for literally everything, which was owned by admins. They almost certainly got complaints that turning it off it made search suggestions not work in sheets/docs/etc. So they introduced a new separate control for that, and made it controllable by the user. Because it's controllable by the user, it also means the admin can't see the doc search history of the user, only the user can. Nor can they delete your history for you - only you can control it. Your auto-delete preferences, etc apply to it.
None of this seems bad or horrible to me. The only thing even worth picking a bone about is whether the new setting should take it's default from the old setting or not. There are issues in there of course, because as mentioned, before admins could turn it on/off for everyone - and this led to lots of complaints. do you take the admin setting? Do you take the setting the user had if it hadn't been overridden by the admin? etc
The claim, in that story, i that it is forced on by default even if you had it off. This claim is wrong, afaict. It was not worth wading into that clear firestorm of a thread.
The support doc clearly says: "If you disabled Web & App Activity in the Admin console, it is turned off for Google Workspace users. Users can turn it on if they want."
IE if the admin had it disabled, it will be turned off. It also says if you had it disabled yourself, it will default to off.
It goes through this quite clearly if you click "what if i had it disabled" and "what happens if i don't do anything".
Overall presenting this as "re-enable tracking that was manually/intentionally disabled by users" is a real stretch.
That's right, because I wasn't attempting to address that. Instead, I was addressing that the cynicism towards Googs is not unjustified.
It'd be much more interesting if you would engage with the actual conversation being had.
No, it isn't. A cell phone connection is sufficient for that.
I'm happy to change it to "wifi helps with a bunch of features they have". AGPS is not even the only one. Wifi scanning is happening in the phone anyway to know when to switch networks, etc, so they are saving power when it is used instead of cell phone.
GPS is just fine for Garmin, TomToms, etc. Why suddenly for a phone that has WiFi in them Maps suddenly "requires" wifi is just not convincing me of anything other than Googs wants all the data.
Yes, it does. See below.
> I'm happy to change it to "wifi helps with a bunch of features they have".
This does change what you said, because "a bunch of features they have" is not at all the same as "GPS, a feature which pretty much everybody wants to use and wants to start fast".
> Wifi scanning is happening in the phone anyway to know when to switch networks, etc
Not if wifi is turned off. If I'm in my car, for example, I might well turn wifi off to prevent the phone from wasting power scanning for wifi networks that aren't there. But I'm still going to have my cell phone connection on.
> they are saving power when it is used instead of cell phone.
Nonsense. See above. Plus, the cell phone antenna is always scanning too, but on a phone it is far less likely to be turned off to prevent that than wifi is.
What I really can't understand is how any of this is connected to your (correct) claim that search is hard and content curation is hard and so we shouldn't expect Google, or anyone else, to just magically solve it. Google could easily serve the needs of users much better than it does, even if it can't do it perfectly. But it won't because it's not in the business of serving user needs, it's in the business of serving advertiser needs, and serving users is just a by-product.
If Google really wants to solve the problems of users, instead of advertisers, it should have users pay them instead of advertisers. But of course it won't since it has spent more than two decades now getting users used to the idea that search and other "hard" services should magically appear for free, and hoping those users won't notice that they are actually the product, not the customer.
This is such a uncharitable assumption i don't even know where to begin. Nobody can serve everyone well. Trying is a quick ticket to failure.
The choice not to serve certain people is clearly the correct one, even if one could theoretically do more than you do now. Why isn't the simpler, more charitable answer that they are just serving the majority of their large userbase, or whoever in that user base they care about? Why is it everyone who feels like they aren't served has to decide the reason is because someone is doing the wrong thing, rather than "guess i'm not the user they are looking for".
As for whether you can do more - Good strategy (and product choices) are framed as mutually exclusive choices: https://hbr.org/2012/09/bringing-science-to-the-art-of-strat...
That is always going to make someone upset, because yes, you could do something more than you do for them. That doesn't make that path make any sense. Trying to do more of everything for everyone is pretending that you don't need to make choices, and that those choices don't have consequences (for example, what users you will retain or not). That is, as i said, a quick ticket to failure.
I never claimed that anyone could. You are attacking a straw man.
You are completely ignoring the actual point I made, which is that Google is a business, and every business serves the needs of those who pay it money, and in Google's case that is not its users, it's advertisers. So serving user needs has to be a by-product.
> Why isn't the simpler, more charitable answer that they are just serving the majority of their large userbase, or whoever in that user base they care about?
Why isn't the simpler, more obvious answer the one I gave, that Google, like every business, serves its paying customers first, and users are not paying customers?
You seem to be going to extraordinary lengths to ignore this obvious fact.
You do a thing (wifi alway on), so clearly everyone does, and therefore clearly we should force it!
The much simpler explanation, given that the product doesn't seem to be failing by any measure, is that maybe you just aren't who they are targeting?
One of the main reasons products fail is because they try to please everyone, and have so many knobs that the UI is horrible.
Good product design requires choice. You don't like those choices. Okay, awesome, it's not the product for you. That doesn't make it the wrong choice, or a bad design. It just means you aren't in the user base they are catering to.
Most of the time, if it turns out the user base thy are catering to is unsustainable, the product will fail, and something that does cater to that user base will take over. So how is this not a uniformly good thing?
The market has products for people who like different things - those users will decide what wins or not.
Google does not need WiFi in order to give directions to locations in the Maps application. However, gaining access to WiFi locations is beneficial to Googs in other fashions. They choose to force people to do it for their gain, not the benefit of the Maps user.
Now, you're trying to say that because some people are technically aware about things and have become vocal and push back against the tide that is Googs' overreaching into people's lives is some how in the wrong. You also seem to be implying that the fact that the vast majority of people are not vocal on the point makes it okay even though they have no idea what is actually occurring. Since they are not complaining, everything is okay, right? That's not how that works.
The location options for wifi or not, are in android settings, part of system wide settings.
No change to google maps will make this setting disappear. It is not an extra control google maps can drop.
Your entire UI tact us baseless, and makes zero sense.
You're been spreading misinformation and misconceptions through this entire thread.
Can you at least argue with facts?
This is 100% factually untrue. AGPS data is downloaded via the net, giving a quick bootstrap.
This statement of your is just used to deflect the truth ; Google wants more, moar, more, always more.
Things are simply never this binary.
Is it though? There are plenty of examples of businesses or even entire industries trying their level best to slow down changes and innovations that will hurt their profits. And in many cases the products may become worse, but more profitable, over time. In the 1970's, you could buy a washing machine in the United States that would last you 25 years. Try doing that now. I'm sure a lot of folks would like their washing machine to last two and half decades, but that's not that profitable if you're in the business of selling washing machines.
But a large part of the problem is that Google can't be bothered to follow their own rules:
Again and again I catch them ignoring my doublequotes and verbatim options.
There aren't any conflicting incentives around that, are there?
Either you respect user queries or you don't.
Also, look at it another way:
Kagi manages to do this while building on Google! It is in another league, just like Google used to be.
Even nano scale Marginalia (1 guy, 1 tower pc, 4 drives) manages to respect my queries.
There is lots of support for assuming the worst of google in this case.
Then I found Kagi last year who manages to exeed my expectations, proving that my original assumption is correct:
It is still possible to provide good results today, Google just fails at it.
I'm not saying it is easy. But it is possible.
(Extra interesting: Kagi builds on Google and Bing, two engines that drive me crazy, yet Kagi manages to be great!)
It’s perfectly reasonable to speculate about whether search quality is sacrificed in the interest of some incentive Google is trying to balance in favor of its own revenue stream. The fact that it’s complex is immaterial.
Search is a solved problem. But you're starting from the false premise that Google and your expectations are aligned, they're not. Google shows you what they want you to see, typically spam sites carrying their ads. That's not what you're looking for but from their perspective they've 'solved search'.
Don't forget that Linux became only possible because 20 years of OS research was carefully studied, analyzed, discussed and thrown away.
Ingo Molnar
From a message to the Linux Kernel mailing list in 1999.
https://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_deb...
This is delightful
Especially not when the incumbents are busy digging deeper holes.
It's not perfect, but it's scary how well it works sometimes.
Perhaps what's missing here is a garbage collection algorithm. Whatever they are using as 'roots' is not working, and they can no longer infer trust by association. They need to curate and review sites. Or maybe they already do, but the humans' focus is directed by heuristics and those heuristics have been gamed.
Does anyone know if Google still has a functioning QA department? Perhaps this is the same disease we see everywhere of developers thinking they understand testing when really we know just enough to be dangerous.
PageRank worked when there was very little tangible value besides vanity in being number one for a particular search term, and when there was little competition between sites. Once SEO became a key strategy to growing a business PageRank had to shift from finding the most relevant results to removing the things that were pretending to be relevant results. That challenge is much harder.
I contend that practically every update to Google Search over the past 15 years has been to try to defeat SEO spam. There doesn't appear to be anything else. And, unfortunately for all of us, it's a battle Google are not winning.
Perhaps it might be accurate to say that as email has declined in popularity, “spam” has simply moved to the Web.
Even worse, they are completely disincentivised to do anything about it, since presently the ads occupy most or all of the real estate most users see anyway. Buying ads to show up in that section so you aren't drowned out by spam is exactly what benefits GOOG's shareholders. And I say this as a GOOG shareholder, lol.
I've come to believe that the organic results are a niche product that Google could eliminate tomorrow and most consumers wouldn't really realize anything had changed. They'll go to the first result 99% of the time, and 99% of the time that's an ad.
They got a lot of bad sites, but they got rid of a lot of good ones too.
Now google is the modern yellow pages imho. It's more shallow than ever, but it's okay for many, and they can shake down legit sites for adwords to show up.
Totally agree with the main article, sadly however it's like what, 90% of people never go past the first page? And many still assume the top ads are vetted results that are better than all the rest.
Looking forward to leaving google as the yellow pages and encouraging more niche search engines, web rings, and similar.
We choose not to offer the same thing based on our values; no-tracking and infrastructural independence, meaning we aren't captured like other search options: https://blog.mojeek.com/2021/03/to-track-or-not-to-track.htm...
A potentially standard way to find other related feeds, instead of just consuming the content of one author/website. Curated by real humans that produce useful content (that's why I have them subscribed).
Simple thing that could have been very powerful for discovery if this was thought about early enough.
Is that so? Where?
We'd have to define 'great'.
If you mean Tik Tok dance videos and instagram photos/art, then yes. If you mean anything related to philosophy, sociology or explaining how the world works at large - no.
You don't need any tools. You just need to pay people, who have a working brain they've accomplished something with.
It's not an algorithm problem, it's a 'fuck you money' problem. Until people with a brain have fuck you money, they aren't going to waste their life fighting braindead Karens and corporate/political interests who don't want people with a brain to explain how the real world works.
Look at the amount of shit Joe Rogan is getting for having accomplished scientists on to talk about their work. He has fuck you money. Do you know how many other people have fuck you money and don't say shit? A lot. You need to spread that fuck you money around thick, so that one in ten thousand will have the guts to put up with braindead Karens of the world.
I'm the wrong person to ask for sociological links, but here's a whole flock of actively maintained black swans pertaining to philosophy and other topics:
The problems are social and economical, they require "political" solutions, not more technology.
The other is Goodhart's Law: When a measure (being highly-ranked on google) becomes a target, it ceases to be an accurate measure. Pages now optimize for search as measured by Google.
If Marginalia because the #1 search engine in the world tomorrow, I suspect SEO spam would start trying to beat it a few days later, and the results would be crap inside a month.
This is only actually a problem if there is one big search engine. Monocultures of all varieties are vulnerable to specialization, this is as true in a country that grows nothing but wheat to a world where every computer runs Windows XP. If there would be two or three big search engines with conflicting means of evaluating a site, it would decimate the effectiveness of SEO.
A lot of what I'm doing to find good results is promoting the sort of sites that Google rejects. While there are search engine optimized sites that aren't SEO spam, there is no SEO spam that isn't search engine optimized.
> The problem is that search engine results – especially on mobile – are not how you find that any longer.
I think the biggest story of the web in the last decade is the slow but steady decline of Google.
Google has fallen victim to spam. Goodhart's Law wins in the end. Almost every site on Google's front page is there because people have spent a lot of time and money optimizing it for search engine ranking. But just because a page is good at gaming Google's algorithm does not mean it is the most useful anymore. Google put up a good fight, but it seems like they have finally lost.
It also seems like the company has lost it's way. I was recently using Yandex mobile browser to browse some Russian sites and it can translate text inside images and now even voice inside videos. Chrome on the other hand often fails to detect other languages and therefore does not translate even text. It feels like even 15 years ago (when I was becoming a Google fanboy, I have since grown out of it) I would have expected Google to get there before some obscure Russian company. I get it, Russian web is smaller than English web and therefore it is probably more useful for Yandex, but Google's revenues are many times that of Yandex, they should ideally be getting there by now.
What knowledge cant be found with appending "wiki" to any keyword, what person cant you find with linkedin, what movie cant you find on a private bt tracker, what news cant you have from reddit ?
I think people over estimate what the web was before Google and under estimate how much easier it s been to find anything.
Name me one thing you struggled to find on google and Ill probably be able to describe you the nightmare it would have been 20 years ago
Ad less pages. That provide the content without trying to monetize it.
Edit: This is a better example: https://www.mechtype.com/category/reviews/
I'll show you what I'm looking for:
https://labs.tomasino.org/mechanical-keyboards/
https://www.paritybit.ca/blog/vortex-vibe-keyboard
https://wiki.installgentoo.com/wiki/Mechanical_keyboards
https://www.vegard.net/the-ultimate-keyboard-part-2-switches...
http://boxbase.org/entries/2015/nov/30/ergodox/
Where are those types of search results?
This is probably a better example: https://www.mechtype.com/xiaomi-yuemi-pro-mk02-review/#more-...
There are some basic economic realities underpinning why most people take the time to create content like this. I don't think it's reasonable to assume that just because someone expects to make a small amount of money from creating a review like that (with either affiliate links or buy links), that that person's opinion is inherently at conflict with honesty just because there's a monetary element. It may be true for many sites and many people, but it's pretty straightforward to parse whether someone is writing a review earnestly and passionately rather than for $.
Given Sturgeon's law (90% of everything is shit), why are most of the reviews 4/5 or above?
> There are some basic economic realties underpinning why people take the time to create content like this. I don't think it's reasonable to assume that just because someone expects to make a small amount of money from creating a review like that, that that person's opinion is inherently at conflict with honesty just because there's a monetary element.
I updated the post with several examples of the type of content I want you to find.
> This is probably a better example: https://www.mechtype.com/[...]
This one has affiliate links
The point I'm making is that I think your underlying premise, that all sites which have any kind of monetary element to them are fundamentally corrupt in terms of intention or conflict of intention, is flawed. I've now read multiple reviews on https://www.mechtype.com/category/reviews/ that are quite clearly the work of a keen mechanical keyboard hobbyist with sufficient details, knowledge and effort to lead me to believe the work is genuine and useful rather than merely financially motivated alone or spammy. There are many examples of what you suggest, and I agree that many sites with affiliate links and ad popups are disingenuous and fall much closer to the spam end of the spectrum. But there are still useful results if the user is remotely competent at parsing information. If your argument is that users shouldn't need that competence, and search engines should be kinder to less competent users in that regard, then that's a separate (and probably more interesting) conversation.
That site also has a clear affiliate disclosure along with this:
https://www.mechtype.com/disclosures/
>Paid Reviews/Sponsored Posts. MechType does NOT accept paid reviews or sponsored posts. All thoughts expressed within our reviews are our own opinions. However, MechType may on occasion accept product samples for review. When a product sample is provided for review it will be clearly and fully disclosed in the post along with the company/persons who provided the sample.
I did not in fact claim that. You are confusing me with someone else in this thread. I agree that some things are hard to find on google, but I reject the notion that anything with a financial element is fundamentally corrupt in terms of intention.
One incentive would be that they earn more money through affiliate (or the company is paying them) to promote one product over the other.
Google is intentionally dropping five year old results from the index and prioritizing five hour old results. They have complete control over whether they're doing this. Part of SEO is making five year old results look like five hour old results, which is something that Google did to spammers, not something that spammers did to google.
Google has decided that it is more profitable to make sure that somebody who types "chicken" into the search box gets either chicken restaurants nearby, or listicles and video covering the story on the news today about a chicken that scared away a bear. They've given up on depth.
A bad search deserves a bad result. If you walked up to me and just said one word "chicken", I'd be insulted. If you said one word "chicken" while pointing at your mouth, I'd perhaps think you spoke little English and wanted a place to eat chicken. Providing a bit more info is useful to get decent results.
As Jerry Maguire once said, "help me help you"
Google's "problem" is that they need to cater to the most common intents per query (see: lowest common denominator), and believe it or not, those intents are usually not biased toward in-depth educational content - isn't this why HN is such a nice niche?
Google could think about adding a different search result sorting option that attempts to prioritize high quality, thought provoking, or otherwise educational content above all else, but it makes sense to me why their primary search ranking algorithm works the way it does.
Preheat oven, insert chicken.
These days, the search volume went way up and my article went down to 6-10. The first three spots merely give a rough explanation of my in-depth topic. Quite insulting actually.
But upon further reflection I realized that most people these days only want a rough definition and no in-depth explanation. Google gives this to them.
The people who used to search for my keyword now add 2-3 more words to get to my in-depth article. The typical long tail.
What I find worrisome is that sometimes, the long tail query I use to find quality content gives me the same results as the unspecific short query. Some stuff is close to impossible to find without knowing a very topic-specific thing.
Try something like this (just imagine a word or two that would appear in quality content about chicken and put it into marginalia):
https://search.marginalia.nu/search?query=poultry+farming&pr...
> We might be starting from the (slightly) false premise that the internet is full of quality content.
Well maybe it isn't but at least using Marginalia I get more of the results I want for these kinds of searches.
(I've also had good luck with a query about git a few weeks ago and yesterday with one about kubernetes.)
After testing with marginalia I found that much of it is still there - it is just suppressed by Google and Bing.
edit: and to directly address what you said, excusing a search engine's loss of content because sites have disappeared because the search engine dropped them is circular.
I mean, forgive me for thinking that "chicken" is actually a term referring to live birds. Instead, humanity's view is clearly very distorted by our stomachs.
One does not use the internet for facts. One uses the internet to be directed towards where to spend their money and/or time as sold to the highest bidder, who is just looking to line their pockets with money--a digital sugar rush for digital stomachs. The internet doesn't think with its brain, it doesn't tell you what you want to know, it thinks with its stomach and expects you to as well.
My optimist side tells me we will have a revival of bookmarks in the mobile browsers too (now they are way too hidden). My pessimist side tells me Google will fight this trend to the user's detriment.
I was so frustated to see Facebook/Twitter using the fundamentals of RSS to their advantage whereas everybody should have started to use RSS for the greater good.
Most CMSes by default don't do a good job generating RSS either. Most of the time a feed is just a grab bag of posts on all topics. I don't know of any that understand inline hash tags for instance. A separate field for adding tags is just more work for an author. As a reader if I'm looking for blog posts on a topic I don't care about Grampa Moshe's goiter. Yet if I put a blog in a category in my reader software that's what I get.
In very specific circles with specific topics RSS can be really useful but outside that narrow range it's no better than the shit show we currently have.
Only problem is we've been spoiled by the thought that ALL information is on the web. And a lot of it is buried in the junky, spammy 990X part. I don't like having to watch an ad-riddled Youtube video for a simple thing like where did they hide the last screw you need to remove. But at least the information is there if I need it badly enough.
I also don't like to wade through the "sign up for Amazon Prime" dark patterns. But I do like being able to order a gadget for a good price and reasonably soon delivery.
It's funny that now i have a reflex to immediately skip the top half and scroll to the bottom in a vain attempt to find what i'm looking at/for.
At least it brought us the likes of Lex Fridman.
I think the part that is missing is how to discover the link collections to subscribe to.
I'm building https://linklonk.com that merges the collection, the consumption and the discovery:
- You add links to content you liked to your "channels".
- LinkLonk connects you to RSS feeds and other user channels that posted the content you liked and shows you new content from them. The more content you have in common with a feed/channel - the more weight their new content gets in the "For you" list.
You can also share your channel via a public link (example: https://linklonk.com/channels/programming-yUlhkwDXFb) or an RSS feed (example: https://linklonk.com/api/publicChannels/programming-yUlhkwDX...) .
The problem is see is that without an account, it is impossible to submit links. However, unlike reddit or hn, one needs to submit links to get useful content. Thus I would assume that by the 1% rule [1], it is very difficult to retain visitors.
Could you offer options for those who don't want to make an account? Maybe you could create some ready-made selections to see good content? Or you could allow to post links in a hidden way to get the algorithm going and you and some moderators approve them later-on.
[1] https://en.wikipedia.org/wiki/1%25_rule_(Internet_culture)
If you don't want to create an account, consider creating a guest account - it does not require any personal information from you. The account is just a number that is issued to you and is stored in your browser cookies. If you don't end up using it - all information associated with your guest account will be deleted after 30 days of inactivity.
If you don't want to create a guest account then the best I can do is to show generally popular content - which is what the home page is filled with for brand new users.
- bookmarking - you can search your past liked items
- subscribing to rss feeds
Please let me know if you have any other suggestions along those lines.
[0] https://www.thinking-about-things.com/ [1] https://essays.findka.com/
There is also https://refind.com
However, this is a very print-centric approach, where you print something and cannot change it afterwards, and maybe throw the old newspaper away that day. I wonder why they cannot adopt a different journalistic paradigm, similar to for example Wikipedia or Wikivoyage, where instead of publishing hundreds of articles about a topic, one a day, they would instead create a sort of curated portals which would be fine-tuned and updated over the longer time periods, focusing on quality of information.
The same goes for most online discussions; they are extremely ephemeral and stupid points are argued over and over. It would be nice to summarize the main points in a particular discussion from time to time and then continue.
So I think there is plenty of space to innovate like this.
Take a random Discord server, where we try to figure out some details of a DAO in the making. People from other timezones have to hop in and out and are welcomed by a wall of text when they come back.
So did the issue from yesterday get resolved already since you were gone? Maybe, but if so, how did that agreement come to fruition? And which part of the above wall of text do we "mark as resolved"?
What part to keep, what to formalize as a rule, etc. Those who were "in the room" when the discussion happened now have to find a way to re-tell the story in another ritual (meeting, weekly summary, task, whatnot). Sounds like the work of an editor or journalist to me.
But most people would rather add onto, instead of editing or removing only [1]. Like with maintaining legacy software or editing existing wikipedia entries, most prefer to instead just "add some more".
But even more important, some people tend to disregard even scientifically proven advice and prefer living through things themselves, no matter how redundant the experience.
[1] https://www.washingtonpost.com/business/2021/04/16/bias-prob... (which i found via HN back then)
I pity anyone who's using accessibility tools to help us the web as sites often have a timer before they cover the content with a nag to sign up to their newsletter or they track your mouse and display something else because they don't want you to leave.
I recently was inspired by a hacker news article to try using the web without an adblocker. Holy hell. It's almost unusable. I made a video where I recorded this on my PC, I'm going to do the same on my phone. I had no idea it was this bad.
In contrast, the BBC website still seems pretty usable because they don't need to run ads (at least when you're accessing from the UK). They do, however, have the utterly ridiculous concept of having autoplay on by default so if you've clicked to watch a video about a current event, if you don't click 'cancel' in 5 seconds they'll load you another video about something else (seemingly at random) when there is no reason to do so at all.
The web reminds me of the state of movie and music piracy before Netflix etc. In those days you bought a DVD and were forced to sit through an unskippable FBI warning and in some cases unskippable trailers, whereas if you pirated the movie you got a superior product. You got a copy of the movie that let you watch it on any device, let you make backups and didn't punish you by forcing you to stare at unskippable content.
If you were a music pirate then you didn't need to worry about Sony putting malware onto CDs that installed rootkits in your pc.
Software pirates didn't need to worry about similar rootkits that made the game run slower than the pirated version or generally buggered up your computer.
We've reached this state with the web. I am forced to use an ad blocker for my security and sanity. Most adverts are lies, scams and are too damn distracting.
This isn't the vision that Tim Berners-Lee had.
Look at this: http://lite.cnn.com/en
See how fast it is? See how well it runs on any old device you may have? I'd rather have this than the current web. Surely there's an in between compromise somewhere.
Like, here's a site that (among other things of interest to fans) chronologically orders Korean historical dramas: https://thetalkingcupboard.com/resources/sageuk-timeline/
No dark patterns, no clickbait, no videos, no ads, just a labor of love by someone with a lot of time on their hands.
The problem for search engines is that it's trivially easy to copy this list and put it up some garbage blogspam domain, but even if the content is exactly the same, I want this list, not the copy, because if I find this list, I can explore the rest of the site and find more great stuff. Even a spammer copies the entire site, this is the one I want because this is the one with a real person behind it who will be updating it, engaging with readers, appreciating the traffic they get, etc.
It starts to become clear that what I want my web discovery tool to do is less find individual pieces of good content but more find content from good creators.
I don't see any way a search engine can ever do this kind of discovery as well as a human curator.
The web has changed so much from when Google launched. Google is optimized for finding content about some keyword because there was actually a time where the was a legitimate problem. You searched "flowers" and Google helped you find the places to buy flowers online. You searched "guitar tabs" and Google helped you find where to get them. Google is still really good at this. Other search engines too. So good in fact that it's become almost invisible to us how easy this is compared to how it was in the past.
But now instead of buying flowers, I might want to know... where are the communities where professional florists hang out online and talk about their craft? The Google results for that query are pretty much equivalent to a random listing of websites. But I can almost guarantee there is such a community. Is it a subreddit? A facebook group? Some old vbulletin? A mailing list?
Or I might want to know... where are the blogs/channels of the top authorities on web accessibility? What are the best craft coffee roasters in Sweden? What are the essential reads on the history of Botswana? Who are the most consistently good travel vloggers in Spanish? Where can I get help with sourdough starter problems? I can 100% guarantee there are people out there that have great answers to all these questions, but they seem unlikely to be solved by search engines any time in the near future.
If someone can crack the problem of leveraging genuine, interested-motivated human curation to deliver results without falling victim to abusive, quick-buck-motivated spamming... they could replace search engines.
That's the tension right there. The moment someone solves the first part is the moment someone will try to hijack it for their own gain. The Internet is dark and full of husslers. They want to exploit people's goodwill and trust---that stuff is costly and doesn't scale, which makes it valuable. If you want genuine interactions, you go to a quiet corner with a handful of folks and be very very careful about who you let in.
The builder would need to load the first few hundred thousand results and then add some sort of user moderation system, where search results get a reputation and site submitters get a reputation as well.
Donation funded, with maybe one sidebar section for curated ads that are tied to search results and not with a tracking cookie or fingerprinting.
With a few tweaks, that would be a really good start, right?
Once it reached a certain point, it would be overrun by the same systems, using human and bot armies to bias results towards the results that would make them the most money.
As long as there is a profit to be made somewhere, people will undermine any obstacle placed in front of them to make it, and I can't imagine any fix for it that isn't "run your own servers and pay for it out of your own pocket".
Bingo. What's interesting about this is that while maintaining a list of all useful content on the web is probably intractable, maintaining a list of all useful content producers might be doable.
You could have moderators for each niche, and as the list of producers for a given niche grows, it could split into sub-niches which would get their own moderators responsible for keeping track of quality content producers.
Sort of like an awesome list for content producers.
I'm certain this already exists in some form. The original web was build around curated content like this. But people aren't used to looking for content this way.
EDIT:
Saw dmoz.org mentioned below which sounds like exactly what I'm talking about. Apparently they're called web directories[0]. Sadly DMOZ is dead but appears to have a community fork called curlie[1].
[0]: https://en.wikipedia.org/wiki/Web_directory
[1]: https://curlie.org/
Some of it, though, is a natural consequence on the commercial potential of the Web being realized. "If you put up an abreuvoir, the pigs will come", to quote the German supreme court (out of context).
Perhaps one could design another protocol that is designed so that it cannot be commercially useful. Because we live in an "attention society" (attention = power = money) this means in all likelihood that such a WWW successor would by definition not be able to get as big as the WWW (which is an upside and a downside at once). Decentralized ideas are very appealing, but centralization is used to make the existing Web fast and convenient - this is also something that will have to be addressed somehow (maybe there is a way to have physical centralization but logical distribution and a way so that the two levels - physical and logical - can't see each other).
Businesses want to retain customers and have repeat customers, not necessarily just active user numbers. Do you think it is not in a businesses interest to do so?
> We don’t create content for the web and for longevity. We create content to show ads around it.
Not just ads, but regarding social, what is more valuable to people, a library or a mechanism to get attention and influence from people?
The broader "rat race" here is how people must earn a living in todays's society
> I didn’t have to wait for a certain time to learn about a new movie – I could read up on it any time I wanted.
Who created that content and if it was from a professional source, did you benefit from it?
"information wants to be free" sure sounds great if you're not the one supplying it.
It doesnt sound like the author was a contributor to the free and open web. I'd be interested to hear a perspective from one of the bloggers he referenced who tirelessly produced content and who's blog died in 2010 with nothing to show for it.
You would have to agree that this scenario would at the very least be frustrating, not only for you but also for the people who want your free content, right?
My "information" here is not "content" like How to fold a Shurgard moving crate or Movie Reviews i.e. produced with intent to inform/educate the general public about specialized topics. My post is simply part of the social web of personal opinions, and he doesn't seem to think twitter/instagram/tiktok are solving his problems.
Forums do exist. Hacker news and Reddit does exist. But I think the OP is not satisfied with Reddit and would not be satisfied if we just had a bunch of additional forum sites.
By search engines they mean google. Other search engines (yandex, bing) have pretty good image search.
DuckDuckGo is just a front-end to Bing, though, and as of mid 2021 I've begun seeing more and more low-effort SEOey links in the first page. Google started the trend, but it seems like all major search engines are increasingly disconnecting search results from the actual search terms.
I actually find myself using archive.org to search if I need super-specific old-school AltaVista-style results.
I dread if Reddit going public ruins the site (well, ruins it more than the redesign did), because currently, appending "reddit" to your query is the only way to get results that seem like they're actually written by a human.
edit: oh, and in regards to his bit about web notifications -- I totally agree that they're one of the worst things to happen to the web, and it's the one web standard I'm actually happy Apple refuses to implement in Safari. It's only used maliciously, and it's a horrible UX. The amount of times I've looked at the notification shade of a non-tech-savvy Android user's phone and seen a bunch of ads from random websites, it's obviously not worth it. Users have been so trained to click "ok" and "accept" on the millions of popups that are shoved in their face as they try to navigate the web, it's no wonder they're enabling notifications on every website they enter. Get this stupid popup out of my face so I can see the content!
I have to disagree on that one. If you search for some API or error message, 9 out of 10 results will be these malware-ridden autogenerated clones of Stackoverflow. I can't believe Google engineers can't do something about this. They must be using Google to search for this kind of stuff everyday...
This is becoming a meme. Yes, I also think Google results are getting worse.
But: The real underlying core reason for this is that the entire society has come online by now. You can't compare the old, elite web to the mainstream web of today.
At the end of the day, Google is just a machine optimizing for the masses.
And that is the difference between the artisanal/hobbyist web of the pre-Facebook/Youtube/Blogging Gold Rush era, and what came after it. The web is dominated by businesses that consider content to be a means to an end. Kids' cartoons on TV in the 60s were designed to sell toys and merchandise after all.
This isn't just limited to websites that 'review' things with a metric tons of Amazon affiliate links, or gate white papers behind an email signup. Youtube and Instagram went this route years ago. At least YT is still generally good quality. IG feels like a wasteland of influences shilling some product with a discount code that attributes sales to them.
I think this is an interesting observation in the context of Apple adding PWA notification support to iOS. It's also the reason I won't be enabling any notifications from Safari once that feature goes live.
A complete lack of examples. And when there is examples, they're absurd.
Of course the results are general for a super general search intent keyword. Of course the results are bad for super obscure 0 search volume keywords.
I don't think Google's perfect, far from it, but a lot of the criticism in these types of threads is limp and based on unrealistic expectations.
Consider the difference in results between Google and what you might expect if directories rather than search had remained the primary way of finding things on the web. A search for "instant noodles" assumes you either want to buy them, find recipes for them, or read about whether they are healthy. But a directory topic "instant noodles" would find sites about instant noodles. And sure enough, if you hunt through search results, you can find things like https://www.theramenrater.com/ which definitely fits the bill, but it takes some hunting. It's a "page four" type of result (in reality, it was on page five).
So, yes, this is a general topic and you get general results... but that's not how directories worked. A directory listing for "Instant noodles" would never have included Walmart and Amazon searches. It wouldn't have included random news articles on whether or not they're healthy. And in the days of blogrolls and web-rings and such, once you found one site, you usually had a way to get to more human-curated related sites without even returning to the directory.
Not denying there's some rose tint here, but there's a big difference in how human-curated topical browsing differs from search.
Your example of Ramen Rater is the #1 result for the query [instant noodle reviews]. The more broad [instant noodles] has Wikipedia as the #1 result, which covers the "about" intent case you mentioned.
I didn't specifically want instant noodle reviews. I just wanted sites about instant noodles. If they happen to be reviews, cool. If they are photo galleries, that's cool too. Message boards, also cool. But I don't want articles or encyclopedia or dictionary entries about instant noodles. I want full sites, or at least large subsections of sites, where the primary topic is instant noodles. This is what directories list. Sites where you open the site and ask "what's this site about" and the answer is clearly "instant noodles". Not at the page level, but at the site level.
We enlisted a designer and she sent us over 30 incredible online stores that we had specifically been looking for that none of our previous searching lead us to. Most were small independent places and had incredible quality/value and originality.
It was exactly like taking the red pill.
Which means that something that for you as the 99% tail is not good anymore doesn't matter for my mom that doesn't even know what you're talking about as the web being different before. It's the digital version of the old "Nobody goes there anymore, it's too crowded".
That being said I believe a major part of the problem is the asymmetry between consumption and creation of content. If creating content was somehow harder you would find way less auto generated attention grabbing websites.
Compare a blog to a podcast. A podcast is long form content. You know clearly how a podcast is monetized, and accept the advertising or Patreon or whatever.
A blog, OTOH, might literally be malicious to your computer, privacy, and attention. You dont know if it’ll try to hijack your browser or what it might track about you. You don't know if you’ll have a pleasant reading experience or get twenty prompts to signup for a newsletter.
Intermittent reinforcement being the strongest kind, we begin to associate the web / blogs with malicious spam that slows down your computer, even if many aren't.
'VanVan motorcycle starting issue reddit'
'FJ Cruiser clutch chirp forum'
Adding the "reddit" or "forum" tag makes the results way more useful. Is there a way to tell Google to only include forum-like results?
Is it not the natural result that monetized content dominates free content on any platform that permits both? After all, monetized content, because it leads to some sort of revenue for the creator, has an advertising budget, and free content does not.
The only way to to avoid such advertising is to not permit it on a given platform, or to require it to be explicitly financed (e.g., a subscription to a digital magazine). The internet, as a platform, has no such rules, and so it is dominated by advertising.
It’s just not that easy.
This whole post is a disguised attempt to get help with this, isn’t it? (I say this after spending 15 minutes trying to find a tutorial ;-)
You mean the "app" that has no search and requires you to sign up to see anything, then pass some obscure initiation ritual by the mods, then be pestered with various bots. And that app doesn't even make money yet, wait until they decide to become profitable.
Sadly, it's a proprietary space, yes, but still.
The difference with "social" networks (and now the bad state of search engines) is that you actually get to be part of a community. As opposed to shouting in a loud bar to compete for likes to get heard better maybe if some algorithm likes you.
I tire of answering the same question hundreds of times
Discord communities are nearly as much readable as geocities pages were as well, but that is not a compliment.
I agree with the author's conclusion, but the example is a poor one. I would expect the sort of content to come directly from the manufacturer. Is it surprising nobody has blogged about a specific moving crate?
Anything that is at the mercy of an algorithm will be gamed once they figure the algorithm out.
Capitalism in this analogy is your ownership of the hammer, not who you are or are not obligated to lend it to.
Though it’s not really a great example, a lot of things are missing from this example, such as how you came to have ownership of the hammer, what property rights you are entitled to with the hammer, and why you even need to have ownership of said hammer in the first place. Alternatively, there is some sort of public ownership (as opposed to private, not necessarily implying state ownership) of the supply of hammers, and you are able to obtain a hammer to fit your business needs as needed.
I don't care about the word 'ownership' or what makes it valid, I care about limiting access to limited resources, be those resources hammers or the land my ancestors are buried on.
Ownership is who decides access (a more appropriate term would probably be who governs utilization), not where access can or cannot be limited.
So to directly answer the question, yes, within certain definitions of “socialism” bobs access can be limited.
Again, ownership is the operative idea here.
An obvious example would be feudalism.
The critique ultimately traces back to Marx, who asserted that social relations within a society are dictated by the “mode of production”.
I don’t necessarily alway agree with people when they say “capitalism caused this” but this is by best attempt at relaying the reasoning.
Ultimately the critique that socialists are making, to bring it back to your analogy, is that most people don’t own their own hammer, capitalists do, they just do all the work with it to build houses while the capitalists take the lions share of the profits.
Since few years I hate using Internet, with every year I using it less and less.
Google wants to intermediate reality for you. A perfect example of that is how it tried to kill rss - first by making a fantastic rss client so that most people used it, only to then to stop the service. This was not due to budgetary considerations, - it was by design! RSS is puts you directly in touch with the content you like - this is a threat to a company such as google and whoever runs them.
If you dig deeper, you will see that google has long-standing associations with military intelligence.
"Both Brin and Page were working on the Stanford Digital Library Project (SDLP). The SDLP's goal was "to develop the enabling technologies for a single, integrated and universal digital library" and it was funded through the National Science Foundation, among other federal agencies.[9][10][11][12] Brin and Page were also part of a computer science research team at Stanford University that received funding from Massive Digital Data Systems (MDDS), a program managed for the Central Intelligence Agency (CIA) and the National Security Agency (NSA) by large intelligence and military contractors."
from: https://en.wikipedia.org/wiki/History_of_Google
If you look at the CIA's venture capital firm - In-Q-Tel, you will see that they funded google too:
"In-Q-Tel sold 5,636 shares of Google, worth over $2.2 million, on November 15, 2005.[10] The shares were a result of Google's acquisition of Keyhole, Inc, the CIA-funded satellite mapping software now known as Google Earth"
https://en.wikipedia.org/wiki/In-Q-Tel
(Why In-Q-Tel require a funding arm at all, is another question you can ponder.)
When the Edward Snowden fiasco was running, I also recollect reading about 3-letter agencies having offices in google's offices - no questions asked.
TLDR
If you think that the 3-letter agencies are about keeping you safe from the world, you have this the wrong way around. They are about managing the people - they cannot allow information to be unrestrained. Google is an essential part of the infrastructure.
Even if you don't like my assertions, I'm providing sources for them!
I have to say downvoting without engaging with what info is provided is one of the worst elements of this site. Surely downvoting should be used sparingly. It shouldn't be that just because you are a power user (with 500+ upvotes) who feels discomfort with whatever is being raised, right? Perhaps its hard to justify working at a big tech company, but downvoting would be a rather self-serving ideology, don't you think?