What software engineers should know about search (2017)
scribe.rip
scribe.rip
No really. Look over people's shoulders sometime.
They'll just go to Google and type in their search followed by terms such as Wikipedia, imdb, Stackoverflow, YouTube, Bandcamp, Amazon, eBay, Yelp... all sites that spent a lot of time on their search and have done quite a decent job. Oh well.
So unless you really need it for some critical reason where you know your users aren't going to do their regular patterns of going through one of the general engines, close the ticket as out of scope and go home early.
Don't bother implementing hard to do and expensive to maintain features that nobody will use. It'll become more headache than fun real quick.
VueJS and Tailwind CSS both are indexed, but on those particular site I use the web site search _when I'm looking back for the reference of something I know_, because it's faster and more accurate than googling.
Granted, it's rare, but if you manage it, it's great.
Not consciously. They'll just do it and expect it to work which is why effective indexing and "SEO" (as in, the engine can scrape the content and crawl around without getting confused) is likely the actual work to do to implement "search".
I cofounded ZIR AI to provide ML (vector)-based search as a PaaS solution, similar to what Algolia or Elasticsearch do for keyword searching.
We have a demo (https://zir-ai.com/demo) running over Quanta Magazine articles. Not only can it outperform the keyword search embedded on quantamagazine.org, but one some queries, it even outperforms Google with a site restrict (e.g. how old is the universe site:quantamagazine.org).
> Granted, it's rare, but if you manage it, it's great.
ML-powered search will make this commonplace, once the tech goes mainstream in the next 2-3 years.
Really depends on what kind of content your dealing with. Search can be as hard as you want it to be.
A search engine (and its backing index), just like a relational database, is a technology that allows you to build all sorts of solutions on top of it. Nobody says don't use a graph db because Facebook already has the best friend of a friend solution you could hope to find.
But products are constrained by requirements and most of the time the manager thought search was in there, after a little talk, search was not in there any more.
There's resources, deadlines, budgets, contracts to satisfy, blah blah blah. The insight is this costly feature can usually be shaved from the requirements with a little conversation.
A big secret of "10x" programming is realizing "90%" of the work isn't truly necessary.
But yes, search is totally a fantastic thing to do if you have the space and expertise to do it.
I run https://volt.fm and search is one of the most used features. If it didn't have built-in search, I doubt any of the users would use Google instead to find the artists/songs they were looking for.
It looks like there is a modern alernative - seems a little more complex to get going with which might be why it's not so common.
It's not the greatest, as it seems to prefer to return references in D forums in preference to the manual pages, but it works well enough.
It’s not common, but it happens. For small sites it should possibly happen more. If you’re working hard to achieve mediocrity, you should put that energy somewhere else where you can at least get to good.
I want them to believe that if it is there, they’ll find it with search. But that depends on how good it is and how helpful the feedback is.
If the GP's right and people really do just drop out and use Google every time (even though the target site has a search function), then I think it's hard to argue that it's bad UX. Unless you want to argue that the specific sites' search functions are poorly made.
You lose trust in the search function very quickly, if you gain it at all. Whether yours is effective and helpful does not only depend on the search implementation but also on whether your site actually has the information (that it should or is assumed to have) to be effective and helpful. A lot of factors come into this, representation, content, design both for search results and for the actual target pages.
Now if you have the trust, are helpful and effective, then the UX is drastically improved.
For one, search engine results have been degrading. There is _so_much_spam_ out there. Countless SEO sites that just crawl the web and generate crap output to show you ads, some seem to be handwritten as well. It is increasingly probable to get top results that are just presenting "stolen" content in some form or another. You get low effort blog-like posts that are just restating things they read in a discussion that is actually not the primary source, but a generated site, which is referring to the actual information somewhere else.
Secondly if you have in-built, decent search people will use it and they will be happy for it, because you'll be presenting them much better suggestions in a better way and you'll do it faster.
Think of some of the _best_ sites like MDN. Sure, you might enter the site via DDG or Google (in the former case you'll be searching directly like so: "!mdn [search_term]"). But when you're on the site you'll be happily navigating it via links and search, because it's just a very good site.
Other examples of this are: wikipedia, tailwind, clojuredocs, reactjs, hn.algolia...
They all have rich content, good search, useful links. It's a very effective combination.
Note I'm not saying we shouldn't care about search engines. They are extremely useful and important. I'm saying if you have the type of content that benefits from being navigated via search, then consider direct control over this.
And for something like EBay, or Amazon, they'd be mad as hatters to send any search traffic on their website out to Google where competitors are buying ad space. They don't really have a choice, they have to try and implement their own search even if only a small number of people use it. That is likely free money as far as they view the world.
Granted, a lot of sites out there that provide a search function do not necessarily need one, and would instead benefit from an organized, hierarchial index. With these types of websites, using the search function only adds unnecessary friction.
In my experience so many sites have searches that return bad results, restrict your ability to see a list of results due to some poorly thought through typeahead functionality, or simply do not work without third party scripts and cookies that I often resort to "site:<domain>" in ddg or worse, google.
I find hierarchical indexes only of marginal use when looking for information. I can only know where in the hierarchy something is if I'm very familiar with the website - because many things could logically be in more than one place.
Yes!!
So many blogs without a list of posts in chronological order. So many sites without a list of pages. So annoying.
Also Google search is very "fluid". Sometimes, for example for searching documentation, you need something more advanced.
* Are Google indexes updated frequently enough to capture important changes?
* If you consider certain search terms synonymous, would Google do so too?
* Are Google result rankings compatible with yours?
It used to be that you reliably had the Wikipedia article at the top of your results to provide context and basic information in case you didn't know what your search term means, you could expect it to be there if it exists. Now you have to waste mental energy hunting it down, which is a waste if there's no article at all.
Their results seem to have been turning to trash but then again, so has everyone else's.
There's a few explanations. Easiest one is me, I'm getting dumber with age or have changed my standards. Second one is they are all using similar approaches and SEO'ing has ruined search. Third is Google sets "the standard" and the other engines tweak themselves to follow the goog, regardless of results.
Reality is it's probably all 3 and a few more that I haven't thought of
Without the ‘reddit’ qualifier, the results are nearly always spammy and useless (as much as modern Reddit tries to compete here notwithstanding)
Either search is not an important feature, and a suboptimal, DIY implementation that looks OK is good enough. Or, search is a primary feature and then you need control over it. IE, if you have an online store, travel site or dating app with a search based UI, then you'll probably roll your own.
There are cases where google really is the best way. As you say, stackoverflow, wikipedia & such. Even so, you'll eventually roll your own. Spolsky original UI concept totally leaned on Google for search, but SO still has its own.
At some point, you'll need results to take inventory into account or make autocomplete smarter about tags... and now the headache is yours anyway.
It would have been cool if the web had really developed into the hopeful, "semantic web era" where this kind of approach works. That didn't happen. Half the game is over control, and controlling UIs matters the most.
Also, the power of pagerank has dwindled as the web itself changed. Links aren't what they used to be. That makes Google relatively worse at what it was once best at. Google search as a whole is much, much richer but most of what makes Google good today has less to do with your use case anymore.
TLDR, Maybe google search works for searching wikipedia, which is perfectly loyal to the original WWW concept. Even they have a DIY search. If you work at Reddit though, and your search sucks, google will not fix this. Your search will just suck, and your app will be less usable.
Most stores have garbage search and even worse filtering. >98 % of stores have only categories for filtering and only trash like "price" for sorting. In the 2 % of stores that can actually do faceted search the number of facets is too small (e.g. shoe stores which only have a "size" facet plus categories) or the data in the facets is garbage. This is likely a big contributor to sites like Geizhals existing, whose sole purpose is to offer decent search and filtering.
If you have bad search, and search is a primary feature, then your ux sucks. I just don't think "just use google" is a viable alternative to making search good.
If you're making an online store, search is probably important, and a major determinant of how well the software works. The best way to handle that is with search that doesn't suck. It may be hard, but that's the job. Fair point that search is not trivial to do well, but that doesn't mean it isn't the job.
That doesn't mean that having a great search can't help your shop get repeat customers, just like good service, fast shipping or a good checkout flow can.
Or as former Google design ethicist Tristan Harris wrote, "Whoever controls the menu controls the choices."
https://observer.com/2016/06/how-technology-hijacks-peoples-...
As a user, there is no site that does not have search, no matter how good it is, that does not feel quite horrible because of that. Even the search by Google feel horrible -- romhacking comes to mind. It is also frustrating because some instances of title searches are notoriously unindexable, while a nice site search lets you narrow the context through tags or model specific attributes (system, genre, ...).
The advocacy is to push back on search as a difficult and often unnecessary problem to solve.
The caveat is my increasingly toxic pattern to push back on almost everything as unnecessary. It's probably overly antagonistic.
I'm a techno pessimist programmer. I didn't understand this attitude when I was younger but then again people often become that which they fail most to understand
For instance, one Front-End engineer was really proud of using a very recent and trendy framework to rewrite the whole site from scratch. After deployment, people are ordering the wrong products. His reply was, "keeping the query filters in sync is too hard of a problem". It's really not.
rather uncharitable description - that PHP implements dozens of interesting features and recall. It is true that it is a mess, however.
If someone has any idea why forum search experience is so bad and know how to improve it, please chime in. I have my hunches, but let's not get ahead of myself.
Privacy may be a thing, however.
> Biggest advice I can give is you probably don't need search if you're indexable by search bots.
A/B testing showed significant (I don't remember the numbers) loss of visitors on that page. When presented with our own internal search, visitors would stay on the site the vast majority of the time. But when presented with the google search, visitors would often leave. I would say that the results that google returned were no less relevant than those our internal search provided, in most cases near identical except for ordering.
I've chosen to block them all and only spread the word organically instead.
I have yet to begin tackling the search problem :)
It's such a bad practice to let the users leave for whatever reason.
Sites are putting every effort not to leave their site and this is anti pattern.
Also I'd think the site doesn't even enough budget to put a search function and the business or the manager has some problem.
I use that for my static site as it means I don't need to introduce any moving parts. Probably not the best search experience but it gets the job done
I'm someone who does this, and I do it because I've found that it either works better than a site's specific search and/or it's so much faster to go to Google and search than to try to find the search functionality of the specific site.
If your main business is selling stuff that users find via your search features, then doubly so of course since you are literally losing cash every time your search ends up not doing its job. Easy to measure too and most eCommerce companies that survive long enough do that obsessively because it shows in their recurring revenue if they don't.
Also, if your competitors have awesome search and you don't, your users might realize and jump ship. If you have content that is great but nobody is finding it, you might want to fix it by allowing them to find it via better search functionality.
If search is actually not critical to your UX or product, then by all means, cut corners. Google will happily redirect users to your competitors as well. Make sure to give them plenty of money for keywords. If you don't, somebody will. Either way, your analytics will tell you how people come to your site and what they do once they get there.
Either way, it's not that hard to build a decent search experience if you know what you are doing. The key point of this article is that many engineers kind of don't know what they are getting into and mess it up.
Most people think of search and immediately think of large data sets, but the problems that plague smaller datasets are equally interesting. It's less about performance and more about relevance. For e.g. searching across multiple fields for a compound query like "taylor swift style", requires breaking the query into segments (taylor swift | style) before searching for the appropriate fields. There are also a class of problems that traditional search engines that rely on BM25 or TF-IDF for ranking cannot reliably solve (e.g. searching on small texts like titles) where you have to consider distance between matching words (which TF-IDF and BM25 miss). Lastly, there is also personalization which is almost always left as an exercise to the reader :)
The latest RC build is `0.22.0.rcs18`.
Lots of great additional functionality on top of search, for example security, with scoped API keys and the likes that we're looking forward to making use of.
Major search engines have grown to the size where it is theoretical more beneficial to train a model and then query that model. In theory this will win but in the real world this will not work as we already can see with the failing Google search engine.
Github search is a great resource for any engineer. It has saved me so many hours trying to figure some API out when I know there must be so many people who have used it before.
"SomeApiCall" extension:.[js|rs|java|cs|etc.]
Or to find projects that use a dependency you are not sure how to use
"SomeDependency" extension:.[toml|json|etc.]
[0] https://docs.github.com/en/search-github/searching-on-github...
HN's special treatment of medium.com links doesn't apply to scribe ones.
Previous discussion from the time, of this excellent article: https://news.ycombinator.com/item?id=15231302
However, I personally am now focused (and bullish) on DNN-based semantic search. Having built several search experiences based on it I'm convinced it is the future.
Regarding DNN-based semantic search, what would you say are the top notable works that one should dive into? What search experiences have you built that you feel was strongly enhanced by DNN-based methods?
When building an inexpensive app, the client will often ask for search. The UI designer will oblige and add a search bar to the app. Neither will give much thought to what the search will actually do except to say "make it work like Google".
> Use existing technologies first: As in most engineering problems, don’t reinvent the wheel yourself. When possible, use existing services or open source tools. If an existing SaaS (such as Algolia or managed Elasticsearch) fits your constraints and you can afford to pay for it, use it.
I work at an AI search company (Relevance AI) and even we see that InstantSearch.js is all some people need in terms of UI. We created a version of it that uses our NLP-backend but is still the same Algolia components on the frontend: https://www.npmjs.com/package/@relevanceai/instant-relevance
The reason was because those components work. Think these days you'd need to ask a lot of questions before completely rolling your own UI or NLP handling.
seriously though... default confluence search-as-you-type box is horrible, however sometimes (often ? always ?) there is a more traditional search endpoint available that seems to give better results. see if either of these works for you:
<your confluence domain>/dosearchsite.action
-or-
<your confluence domain>/search/searchv3.action
I've been running a news search engine API [1] for about 18 months, and I found a lot of new things.
The biggest takeaway (IMO) is to use existing SaaS at first. We've been using ElasticSearch Service for our v1, and it worked great. 0s downtime with no master nodes.
And yes, ElasticsSearch itself is just a tool: you'd have to write your own search logic on top of it.
Also, start with something "strict" that gives decent results for ppl who won't read any tips or docs: 99,99% of people who use Google Search don't know about any advanced tips, and still get relevant results.
My experience using natural-language search queries has given me a set of expectations, as a user, that set a high bar.
I’ve written a fairly substantial backend for a search, and most of my original frontend code has been ripped out and replaced with new frontends, by now. I feel that the new frontends are an improvement, but we still have a fairly strictly-guided “guardrail” search.
Google do this and they're very good at it, but a lot of companies need their own search capabilities - think about those internal help pages. They usually seem super outdated compared to the semantic search capabilities of Google.
In the end there's only a few components to it, you use some NLP model to create what are called 'dense vectors'. Then you put all these dense vectors into an 'index' which is optimized for fast search (that comes under the umbrella of ANN search). Then given a new query you just compare that to the items in the index and return the most similar results.
I covered the search part of it (https://www.pinecone.io/learn/) with Pinecone, who provide managed-search - although here we mainly focus on Faiss (a great engine from Facebook AI). However, we're also looking to create some content covering the first half too, which is 'how to build dense vectors' using models like BERT, we have one post so far on that (https://www.pinecone.io/learn/dense-vector-embeddings-nlp/)
Google does have an UX bug in its search/map product.
Let's say I google for "kuka deutschland". In my case, the first result is a link to KUKA, with sublinks to different sections of their website.
Next there's a map with A-, B-, C-markers, and below them fields with A: KUKA Deutschland GmbH [other info, website, route] B: KUKA Systems GmbH [...] and so on.
If I now click on A: KUKA Deutschland GmbH or on the map, then the map opens completely, with a left sidebar containing more related places, a popup with the currently selected item open which contains plenty-full information about the marker on the map.
The issue is that this is not a Google Maps map, it is not hosted under maps.google.com, but any normal person would expect to be able to interact with this map as if it were a normal map: have a button for satellite imagery, 3d-view and all of this, the ability have the normal Google Maps sidebar at my disposal. But this does no exists.
The worst thing about this view is that there is no link to Google Maps which would open that same view in Google Maps.
Take clothing search, "blue t-shirt", what is the search space? Most engineers would naively shove the name/description into Elastic Search and then be surprised when it doesn't work. Why? Because most product descriptions are not as explicit as "This is a blue t-shirt" (see also YouTube video descriptions).
What we actually found was that by searching categories (where blue t-shirts is roughly a category), and then just filtering clothing items to that category, search worked far better. Understanding what terms people used (categories/colours) and what was actually in the data and what data models we should be searching (not products!), we built a far more effective search experience.
If the whole string matches facets, then the full text search never gets touched, as in the example above. But if the example were e.g. "blue anime t-shirt" then you could facet on "blue" and "t-shirt" while FTS only has to contend with "anime".
It really is the gift that keeps on giving in terms of interesting problems, you get everything from large scale graph processing, esoteric data structures, down to bit twiddling optimization, language processing, dealing with seriously messy and sometimes adversarial real world data. Everything is challenging, but none of it is impossible, even as a solo project.
Want black shoes? I could search for "black shoes", and receive shoes with the brand/product name "Black". Or I could select 'category: shoes' and 'color: black' in the drop-down box. Hey, look, now I have a list of black shoes.
black shoes site:https://shittyeshop.com
a) speaking "black shoes" into your phone is a better experience than scrolling down an arbitrarily long list of categories.
b) Is shoes a category? Is it clothing? Is it boots? Oh I need an ontology now? Will users find it intuitive to have to drill down on it???
c) Wait, the item comes in with an automatic description from a thousand different vendors. How do I decide what part of it maps to my "color" field? What if the color isn't in a separate field? How do I make sure that this item isn't dead in my inventory? How do I make sure that this popular item is retrievable for me too and not just my competitors that have a better search??? In this case just searching for "black" anywhere in the fields of the product, weighted for significance, gets me valid results without communicating to the user that they filtered on a color.
I don't do any of this. I filter by shoes and look with my eyes. Retail search is a poor example because existing retail tagging is poor.