Can the Octopart guys comment on why no existing software was up to the task? Also, I'm curious if there's something that makes ThriftDB a particularly good choice for HN search as well.
Lucene/Solr is great for search but it's useless without a fast datastore. Additionally, you can't expose it to the open internet so you need to build a REST API wrapper around the DB/index.
One particularly acute problem we had at Octopart was search across a frequently changing schema. We have 15M parts in our database so it was a real pain to change the schema. ThriftDB uses the Thrift serialization protocol internally to maintain a flexible schema so you can change the schema independently of the underlying data.
The current implementation of ThriftDB takes advantage of solr in smart ways to simplify app development dramatically. At Octopart we found that we were spending a lot of time building custom search solutions. Now we're using the same backend technology to search for electronic parts on Octopart and comments on Hacker News.
Using ThriftDB for search is like going from a compiled language to a scripting language. It takes care of a lot of issues so you can focus on app development.
Hope that explanation helps! Please let me know if you have more questions.
Solr can be run without a schema quite easily, or use the very common trope of the dynamicField typed schema (string_, text_, int_* or whatever)
Lucene/Solr is great for search but it's useless without a fast datastore.
This is true at real scale (EN has 500m docs in our bigger solrs and we have to back it w/ TT) but for 15m docs, no way. Stored data in lucene/solr is certainly fast enough at that level. Think <50ms including data even on AWS.
Additionally, you can't expose it to the open internet so you need to build a REST API wrapper around the DB/index.
This I don't get. Surely you have a web app layer that talks to your services? That is, you don't have a text field in HTML hitting solr on port 8983, you call your web app's search which then hits solr's HTTP API?
We're using solr dynamic fields in our current implementation of ThriftDB. ThriftDB adds a layer on top though so you can change attribute names on the fly.
This is true at real scale (EN has 500m docs in our bigger solrs and we have to back it w/ TT) but for 15m docs, no way. Stored data in lucene/solr is certainly fast enough at that level. Think <50ms including data even on AWS.
In our experience, stored data in lucene/solr doesn't scale well.
This I don't get. Surely you have a web app layer that talks to your services? That is, you don't have a text field in HTML hitting solr on port 8983, you call your web app's search which then hits solr's HTTP API?
It sound like we're saying the same thing. If you use solr you're responsible for the web app layer. With ThriftDB you get a JSON REST API out of the box for every collection you create.
But do you guys have a lot of experience with solr / lucene scaling (I mean well beyond the 15m docs octopart has)? What happens when an API customer starts ingesting >100m docs? And then 10 do it at once? Are the indexes on different boxes? Are you on AWS? What's the disk backing it? Does each index fit in RAM? Does the user have any control over caches, tokenizers, stemming, triefields? Like... when I add a date, you're not indexing it with millisecond accuracy, are you?
After EN gets bought on accident by the norwegian fish cannery I plan to retire on giving speeches of 5 years of Solr scaling woes. Unless Otis G is running it there's no way I would trust a hosted solr solution that I didn't have full control over.
We have a lot of experience scaling faceted search with solr. 15M docs might not sound like a lot if you're doing full-text search but faceted-search adds another level of complexity:
http://octopart.com/partsearch/#search/requestData&q=cap...
As far as scaling goes, most of the demand right now is for smaller indexes so that's not a problem. We know how to scale out though so we're pretty excited to get customers with >100M docs.
We're hosted on AWS but we're still iterating on the architecture. The indexes fit in RAM. Currently you can't control caches, tokenizers, stemming, etc. but that's in the pipeline.
Good luck with your new Norwegian cannery owners! I hear you on hosted search. We're trying our best to create a headache-free hosted solution.
Here's a link to the ThriftDB docs in case you'd like to learn more: