111 karma · joined September 10, 2010
But it's moot, since Apache Lucene is part of the Apache Software Foundation and has much stronger promises about its licensing and governance. Which is not a small reason why Lucene is the de facto standard for search technology.
It's because Heroku's choice of vendors shouldn't matter to their customers. They see it as an implementation detail, and their responsibility to manage.
So I don't think that's an obfuscation. The people I know at Heroku all have an attitude of, "The buck stops here."
Addon provider here: haven't heard anything official from Heroku on this, so this is my own personal speculation based on the current public Provider API. It seems that Heroku Postgres and Redis are available, and while they're _technically_ addons, they naturally have access to somewhat privileged APIs and architectural information that other addons do not have.
Currently, when an addon is provisioned, we're given a region identifier for the US East and EU public regions. My uninformed guess is that Private Spaces amounts to "your dynos run on servers in a private VPC." IF the Postgres and Redis integrations were "quick 'n dirty," they could very well get provisioned within the same VPC. However, it also seems plausible that AWS VPC peering can be used for other addons to provide their own Private Spaces support.
So it seems to me the question comes down to whether Heroku can (and/or _wants_ to) support VPC pairing with addons via their Provider API, so that other providers can provide their own private spaces.
In the context of reasonable hobby/development costs, I our $10/mo plan fits well for that and is pretty popular. Pricing wise it compares well to a $7 dyno or $9 Postgres instance. And addons are prorated to the second, just like dynos.
Echoing mrkurt, ES is indeed hefty to run.
Elasticsearch is going to require a hundred+ megs of RAM just to run an empty node. And then you'll want enough heap and CPU that you're not GCing yourself to death while importing data, or while churning through filter and caches. And then enough memory to run those fancy aggregations in Kibana. Times the number of nodes you want for redundancy, and for your master-role quorum.
It's possible to do long-tail low-volume ES hosting for hobby development. But to do so you need to dig in and consider ES hosting as a holistic system. We've worked hard at Bonsai to build a hobbyist-friendly hosting system. Our approach is to fork ES itself to introduce native multi-tenancy. That means everyone in the cluster is amortizing the overhead of all that JVM memory (and redundant data nodes, and master-node quorum, a load balancer and authenticating reverse proxy, and analytics... etc...)
The problem remains that ES, while easy to spin up, is still a pretty heavy system to manage at scale. So it's generally a lot easier to chuck some extra resources at it, maximize your isolation, and eat the higher entry cost.
Indeed; more generally, I said CLI. TLDR of the previous link: it's an interesting general-purpose take on using ssh public/private keys (and methods) to sign and authenticate HTTP requests.
I've done a couple of HTTP auth schemes for my dayjob and am thinking of using something like this for my next.
1. https://github.com/joyent/node-http-signature/blob/master/ht...
It was an old API access key that got leaked, not our account credentials. We're still investigating how and where the key got leaked, but bottom line, it should have been revoked ages ago.
2FA is great, but it doesn't cover API keys. Rotate your API keys!
It's also possible to download and install ES locally and run any number of front-end interfaces, some of which include query builders. ElasticHQ seems like a decent option for that. The venerable Elasticsearch-head is another.
I think now that ES 1.0 has shipped, more experimental tools will start to emerge that help people learn and interact with ES itself. (If anyone out there is a front-end whiz and wants to help me build something like that, please email nz@bonsai.io!)
1. http://www.elasticsearch.org/guide/en/elasticsearch/referenc...
But I personally suspect Lucene won't ever get away from the dreaded "just reindex." And to the larger point, I think recent resurgent interest in data stores and distributed systems have shown pretty clearly that there is no holy grail. No single data store can provide all the semantics necessary for all use cases. Maybe not even for most use cases. There are just too many tradeoffs to consider.
Believe me, I earn a living hosting Elasticsearch, so I'd love to see it become a robust primary data store. There are some use cases where it actually does make sense—just look at the amazing traction ES is experiencing for storing and indexing time-series data.
But as a general-purpose primary store, I'm not really holding my breath. Maybe I'm just becoming battle-worn and bitter. I would love to be proven otherwise over the next few years!
The main thing for good stability and performance is to be very good at batching your updates. You don't want to sling a ton of highly-parallel single-document updates at Lucene, lest you thrash the JVM and start garbage collecting like crazy.
From there, on the query side, you'll want to get a good working knowledge of the different tokenization and analysis options. There are a lot of subtle and interesting combinations to be had in there that influence performance and relevance of your search results.
Having supported Solr/ES/Lucene in production for 4+ years now (websolr.com / bonsai.io) I would be pretty hesitant to trust Lucene in general as a primary data store. Beautiful for secondary indexing, but otherwise, Why Not Postgres?™ ;)
Same point, though. The superfluous apology lends an implied sense of influence and responsibility.
Is lack of diversity in our industry a problem? I share a strong hunch that it is. That's enough for me to want to talk about it.
So bring on the correlations, the symptoms, the graphs without solutions. Given enough eyeballs, all bugs are shallow.
The only down side I've felt is the loss of the Rails asset pipeline. This isn't so bad if you're just building some one-off static site. But when you're also working on a corresponding Rails app (e.g., Jekyll is for the static content, docs, and blog; Rails is all the user-facing interactive stuff), then the duplication of styles and layouts starts to hurt.
So I have been hacking on a bit of an experiment to run Jekyll alongside Rails. My goal is for Jekyll to generate everything in the public directory, and have direct access to the Rails asset pipeline. So far it's going a lot more easily than I expected.
If you're into Jekyll and Rails, take a look: https://github.com/nz/jekyll-rails-hybrid
There are a couple of open questions, and some improvements to asset tags in the works. But I don't see any reason why tightly integrating a Jekyll and Rails app should be more complex than a short readme or blog article. Pretty painless stuff.
I haven't read through all of Lunr's docs and source, but based on my Solr/Elasticsearch experience, I'd expect to see (in time)…
Tokenization and (presumably) term normalization/analysis; a faster and smarter query language, for term order independence and boolean combinations of clauses; relevance scores and maybe even score boosting per field.
Better queryability really shouldn't be understated here. Just having term order independence focused on a specific set of JSON is going to be way better than naively matching any substring on the entire rendered page.
If IronMQ was easier to hot-swap as a Resque back-end (or job processor like Sidekiq, if I understand it correctly) then I'd be more strongly tempted to jump over for some of its smarter queueing features that I haven't gotten around to rolling in to my own Resque setup.
There's something comforting about knowing that any one site's individual requirements are a small fraction of the total capacity of a system. I may be preaching to the choir here, but that's one of the things that I like most about using (and building) cloud services.
Something that fizx and I have observed while building websolr.com (our hosted Solr search service) is that as you scale the larger aggregate, the challenges of the average use case become increasingly trivial. For example, we're serving hundreds of millions of requests per day, and indexing billions of documents. That's a few orders of magnitude larger than any one of our customers, but they get all the benefits of a system that's designed to operate at that scale.
Put another way: how often do you really take the time to benchmark your message queue (or search engine, or whatever) to make sure it can scale to 10x or 100x growth over what you currently need? Or take the time to actually scale it to that capacity? Does that even make business sense? And yet, in a properly built cloud service, a 10x or 100x growth for any average customer is going to be a much smaller marginal increase against the aggregate.
Personally, I'd like to see similar comparisons for other search engines, like Sphinx and Postgres Full-Text. When I talk to people about search engines, the first questions they ask me are to compare one against some other.
That said, they carry a lot of annoying edge cases when determining the adjacency of quadrants, so they're hardly a panacea to geospatial search. Lucene 3 and 4 make a lot of progress in spatial search, but there's still a fair bit of room for improvement.