Comment by liamca | Hacker News Reader

Comment by liamca | Hacker News Reader

Parent Full thread

Hi MichaelGG,

I am a Program Manager for Azure Search and as curiousDog mentioned, yes along with DocumentDB, we also announced Azure Search which is a PaaS based full text search service. We actually leverage ElasticSearch at the core of this service and as chippy says about spatial search, we do have the ability to provide a pretty solid geo-spatial search capability thanks to Elastic Search and Lucene. To nemothekids's point, is it very unlikely that we will offer this as a local (non-hosted) version because we found that although ElasticSearch is awesome, one of the common complaints many admin's have is the complexity around managing systems such as ElasticSearch/SOLR/Lucene at high scale and how difficult it is to implement more advanced search capabilities such as tuning and relevancy. Those are areas we think we can add a lot of value being a fully managed service. Longer term, we think this will also allow us to bring even more value on top of search by adding in other Microsoft technologies. For example, we could tie in Bing Maps to allow you to easily tie in reverse geo-coding right into your search. Or perhaps allow you to leverage Bing's synonym list so that you could allow people to search yet find results that are synonyms to commonly searched words (i.e., user types in shoes, but in your content it is referred to as footwear). Multi-language support is actually one of the big things we want to tackle in the short term and we believe that the NLP from Office will really help jump start us with this.

Liam

We don't need the actual infrastructure or even the same software locally.. what's wanted/desired is an API compatible system that can be used to develop against. It also doesn't need to run on a local cluster for development purposes, or smaller shops.

I know everyone is all about running huge clusters behind the scenes, but most people simply don't need that, and being able to start small, and buy bigger would be a nice option.

For example, a simple node application using leveldb(levelup) for the storage interface could be very effective as a backend for development/testing. From there, you provide an API compatible interface. Open-source that version, with the disclosure/understanding that the Azure hosted version is much more robust.

I think you'd see a lot more buy in from the open-source community... even more so if you accepted PRs to make the open version more robust.

If you are at a point where you are considering the likes of ES/SOLR/Lucene etc, you are likely ready to make the jump to self-host in the cloud, or use a SaaS provider. Where people get a bit concerned is in the lock in. I know why Azure would want to present that, but I think it's a bad idea without an open implementation that allows for self-hosting for development and on the small scale.

Right now, the company I work for is hosting in Azure. I recently switched from a couple RabbitMQ queues to Azure Queues. It works fine, as was a really simple replacement for a flow of near real-time but temporary data keys. I would be much more open to using a hosted MongoDB from MS than DocumentDB, in much the same way I'm happy to see you guys embracing Redis for a cache system.

tracker1,

I hear you on this and to add another justification for the need for openness, let me give you another example. With ElasticSearch there are some amazing tools that are available. A few that come to mind are Kibana and LogStash. Given that we have our own API layer on top of ElasticSearch, we are not able to support these tools even though we are using ElasticSearch at our core. This is most unfortunate.

There are many reasons why we put an API layer on top of Azure Search. One key reason is that for our particular customers, we found that there were things we could do in our API that could simplify the interaction with Search. In fact, we have a system called Scoring Profiles that allow you to easily (I hope) set weights on important fields and attributes to quickly tune the results of your search based on what is important to you. No coding required. Another reason we don't just expose ElasticSearch is that ElasticSearch allows you to run random code. This is generally not a great idea in a PaaS service and can often lead to issues. There are a number of other reason that I'll skip for now.

We still need to do some thinking in this area. I hope we can get Azure Search to a point that we can enable tools such as Kibana and LogStash to work with our service without compromising the goals of what we are trying to build. Not only would this allow us to really open up the types of things people can do with the service, but I suspect it would really help reduce concern around vendor lock in. We'll see if we can get there...

Liam

Reply on news.ycombinator.com