karterk

Recently a friend of mine visited the emergency care in a private hospital in downtown SF with heavy stomach pain. It turned out to be a partial bowel obstruction, which thankfully sorted itself out with some conservative treatment.

My friend was discharged in a couple of days... with a bill of 70K USD. Except for an IV and a Ryles tube, there was hardly any other intervention.

Having just finished college, my friend did not have an insurance plan that could cover such a huge bill. Thankfully the state stepped in to help since the yearly income was below the federal poverty line. Otherwise, I can't imagine the plight.

For once, my friend was thankful for being a broke college grad.

The goal is to be much better than Algolia and Typesense's design is in some ways already so:

1. Sorting order is dynamic, unlike Algolia where you are forced to define a sort order during index creation and you are forced to duplicate your entire index when you want another sort order, e.g. sort by timestamp vs sort by ratings.

2. Run fast on commodity hardware (e.g. a decent EC2 instance) instead of requiring special dedicated hardware that costs a bomb. This is achieved by using highly optimized in-memory data structures instead of relying on fast SSDs and memory mapped I/O approach that Algolia has taken.

3. Disprove the notion that one cannot host their search themselves and need to rely on a cloud service provider. We want to make it really easy to run Typesense and manage it. We have some things to do on this front, but this is definitely the goal.

Working on an open source, fast, fuzzy search engine that's dead simple to set-up and run: https://github.com/typesense/typesense

People who want a good search engine (for instant search etc.) today either need to go for a cloud hosted proprietary offering or have to learn to set-up Elasticsearch and spend a lot of time tweaking it to support typo tolerance. Trying to change that status quo.

I might not be 100% correct on this, but it seems like even Algolia keeps all records in a single host and uses the other 2 hosts for only high availability.

Having said that, exploring a FoundationDB integration is definitely an interesting idea. However, quite a lot of use cases can be served perfectly well with a simple master+slave set-up, so my primary focus is on that until there is enough demand for horizontal scalability. For e.g. Typesense is not a great fit for things like log data that typically need large amounts of storage.

Shameless plug: if you are looking for a simple, fast, fuzzy search engine that you want to host yourself, take a look at Typesense: https://github.com/typesense/typesense

That's because HN search is hosted by Algolia who encourage searching directly from JS (client side) since it's faster (one less hop to an intermediate HN server).

This has been an area of interest for me as part of working on Typesense[1]. If you are looking to implement spelling correction, you cannot but not stumble on this excellent post by Peter Norvig (most of it written on a bored flight journey!).

While it's clever and concise, in terms of raw speed, indexing your vocabulary in a Trie, and then doing a traversal on it using a Levenshtein distance is way faster[2]. By using a trie, you eliminate lot of unwanted look-ups that Peter Norvig's brute-force approach takes.

The other interesting thing for me personally about this problem is that there is a statistical angle as well. You will often find words that are of the same edit distance from a given target word -- you will need to rely on some form of "popularity metric" to rank those tied corrections. E.g. how many times did people type this word and then change it to another word - that's precisely the kind of data that Google has and which makes its search suggestions and spell checking so intuitive.

[1]: https://typesense.org/ [2]: http://stevehanov.ca/blog/index.php?id=114

You can use a combination of open source projects. For example, you can use an OPML parser library like Listparser[0] to get the contents of the pages.

Then, you can index them into a search engine of sorts. You can use Elasticsearch[1] for that, or if you want to try something that's much simpler that I have been working on, try Typesense[2].

Finally, you can use Flask[3] to create a simple website/frontend for integrating with the search engine.

[0]: https://github.com/kurtmckee/listparser [1]: https://www.elastic.co/ [2]: https://typesense.org/ [3]: http://flask.pocoo.org/

Hi HN,

We wanted a simple search engine that was fast, handled typographic errors out-of-the-box and was also simple to run and manage. We could not find one - so we built Typesense and are open sourcing it today. Before you ask: Elasticsearch is a great project but is far better suited for larger teams which have bandwidth to administer, scale and fine-tune it and when you want to store billions of documents. For those of us who are not in that bucket, we hope Typesense would help.

Happy to answer any questions!

Mind if I reached out to you to chat about your experience in running search at scale? How do I contact you?

> Although we were using Elasticsearch in the beginning to power our in-app search, we moved this part of our processing over to Algolia a couple of months ago;

How many records are you storing on Algolia?

An update from Elon on the pricing[1]:

> Fly to most places on Earth in under 30 mins and anywhere in under 60. Cost per seat should be about the same as full fare economy in an aircraft. Forgot to mention that.

[1]: https://www.instagram.com/p/BZnVfWxgdLe/

I'm really happy that the author found a place that he could truly call home. However, his generalisation of India largely based on a single incident comes across as rather shallow and naive. Yeah sure India has many quirks, but I can't find another country which has people from so many diverse backgrounds (religion, language, customs) who co-exist so peacefully. Let's also not forget that the legalisation of same-sex marriage is a recent development even in the USA.

I think internally, Google has already seen this coming and are trying to steer the company towards AI-related verticals. Search revenue is going to be always under threat because of the rise in mobile devices and also ad blockers. Interestingly, installing ad blocking is harder in the mobile though (requires a separate browser), than in the desktop browser (just another extension).

Not necessarily true. If the task is too easy then you will get bored fairly soon.

> I haven't even written a line of code in months.

CTOs generally don't write code. But, in a typical startup designations don't matter so much. How different is your current role from the one you were offered during the negotiations?

> I thought perhaps being CTO in a startup would give me some of that same ownership, control and enjoyment but it just feels like another job.

Do you feel connected to the overall vision of the company? If you did, then you should look at the things you're unhappy with as bumps on a road worth traveling.

Since you do have the ownership (through your designation), you have power to change things that bother you. It's only when you start making visible impact, you will start truly belonging.

Finally, I would urge you to talk to other senior folks in the company openly about what's bothering you. Maybe you are missing another perspective that they will be happy to share with you.

Ah, the armchair critic. I'm by no means a Microsoft fanboy, but things have been much better under Satya Nadella.

Searching google for "producthunt mac voice control", the desired result came at number 4. Not bad.

I'm by no means an expert at machine learning, but, given how organized the scikit learn libraries are, building a simple classifier as show in the example would be a few days of work at the max. In fact, an initial first version can be built within a day. After that, one has to tune the hyper parameters and spend time with feature selection to improve the baseline accuracy.

The most important thing will be the training data. You need a good number of samples, and the data also needs to be reasonably "clean".

1. People don't store such large amounts of data in a conventional database (especially relational DB). Depending on the exact requirements, the popular choices are HBase, Cassadra or even flat files in HDFS.

2. To analyze the data, Hadoop and Spark are great. Spark is especially a great fit for iterative algorithms, and it also has a great python interface.

Maybe this is the norm in big enterprises, but I have not actually come across any company which hosts a local package repository for commonly available packages.

Well done I guess :) I worked briefly at ThoughtWorks, but definitely don't have any vested interests. ThoughtWorks produces not-so-great products too (like Mingle), but Go CI is definitely a great piece of software. And, I say that as someone who also has used Jenkins a lot.

Also, it's now completely open source. I'm not even trying to advocate an enterprise product.

I have worked with many open and proprietary CI tools, and find GoCD to be the best in this space right now. It supports both Continuous Integration and Continuous Delivery. They also made it open source last year:

https://www.go.cd/

Also, see the follow-up post:

http://blog.scalyr.com/2014/07/impossible-problems-often-are...

In the end, they did have to do some pre-computes (which is better than indexing, but still not 100% on-the-fly resolution).

On the other hand, as your system gets more complex, you will start missing tried and test libraries from some of the more mature languages you have mentioned. Maybe the extra time you have to spend writing that code will compensate for the extra memory you would have spent because of the JVM. Not taking anything away from Rust - I'm just giving an alternative perspective.

You can't reverse the trend if you don't know why exactly it's happening.

First talk to those who are cancelling to find out why. For e.g. add a mandatory field on the cancellation form that asks them to describe why they are quitting you service.

Something like this?

http://arxiv.org/abs/1603.09382v1

Here's another way to look at it: Facebook, Google & Twitter sell their software to advertisers that allows them to reach their target audience. Here "selling" is actually a service.

Uber is mostly a software company - the cars are owned by the drivers. When you pay for a Uber ride, you're infact paying a part of it to the software which helped you find the ride.

Apart from that, there are huge swaths of enterprise software companies in every industry/domain you will probably have never heard of. Some of them are even billion dollar companies.

1) Once the training is done, the prediction part is simply matrix multiplications - I would think that's probably faster and far easier to parallelize than any other advance feature extraction algorithm you will be performing real time on the image.

2) Did you try using CNN?

I might be grossly wrong here, but I have gone through this path before, but I have come to the conclusion that clumsy hand-crafted feature engineering on images usually performs worse than letting the algorithm figure out. It's much harder to scale and you will end-up coming across a lot of edge cases.

What task are you learning the features for? If it's classification, instead of treating it as a two step (feature generation followed by classification) problem, I suggest that you just use a multi-layer neural network with a sigmoid function at the final layer so that you directly predict the output class from raw pixels. This way the feature engineering is taken care of by the algorithm (the weights of the hidden layers).

To give an idea of what I mean, see: http://karpathy.github.io/2015/10/25/selfie/