Show HN: NameMonk – fast domain search using web sockets and Bloom filters | Hacker News Reader

Show HN: NameMonk – fast domain search using web sockets and Bloom filters

namemonk.com

85 pointstectonic61 comments

So let me give you a hint.

I am/was the guy who made AjaxWhois back in the days (a similar site, one of the first). It was hugely popular and I was making thousands a month (totally passive) with it until it got acquired and went to shit.

I tested a bunch of things, ads in the sidebar, different registrars, etc, but the only thing that ever worked was GoDaddy, right in the search results. Ads in the sidebars are a waste of your time so don't even bother.

I know how technical people feel about GD (I feel the same) but the truth is, their crazy upselling techniques work and you get paid commission on those, not so much on the domain name itself. I was often making north of $100 per customer I referred to them. Not sure if they still pay the same but needless to say, that was pretty awesome

I also had coupon codes that gave my users a discount AND made me a lot of money. If you can get that, it's the holy grail. It just made no sense for my users not to click on the ad.

Good luck!

Thanks for the tips, cmer! I was always a big fan of AjaxWhois and am sad to see that it's gone. Why did you decide to sell it?

At the time I was working on my startup Defensio and just didn't have time to maintain AjaxWhois. Traffic was high, I was getting love letters weekly but I felt I was I wasn't worthy since I couldn't give it the attention it deserved. Sellin seemed like the right move at the time.

I wish I kept it since even less work was put into it post acquisiiton and the new owner eventually forgot to renew the domain name and it just died. Pretty sad ending for a site that so many people loved. Oh well!

Hey HN, I wanted to show you a site that I've been working on. It provides super fast domain lookup using WebSockets to a custom domain lookup server in EventMachine and a set of overlapping Bloom filters. Would love your feedback!

Nifty!

A few questions if you don't mind?

- What is the source for your list of names to generate the Bloom filters?

- How many names do you estimate are in the list(s)?

- How long does it take to generate the Bloom filters?

- I would assume your filters are based on registered domains, so when you get a positive match do you verify it with a Whois lookup to account for the false positives Bloom filters generate?

Also - any thoughts on open sourcing this? My primary concern with third-party domain search engines like this is them lying / stealing my domain ideas. Running a search like this locally would be very interesting.

Glad you like it!

The domain name list is provided in an 8GB dump from Verisign. You sign up to get access, but the process isn't too bad. There are around 100 million registered .com domains right now.

Building the bloom filter takes about 20 minutes.

It would be smart to double check with Whois data, but that would slow the process down. Right now I link to the registrars instead since they will do the Whois verification.

I'd definitely consider open sourcing or collaborating on the code. Send me a note.

Thanks for responding!

For anyone looking for that Verisign program, this page appears to have the info:

http://www.verisigninc.com/zone

It would be neat if someone could post it publicly. Sounds like a fun dataset to experiment with.

http://domai.nr/ has some cool features that you might want to incorporate.

http://i.imgur.com/d0oGNIe.png

It was the first domain that came to mind. :)

NameMonk was registered less than a day ago, so it's not in the root data dump yet.

Where does one grab such a root data dump?

It's pretty easy to register with Verisign to get access.

Is there an official name for that data dump?

Verisign

If you are using the .COM zone data (note this is different from the root data), then it is an incomplete list of all the TLDs that are registered. It is only a list of domains that have been delegated.

My favorite: http://instantdomainsearch.com

Includes the ability to save and encrypt searches, links to whois data, as well as alternative names based on similar spelling.

And then there's http://instantname.me/.

I realize I can't police the internet, but I just want to say that when someone works hard on something and wants to show it off, and your gut reaction is to show off something you didn't work on at all for karma points, it's the ultimate form of troll-dom in my opinion.

Your comment makes perfect sense if HN is a resource for submitters to get attention for their new projects. In that case, it's very frustrating and distracting to be shown alternatives. Trolling, as you say.

If HN is a resource for readers, then this sort of reply is immensely helpful, because it lets people who are interested in the project and/or its results explore other interesting, related projects. Quite the opposite of trolling.

Turns out HN is both.

I guess I can't argue with that.

Wait, what?

Why do you suppose GP is "showing off for karma points"? I found this link very relevant, and in general showing similar/related solutions is very informative for both the author and people who read comment threads.

You can't have a proper conversation about a tool if you pretend competition doesn't exist. Consider:

* This is a public forum and this is on-topic as most participants are interested in the broader category than just one tool.

* It can help OP by making them aware of products and use cases they had overlooked.

* It gives OP a chance to compare their work with alternatives.

Also, http://domai.nr/ is a good one.

Very fast, but to be perfectly honest, still a bit useless for actually finding a business domain name without support for other TLD's.

My base point of comparison is domai.nr which does a pretty speedy job across numerous TLDs.

Yeah, I'm not sure speed was the metric I needed improved when it comes to domain search, though it's a nice demo project.

I've been playing with other search methods. Once you have all domains in memory, it's easier to do things like lookup every English word or pronounceable combinations of letters very quickly.

I also highly recommend http://bustaname.com

Try http://domaintyper.com Very fast , multiple TLDs and gives you suggestions for names based on partial input.

If you could add more TLDs, you could win me away from https://domize.com/

In my experience building http://instantdomainsearch.com/ I found that the "slow" part was actually the network round-trip to the server. WebSockets don't make packets move across the network any faster. We came up with a way to select the closest/fastest server when you first visit the site:

http://instantdomainsearch.com/articles/faster_domain_name_s...

This works well, but I think I'll switch to Amazon's Route 53 (DNS) implementation now that it can do failover.

How about some better registrars than GoDaddy as the default list? Hover.com, gandi.net are both much much much better registrars, have much better support, aren't scummy, etc.

It's great how fast this is. Suggestion: make the underlined text "not available" be a link to whois information when a domain is taken. The way you have it now, it looks clickable, and that's what I expected it to tell me.

Or, come to think of it, don't do that. Just display the whois info automatically when a domain is taken while keeping everything as fast as it is now. That would be awesome!

Thanks for the suggestion! It's hard to do Whois as quickly, but I'll think on it.

For the humor version with the same backend: http://findafuckingdomain.com

Nice idea, I could really use this sometimes.

However, when I search for namemonk.com it tells me that it's totally available. How reliable is this?

His FAQ page says the data is refreshed every 24 hours.

  > whois namemonk.com
  ...
  Creation date: 13 Jul 2013 03:26:00
  Expiration date: 13 Jul 2014 03:26:00

It's pretty accurate, with about a 24 hour delay. Namemonk was registered yesterday and hasn't been updated in the root data dump from Verisign yet.

Also says domicilio.com is available. I think it's using simple dns lookups, if you made a whois call to the domain you will see that the domain is registered but is not in use.

I wrote a real dirty domain name scanner for this, too bad the results have been incorrect so far. https://gist.github.com/jackdwyer/5994001

/edit: note it is just getting all 4 letter combinations at the moment.

Cool! It seems like it assumes the TLD is .com for now - do you plan on expanding it? Maybe do input.<tld> for .com, .net, .org, .io, etc?

Also it seems like your data source is out of date - for example, your site tells me that namemonk.com is totally available

Thanks. Yea, I'm probably going to follow up with other TLDs in the future.

Also I just view-sourced... holy hell, what are you doing with 130KB of minified JavaScript? I'm guessing its a huge blob of libraries globbed together?

Also your session cookie's name was amusing.

That's what you get when you include jQuery UI for no reason... good catch, thanks!

Also, jffry appears to have figured out the name of the other version of this site. http://findafuckingdomain.com is a humor version using the same backend. ;)

Reminds me of http://www.whatthefuckshouldimakefordinner.com/

You should incorporate colors (green avail, red not avail) so we can grasp quickly.

I'm a fan of http://namevine.com/, it includes twitter/facebook/etc registrations.

http://domfinder.com is agonisingly slow in comparison but does give you some smart suggestions

Very nice. If you could add support for other TLDs, I would definitely use this and recommend to my friends.

Awesome work! How did you use Bloom filters to achieve low latency lookup?

I periodically download the 8GB com.zone root name file from Verisign and compress it into a set Bloom filters that allow for very fast lookup while using less memory than a textual hash. For fast responses, the filters are served over a UNIX domain socket to the WebSocket server on the backend.

That's a great approach I've thought of before -- so how big does the bloomified version end up being?

About 1GB to get very low error rates.

Would you mind sharing a bit about how the process to gain access to the root zone file works? I've seen the application form, but I'm curious how the whole thing goes from application to access. Any insights you can share?

I've been kicking around a few ideas, but didn't want to go through an involved process if it would only end in heartbreak. Does it require a specific use case as part of the application?

The process wasn't bad when I did it, actually a year or two ago. It took me this long to get around to doing anything with the data. :)

If you'd like to collaborate, I'm totally game. Send me a note.

does it work for top level domains other than .com? I own a .in domain and the search did not yield a correct result.

Same speed as instantdomainsearch.com, but that service does not track what you search.

I don't track what you search beyond looking it up in the Bloom filter. How could any site not transmit the domain for lookup? Frankly, I don't understand this comment.

Probably the fact you're using google, facebook, and twitter. All 3 will store some information... Nice name for your session cookie btw :)

Cool! I really would like to see how this was accomplished, wanna open-source it?

Oh No namemonk.com is totally available

Reply on news.ycombinator.com