HNHacker News
TopNewBestAskShowJobs

qhoxie

6,111 karma · joined July 21, 2008

Co-Founder @ Fixture (YC W26). Previously Tech Lead for Elasticsearch, Founder Swiftype (acquired by Elastic, YC W12)
submissionscomments
qhoxie··on Ask HN: Anyone used Amazon EC2 with a database or other high IO operation?
This really would not be all that surprising, but it also would not surprise me if the average Postgres setup was better tuned/architected than the majority case of MySQL uses.

Our setup was surely not ideal, but even with significant tuning, it was not sufficient.

It is really good to hear successes like Reddit's, though. How long have you folks had your databases in EC2?

qhoxie··on Ask HN: Anyone used Amazon EC2 with a database or other high IO operation?
This is a good clarification to make. It was not that our entire application ground to a halt when we flipped the switch. We would experience intermittent and extreme performance issues. We dug deep with tools like cacti and ended up being able to point only at the 'unknowns' of the shared resources and EBS.
qhoxie··on Ask HN: Anyone used Amazon EC2 with a database or other high IO operation?
We have a highly trafficked and large dataset that we aimed to move from our datacenter to EC2. This is all in MySQL. Upon doing so, we had to spend significant effort trying to get our system stable and we ended up narrowing it down to EBS simply not being able to handle the IOPS we needed.

We explored all the options available and worked with both Percona and Amazon on the issues, but it was clear that EBS and EC2 just are not meant to handle such load. Our database is currently back running in our datacenter.

There have been a number of reports with similar results using EBS. It all depends on your particular IO profile, but since you said it was high, I encourage you to tread lightly.

If you want more information you are welcome to email me.

qhoxie··on Ask HN: Best Web Language to Learn
This has been discussed fairly exhaustively, but here is a quick summary:

You cannot really go wrong with python or ruby. Both have a number of solid frameworks for you to try (rails, django, pylons, sinatra, etc). Try a number of them and build some trivial app in a few of them.

I also recommend getting familiar with javascript and a library like jquery or mootools, if you plan do dive deeper in the web dev.

qhoxie··on Ask HN: Where to go in Portland
I've lived here in Portland for about 9 months, and I can safely say this is an excellent list. It should keep you busy for a while.
qhoxie··on Ask HN: Review my Startup, Thumbtack
A quick UX note: I think it would be a good idea to use a more appropriate cursor style on the mileage slider.

Otherwise, nice clean design.

qhoxie··on Wifi at conferences problem
Defcon sets a good example is this arena. It is said to be the most hostile network there is, and for years the wifi was effectively unusable. The last couple of years, though, it has been nearly flawless. I know it has evolved a lot. For a while I believe that dragorn (http://www.nerv-un.net/~dragorn/) helped run it.

Lockheed and the rest of the netops team gave a spectacular talk this year on how the network is designed. You can read about it here: http://www.defconnetworking.org/index.php?itemid=33

qhoxie··on Ask HN: Favorites on HN?
I understand what you're saying, but it seems like it's just worth the upvote to save it.
qhoxie··on Ask HN: How do you pick colors
I've used http://www.colourlovers.com/ for years. Great site.
qhoxie··on Don't Forget: You Can Use Amazon SimpleDB For Free
AWS products are pay as you go, so I'm not sure what you mean by that. This will be free until you cross the limit.
qhoxie··on Don't Forget: You Can Use Amazon SimpleDB For Free
True, it's no Cassandra :)

Anyway I think 10gb per domain is reasonable for many people, but it is another limit of the store that could cause problems. Fortunately Amazon knows the boundaries of SDB, so it can put these in place and help inform decisions about using it for, say, a project that will need more than 10gb per domain, or more than 100 domains.

qhoxie··on Don't Forget: You Can Use Amazon SimpleDB For Free
What you said is very true, and a good summary of shortcomings, but I'd like to note a real benefit it affords you despite these drawbacks.

While 3. is a pain-point in many systems, the fact that you can do selects with where clauses on a key-value store that scales horizontally is impressive. The indexing they do can be very helpful in certain implementations.

qhoxie··on Don't Forget: You Can Use Amazon SimpleDB For Free
There are actually some backup services that leverage S3 and require customers to have their own accounts. It seems to work quite well as a model.
qhoxie··on Don't Forget: You Can Use Amazon SimpleDB For Free
The only way to achieve this would be to shard the file across multiple entries. The limit per is 1k, so I doubt it would fit your needs. There is also a hard limit of 1gb per domain, which is not difficult to run up against if you treat it as a more generic data store.
qhoxie··on Don't Forget: You Can Use Amazon SimpleDB For Free
If you don't throw too much traffic at it you will be fine using it as a memcached alternative, but you will probably notice the overhead of an HTTP transaction. It largely depends on what you are doing.

It seems like you are referring to using it as a cache only, and on those terms, it really depends on your traffic profile. Chances are, though, with 200k uniques you would be pleased. If you are running in EC2, the bandwidth is free, so you only pay for computation. That can keep things pretty inexpensive.

qhoxie··on GitHub's Architecture, before and after migration
The move to Rackspace will bring about a new backend architecture and a lot more servers, leading to a much improved user experience for everyone. - TPW from Github

I don't believe that cost was an issue for them with EY. It was about their system's needs regarding server resources.

qhoxie··on Why aren't tabs a window manager feature?
There are some tabbed window managers out there, and many swear by them. Check out fluxbox - http://fluxbox.org
qhoxie··on Masters in CS?
The answer to this question almost completely hinges on what you want for yourself. What are your career goals? What is your current education? Do you feel any educational deficiency or are you comfortable with your status? Do you have the financial means currently?
qhoxie··on Ask HN: How much?
I don't interact with other developers because I worry that it will pull me toward the middle of the bell curve

This will almost surely be damaging to your career and education. If this means that you are too easily influenced or cannot identify a good influence, then I would focus on improving that in yourself.

Surrounding yourself with intelligent people in a field such as this is incredibly important. You are missing out otherwise.

qhoxie··on Ask HN: My accounts: Twitter hacked? GMail hacked? Computer hacked?
After a bit of searching around I verified that there had been no activity in my GMail account since I left on Friday, so they couldn't have accessed the confirmation email to verify the Twitter account

Be sure to check your filters as a common attack is simply copying all mail to another account.

qhoxie··on The Android Opportunity
Similar to the sibling, I see 5-10% as being a fairly solid impact.
qhoxie··on The Android Opportunity
Most people, even high-end smart phone users, choose their carrier first and their phone second.

I think there is a lot of truth to that, but I also have seen a big shift since the iPhone. Countless friends and colleagues of mine have said something along the lines of "I would never use AT&T if it were not for the iPhone."

qhoxie··on Ask HN: Should we as community create all-star groups for RFS?
How would that work? Who would make decisions about building the groups? Would 'good' people just be placed together - that doesn't make a good team.

It seems like something that could happen, but making it a community (as in HN) effort seems like it has too many issues.

qhoxie··on Ask HN: Who's Hiring? (take 2)
To start, we are hiring at AboutUs in Portland, OR. We need more Rails hackers to help us scale outward and develop new aspects of our application.

We are a top 10 Rails site and get to deal with intriguing issues every day. Solving massive scaling challenges while pairing with people like Ward Cunningham (our CTO) is how we work - it could not be any more fun and engaging.

Check out our jobs page for the full run-down: http://www.aboutus.org/AboutUs.org/Jobs/Developer

qhoxie··on Ask HN: Where do you live?
Portland, Oregon
qhoxie··on NoSQL: If only it was that easy
That is correct. Things like S3 can expand without issue, and others like Voldemort handle partitioning data natively.
qhoxie··on NoSQL: If only it was that easy
So don't people just publish client libs that handle the details for you?

Sometimes, but oftentimes sharding issues are unique enough that they require custom solutions when the backend technology does not support it natively.

qhoxie··on NoSQL: If only it was that easy
One such bad situation would be if you have numeric keys that you shard based on a mod 2. You do this because you want to distribute data between 2 servers, so one will get all even keys and the other will get the odd keys.

A few months down the line your load gets so high that you need to add a third server, but your sharding mechanism is based on mod 2 - so you need to hack around this in one of many ways to utilize the third server effectively.

Make sense?

qhoxie··on NoSQL: If only it was that easy
He's pointing out a situation where it becomes a bigger deal, to answer your question. While there are solutions in practice for this situation, it can be painful.
qhoxie··on NoSQL: If only it was that easy
Hopefully you don't always stop reading thoughtful pieces for trivial reasons.
← PreviousPage 5 of 19Next →