Our setup was surely not ideal, but even with significant tuning, it was not sufficient.
It is really good to hear successes like Reddit's, though. How long have you folks had your databases in EC2?
6,111 karma · joined July 21, 2008
Our setup was surely not ideal, but even with significant tuning, it was not sufficient.
It is really good to hear successes like Reddit's, though. How long have you folks had your databases in EC2?
We explored all the options available and worked with both Percona and Amazon on the issues, but it was clear that EBS and EC2 just are not meant to handle such load. Our database is currently back running in our datacenter.
There have been a number of reports with similar results using EBS. It all depends on your particular IO profile, but since you said it was high, I encourage you to tread lightly.
If you want more information you are welcome to email me.
You cannot really go wrong with python or ruby. Both have a number of solid frameworks for you to try (rails, django, pylons, sinatra, etc). Try a number of them and build some trivial app in a few of them.
I also recommend getting familiar with javascript and a library like jquery or mootools, if you plan do dive deeper in the web dev.
Otherwise, nice clean design.
Lockheed and the rest of the netops team gave a spectacular talk this year on how the network is designed. You can read about it here: http://www.defconnetworking.org/index.php?itemid=33
Anyway I think 10gb per domain is reasonable for many people, but it is another limit of the store that could cause problems. Fortunately Amazon knows the boundaries of SDB, so it can put these in place and help inform decisions about using it for, say, a project that will need more than 10gb per domain, or more than 100 domains.
While 3. is a pain-point in many systems, the fact that you can do selects with where clauses on a key-value store that scales horizontally is impressive. The indexing they do can be very helpful in certain implementations.
It seems like you are referring to using it as a cache only, and on those terms, it really depends on your traffic profile. Chances are, though, with 200k uniques you would be pleased. If you are running in EC2, the bandwidth is free, so you only pay for computation. That can keep things pretty inexpensive.
I don't believe that cost was an issue for them with EY. It was about their system's needs regarding server resources.
This will almost surely be damaging to your career and education. If this means that you are too easily influenced or cannot identify a good influence, then I would focus on improving that in yourself.
Surrounding yourself with intelligent people in a field such as this is incredibly important. You are missing out otherwise.
Be sure to check your filters as a common attack is simply copying all mail to another account.
I think there is a lot of truth to that, but I also have seen a big shift since the iPhone. Countless friends and colleagues of mine have said something along the lines of "I would never use AT&T if it were not for the iPhone."
It seems like something that could happen, but making it a community (as in HN) effort seems like it has too many issues.
We are a top 10 Rails site and get to deal with intriguing issues every day. Solving massive scaling challenges while pairing with people like Ward Cunningham (our CTO) is how we work - it could not be any more fun and engaging.
Check out our jobs page for the full run-down: http://www.aboutus.org/AboutUs.org/Jobs/Developer
Sometimes, but oftentimes sharding issues are unique enough that they require custom solutions when the backend technology does not support it natively.
A few months down the line your load gets so high that you need to add a third server, but your sharding mechanism is based on mod 2 - so you need to hack around this in one of many ways to utilize the third server effectively.
Make sense?