951 karma · joined June 3, 2009
And the only thing I use for this is Shell scripts. In fact I am not sure if there is any other option. But then I am not a system administrator, its my part time duty.
You seem to be doing something different than most of the people here. No wonder you feel like yelling as well.
1. Memory: We have 2 GB and "free -m" shows that more than 200MB (least) is free at any given time. Do I still need to add more memory?
2. Our site scores perfect A on Yslow. So minification , gzip, far future headers etc is in place.
3. We have improved out DB structure a lot. Since we are adding more features continuously we are constantly changing the DB structure. With the help of some more experienced programmers I will get these things tweaked. But in MySQL okay if our user base will hit 1 Lakh in next three months ?
4. CPU profiling is something that we haven't done yet. Will do this asap. Can you suggest some resources starting points ?
You also said something about load balancers. Will you please give me some resources to start up with ( I know I can google but ...) ?
5. Fail-over is something that I did not think of initially. What are the best tools around ?for the same ?
The whole discussion might sound as if I want you guys to spoon feed me. But that is not the case. Let us look at the whole questions as an opportunity to share each others views on the same topic, even the experts might get to learn new things.
We have certainly considered all the factors you have mentioned while implementing our crawler however unlike typical crawlers the objectives of our crawler is purely a research goal and completely different than what most have done.
Keeping costs lower by be a great technological achievement but will that device be helpful is a question to be asked.
But as a blogger I really find your criticism very helpful. Thanks a lot.