Scaling Early
slideshare.net
slideshare.net
Also, what is the URL of the project/site/start-up you're scaling. I am interested very much in seeing the seeds of this effort.
The content in the presentation does make sense for what feedjit does.
A better way would be (since Tokyo Cabinet is thread safe) having a separate thread/process which receives messages (using a message queuing system) that update the database files (ala MySQL replications).
(Or you can separate out the db layer as a separate network service and partition the writes/reads to go to a specific several servers based on the key. Tokyo Cabinet even includes a memcached API compatible network layer for this - the Tokyo Tyrant).
Hate to be a nitpicker but it's one of my pet peeves:
What the slides describe isn't scaling but performance tweaking. Scalability != performance.
A lot of people get that wrong, still annoys me, silly me... :D
- The ability to handle lots of load at all, which comes down to a stability and fault-tolerance issue.
- The performance of the system under load.
Both are pretty valid.
See the Wikipedia article: http://en.wikipedia.org/wiki/Scalability
If a system's performance significantly degrades under load, it doesn't scale.
Quoting the wikipedia article: 'its ability to either handle growing amounts of work in a graceful manner, or to be readily enlarged.'
The first clause is performing well under load, hence the performance requirement.