Can someone explain database scaling issues for web? | Hacker News Reader

Can someone explain database scaling issues for web?

3 pointszxcvvcxz3 comments

I don't really understand the technical challenges behind scaling, when people talk about using NoSQL or Redis.

I mean at the end of the day aren't you still trying to lookup (or insert or delete) into a very, very, very large list? Is one technique to "flatten" out your entire user database into a tree, say by location and then further subdivide so you just go down the tree to choose which smaller DB to access?

Those are my initial naive thoughts on the matter; could someone help me understand some of the non-trivialities to database scaling?

Thanks

View on HN

Nowadays instead of having the sub-systems you are referring to for storing data, many scaled databases use NoSQL and Redis for databasing and caching. The caching is probably the more scalable of the two because caching at any level has to do with usage and not all database data is used. Caching has allowed the most used data to be readily available. The limits of scalability may only require the system you described, but bigger and more complex systems like Twitter and Pinterest have been documented using database scaling via caching with things similar to Redis and Memcache. Keywords for finding more information are "cache/caching".

Problems with the sub-system of databasing are too many transactions (I/O) and the speed of getting the data and acting on it is very inefficient.

If you want to talk to me about it in more detail, message me on Twitter: @compooter or get my email at mayorpear.com

Here's a question: where is all this caching taking place? I understand caching in the context of processors but not so much what you're describing.

With MongoDB, there's a concept called Sharding, that divides a collection based on a property in the collection the developer selects.

http://www.mongodb.org/display/DOCS/Sharding+Introduction

Each shard can be either a single node, or can be divided among nodes in a Replica Set for distributed reads.

http://www.mongodb.org/display/DOCS/Replica+Sets

Reply on news.ycombinator.com