MySQL and Memcached: End of an Era?
highscalability.com
highscalability.com
The party is just getting started for "Hey, check it: we can put a cache in front of our database" for that broad middle of the curve. (This includes applications where relational databases are a great fit but for certain performance requirements in small bits of the program. For example, you really don't want to have to implement a university backend system in a key/value store. Just trust me on this one. It is virtually made for SQL, but while 99% of the use of the system is less than performance intensive, having resources in place to support that last 1% costs the university hundreds of thousands of dollars. I think we could slash that with creative caching.)
"End of an era" signals a major shift across the market, not that the tiny bleeding edge has moved on to something different.
Sure, PostgreSQL may have better implementations of certain features than MySQL, but for most web developers, mysql is good enough.
How would put PostgreSQL against non-relational dbs?
It would also have the benefit of better tools and the option to use a JOIN or a transaction where it made sense. Of course, you could do the same with MySql, too.
It is a bit maddening to see people applying the NoSQL hammer to problems that don't need scalability, but it's partly driven by the realization that they open up opportunities to build entirely new applications.
Personally I don't trust NoSql solutions because they aren't mature enough. I've had little problems using MySql/Postgres and never had loss of data ... and I can always come up with a non-relational scheme latter built on top ... when I really need it.
The other problem I've had with Nosql is that capabilities vary from implementation to implementation, and my software really evolves a lot ... I don't know what my needs are from the start.
I've also used MySql instead of Memcached successfully for a website with tens of millions of hits per day, preferring it because Memcached crashes on me a lot and there are big differences between versions I don't want to worry about ... and yeah, I know async reads are problematic with Mysql/Postgres, but I just threw a web service in front of it, and when using the Memory storage engine it really kicks ass.
Many of the algorithms for recommender systems are especially known for being parallelizable and would be extremely difficult to reliably and quickly implement on top of MySQL (or Postgres or Oracle). I can't even think of a way to do so without making a fairly complex ETL cycle with a separate DWH and a main serving DB (with points at which the site becomes degraded and possibly unavailable). As I've mentioned earlier, with a system like Hadoop, you could start off by implementing these algorithms in Perl or Python (making use extensive libraries already available in the language).
Recommendations may not be real time, but there's a huge time difference between doing an even fairly simple "market basket"-type analysis for a data set orders of magnitude difference smaller than Amazon's in MySQL and doing this on top of Hadoop + HBase/Voldemort/Cassandra (i.e., compute in Hadoop, serve from a NoSQL db which offers Hadoop integration). You could even start by doing the computation in Hadoop and loading the data into MySQL (while, perhaps, serving the data from a slave db).
> Maybe it is for Amazon which got plenty of data to analyze, but not for a startup.
Your start-up doesn't need to have tons of users to have tons of data. This is especially true if you're building a start-up to make sense of the data that's already out there (some YC examples are Flightcaster, Directed Edge). For a consumer start-up, ability to have cheap and scalable storage means you can record data much more frequently.
> The other problem I've had with Nosql is that capabilities vary from implementation to implementation, and my software really evolves a lot ... I don't know what my needs are from the start.
Isn't this the point of separating your data model from your presentation layer? Begin with MySQL (or even simpler, with BerkeleyDB) -- change it to whatever else suits your needs.
Now even if you are okay with low throughput (you don't see yourself become a major Internet destination), you can still benefit from non-relational approaches.
Implementing a recommender system or PageRank on top of a relational database is simply painful and requires expertise in that system's SQL dialect (someone who's done this on top of Oracle won't be able to do this -- in the same amount of time -- on top of MySQL or Postgres). With Hadoop you can start with a Perl or Python script (and make use of libraries like scipy or pysvm) and then gradually re-write the performance critical parts in Java. Even if you aren't and will never get to Facebook's or Amazon's scale, you're now able to build intelligent applications on top of your data.
You can also sleep sound at night (i.e., not in front of a terminal tailing your logs or watching ganglia) knowing that a load spike (which could be start of a new financial quarter, holiday shopping season; something as positive as a NYTimes article or -- much more likely -- as negative as a DDOS attack) won't bring you to an outage.
The algorithms themselves, however, are very difficult and require a background in machine learning; that is even to use an existing implementation of these algorithms (e.g. Apache Mahout). I am not an expert in these topics. I took a graduate course in machine learning, but that only demonstrated to me how little I know and how much more I have left to learn. It's this expertise that coupled with ability to implement a scalable infrastructure with open source/in-house developed software (instead of relying on commercial vendors) that makes your a business idea (you know, the one that the "business guys" think they can hire bunch of MSCEs to build in ASP.NET/J2EE/PHP/RoR) defensible from some "kid" in Harvard dorms (especially if that kid declined a $1mm starting bonus to work for Microsoft instead of going to Harvard).
Don't get me wrong. SQL is an amazingly example of a domain specific language; RDBMs are wonderful works of software engineering. SQL RDBMs are great for many applications (e.g., ERP systems and their functional equivalents like the university registration system patio11 mentioned). Yet they're not made to solve the kinds of problems/create the kind of applications that simply didn't exist when they were created. The fact that they can at times be used for this doesn't mean they should: I can write Russian with Latin instead of Cyrillic Alphabet (or write Lisp in vi instead of emacs), but it's a lot better to use the correct tool.
"Since it was already necessary to abandon data normalization and consistency to make these approaches work, we felt comfortable looking at more exotic, non-relational data stores"
In other words, to get MySQL to superduper scale, you have to turn it into NoSQL anyway. At which point, the only reason to be using MySQL at all is inertia, because it's pretty clunky compared to the other alternatives. Back in the day before Cassandra and etc, there wasn't a concept for NoSQL, and sharding MySQL looked like an exotic way of using a relational DB. Actually, no. It's a way of crudely emulating something that isn't a relational DB.