Amazon reveals new web service: Amazon Simple Database
seattle.craigslist.org
seattle.craigslist.org
In the meantime, I've been developing a framework using EC2, S3, S3DFS (DFS = Distributed File System) along with SQLite to try and solve the database problem on AWS.
S3DFS allows you to mount an S3 bucket to one or multiple EC2 instances as if it were a local filesystem. It works at the block level and has read/write caching - so is very fast. (Note: It's free to use for development, but otherwise requires a paid commercial license.)
Since SQLite uses flat files and has no database server or client, it is ideal for use on S3. SQLite only scales well to a point, so to get around its limitations I am separating each user into their own database (each database is a separate flat file). This can sometimes be a bad idea, but SQLite allows you to attach multiple databases together to essentially create a temporary master database where you can run queries across all the user databases at once. This will be handy for search indexing, site-wide stats, etc. In order to change the user databases I will have utilities to allow global scheme alterations.
So, by adding EC2 instances that are all attached to the same S3 bucket, and using round-robin dns to distribute traffic among them, you can theoretically scale a web service infinitely. Just launch a new EC2 instance and add it to the round-robin dns pool and you're done! Even that could be automated to respond dynamically to traffic.
Imagine the long-term cost-savings from only having to build your app and architecture once and not having to go through a more complex scaling process like this: http://www.scribd.com/doc/21793/Scaling-PHPMySQL-Presentation-from-Flickr
Certainly you should only worry about scaling after your app is up and running, but what if you didn't have to ever worry about it? That's the goal of this framework.
This is pre-alpha of course and I'm still in the development and experimentation stage, but I wanted to share with the community and get some feedback!
http://www.openfount.com/blog/s3dfs-for-ec2
http://docs.amazonwebservices.com/AmazonEC2/gsg/2006-06-26/
http://docs.amazonwebservices.com/AmazonS3/2006-03-01/gsg/
Update: Since this was so lengthy, I also added it as a post here: http://blog.nanobeepers.com/2007/04/07/infinitely-scalable-framework-with-aws/
Update2: Please leave any feedback here: http://news.ycombinator.com/comments?id=10001
Were looking for an experienced technical expert in highly scalable distributed systems technologies to join our Software Platform group as a Sr. Level Engineer. The engineering team is responsible for the overall design and delivery of all the hard-core technical components that drive SimpleDBs huge multi-tier service-oriented architecture. The product (SimpleDB) is a new product designed to optimize real-time lookup of information for internet applications. As a senior leader in the business, the successful candidate will work closely with Management, Customers, QA, Design, Product Management, and Customer Support to maintain and enhance existing product as well as to develop new products. You will help architect, design, and implement highly scalable distributed infrastructure that has availability, reliability and performance guarantees. Drive server-wide and cross-group initiatives, and be the domain expert. Ivory tower architects need not apply - this is hands on pos!
ition where you will be asked to do everything from build rock-solid components to formulate strategy and evangelize technology, lead company-wide architectural changes, mentor other engineers and provide training and support for our supported technologies. You need to not only be a top software developer with an established track record of delivering, but also excel in communication, leadership and customer focus. If you want to test your limits and tackle technical challenges you won't see anywhere else then come join our group
Key Responsibilities:
Translation of complex functional and technical requirements into detailed architecture and design Be very hands-on; work with the engineering team to manage the day-to-day development activities by leading architecture decisions, participating in designs, design review, code review, and implementation. Responsible for the over-all systems architecture, scalability, reliability, and performance. Help the development team delivery quality applications on-time and on-budget and execution against project plans and delivery commitments Maintain current technical knowledge to support rapidly changing technology, always on a look out for new technologies and work with management and development team in bringing new technologies. Candidate must be able to work with a minimum of technical supervision and supplemental engineering support, while responding efficiently to multiple program priorities Work with engineering teams to design and optimize a wide variety of development projects using a variety of development tools and practices. In addition, the role involves developing shared components and tools, analyzing the latest technologies and development trends, establishing and implementing standard practices as well as training and mentoring other engineers.
QUALIFICATIONS:
Bachelors degree or equivalent experience Excellent verbal and written communication skills Ability to work well with people and be both highly motivated and motivating Experience working in a small startup, or venture-funded company highly desirable Proven results oriented person with a delivery focus. 10+ year's overall development experience and 7+ year's enterprise software experience. Strong knowledge of data structures, algorithm, enterprise systems, and asynchronous architectures. Deep understanding of web services software architectural and design issues. Effective at architecting complex interactions across several subsystems to resolve real-world issues, articulating the architecture and obtaining buy-in from other teams. Deep understanding of performance, memory, and I/O, threading. Solid highly scalable database application design skills Deep systems-level programming skills in Java/C++. Experience developing middleware systems and/or distributed system software is preferred Outstanding adherence to project deadlines and understands the balance between doing it right and getting the job done. Works well in a team environment and be able to effectively drive cross-team solutions that have complex dependencies and requirements. Experience of leading the architecture of an open source messaging product is preferred. Strong technical vision, presentation and technology leadership skills. Ability to handle multiple competing priorities in a fast-paced environment Previous demonstrated project management success on similar projects emphasis placed on experience with high-volume web-based applications
In joining our team, you'll enjoy a competitive salary, great benefits, a creative and comfortable work environment, and the exciting opportunity to be part of a fast-paced and growing technology company.