2,937 karma · joined May 5, 2008
- Researching sustainable computing in the Department of Engineering Science, University of Oxford - publications at https://davidmytton.blog/publications/
david@console.dev / https://davidmytton.blog
> Currently our production load peaks at around 700K events per second. To account for the future growth and possible disaster recovery scenarios, we settled on a test load of 2M events per second.
In the absence of any other numbers, using the 700k number seems reasonable otherwise you would start to get into modelling without any real data to back up your assumptions. This at least gives you a maximum they would be paying under current production conditions.
> Batched calls are accounted for as multiple operations.
I wrote an article for InfoWorld about this at the end of last year: http://www.infoworld.com/article/3008617/cloud-computing/glo... and have a map of locations at https://www.google.com/maps/d/viewer?mid=zV7qz2PQ6Sz4.kJHcuw...
[1] http://googlecloudplatform.blogspot.co.uk/2015/10/Bringing-G...
a) First responders actually being engineers who know the code well, can reference the open source repositories pointing to where issues are and suggesting workarounds.
b) Full workarounds being coded by the first responder (in Python for our use but they officially support other languages too) which are simple drop in replacement to fix the problem temporarily until the root cause is addressed.
c) Fast troubleshooting of critical issues (within minutes) with helpful debugging advice to resolve the issue.
d) A single contact owning the issue by default, with handover to alternative timezones on requst e.g. when an issue is long running or (in my case) you were in one location for a few weeks then change to another.
e) Consistently achieving the above over 1+ years worth of tickets, including responses handled from Google support teams worldwide to fit into the timezone e.g. Tokyo based support answering questions UK time Sunday night (Monday in Tokyo).
f) The same level of support offered for alpha and beta products even though the official word is "best effort/no SLA"
Where it has been more frustrating is with bugs discovered in the products. These have tended to be escalated to the engineering teams which take a while to fix the root cause, although acceptable workarounds are provided in the meantime. Their support ticket UI is also pretty rubbish.
Google might have a poor reputation for support when you're a consumer, but then you're not paying them anything. For both Google Cloud and Google Apps, their support has been excellent.
To accurately compare them you'd need to look at AWS EBS: "Amazon EBS volumes are designed for an annual failure rate (AFR) of between 0.1% - 0.2%, where failure refers to a complete or partial loss of the volume, depending on the size and performance of the volume."
1) It's true that VMs generally provide bad performance for high i/o applications, particularly databases. There are various ways to mitigate this such as using high IOPS drives like AWS EBS PIOPS or SSD backed storage from Google/Digital Ocean.
2) This is using OpenCompute and is all open source and going to be released, so they can take advantage of the efficiencies of that hardware architecture.
3) It gives you quick access to physical servers connected to your cloud environment. Benefits of flexibility/scalability with the benefits of hardware. This seems to be how they're differentiating against Softlayer who have bare metal servers available via API, but they're billed monthly and take a few hours to provision (which is still pretty impressive).
However, there is no pricing announced - this will be key.
Also, will this start to eat into their managed services perhaps? Anticipating disruption?
Softlayer has a similar product called Global IP where traffic enters their extensive network at the local POP. You can then reroute the IP to any of your servers but it’s not load balanced and is really just another IP. Still a good product which we use to power all our services for automated failover. But Google just one-upped them.
Think AWS elastic IP, but Softlayer allow you to route to any server in any DC. And Google allow you to automatically route to a pool of servers in a location closest to the user.
GMail is excellent at search and spam filtering, everything else is just a standard e-mail service. The tutorial points out how you can at least try and match the level of spam filtering with the community based Pyzor.
* CPU bottleneck. The mongod is no usually bound by CPU except for building indexes on existing data (which shouldn't really happen in production). The issue he's talking about is contention between the web server (or workers) and the mongos. This isn't anything unexpected. It's recommended to put the mongos onto the application server and then you scale this by adding CPUs initially but then by adding multiple application nodes and using a load balancer. Or perhaps splitting the mongos onto dedicated nodes?
* Virtualisation: VMs are notorious for having variable performance because you have the overhead of the hypervisor but more importantly, are sharing resources with others. We run performance critical apps on dedicated servers and reserve VMs for tools or things which aren't high throughput e.g. a MongoDB arbiter.
* EBS: This has had known performance issues for years. It's fine as a basic file store but should never be used for databases. PIOPs are the way around this but local instance storage is also an option.
* No transactions: MongoDB has never had them. This is known.
* Schemaless != no schema design. It makes it easy to play around but you still need to think through things carefully. See http://blog.serverdensity.com/mongodb-schema-design-pitfalls...
* No joins. Again, it has never had joins. This is known.
Some background at http://digital.cabinetoffice.gov.uk/2012/10/16/directgov-a-q...
Our existing provider, Netbanx, have terrible customer support often taking days to reply to e-mails and refusing to answer questions over the phone. They have no out of hours support for critical issues and continually change their backend and API without informing customers. This meant we often had failed payments because changes Netbanx made which were undocumented and broke our integration. Having a stable and trusted payment processing partner is essential and Netbanx failed to get this right.
Braintree support has been excellent for the few times we've needed it (their dev docs are great, as are the libraries for each language) and it means we now combine our merchant account and payment API in a single provider, one control panel, one set of rates and one location for support.
Since you seem to be fairly negative about MongoDB but light on details, perhaps you should write up your experiences so others can learn from what you did wrong.
In my experience replication has been extremely stable and reliable for several point releases so I'm not sure what problems you had.
I benchmarked this at http://blog.serverdensity.com/goodbye-global-lock-mongodb-2-... and there is a video explaining how this works at http://www.10gen.com/presentations/concurrency-internals-mon...
I think so long as you don't try and cover it up and offer those who discover the choice between the new/old pricing, it shouldn't cause any problems.
Any database can suffer corruption if you shut down the server uncleanly - it depends on the state of the filesystem when it comes back up. That's why you have replication to protect against failure of a single node.
Having run Mongo in production for over 3 years, I've never lost any data despite numerous failure scenarios (including unclean shutdown).
The MongoDB installation that Amon sets up is not in a replica set nor sharded, so it leaves you to deal with redundancy, failover and scaling. We use MongoDB for Server Density and I can tell you that managing huge quantities of data is not easy.
Auto deployment is something we're working on improving but right now you can use auto copy[1], which bases new instance detection on the hostname, our auto deploy script[2] or our API[3] to add devices automatically.
[1] http://support.serverdensity.com/customer/portal/articles/72...
[2] http://support.serverdensity.com/customer/portal/articles/72...