HNHacker News
TopNewBestAskShowJobs

dmytton

2,937 karma · joined May 5, 2008

- Writing https://console.dev

- Researching sustainable computing in the Department of Engineering Science, University of Oxford - publications at https://davidmytton.blog/publications/

david@console.dev / https://davidmytton.blog

submissionscomments
dmytton··on How much is Spotify paying Google Cloud?
Yes, but that has no impact on the cost. It's a flat rate in 64kb blocks. Batching and compression would only affect the networking fees from Google, not the per API call fees. If you batch up 64 1kb messages into a single batch then Google still bills you for 64 publish API calls.
dmytton··on How much is Spotify paying Google Cloud?
This is a good point, but it could be somewhat mitigated by their expected growth of up to 2M per second. To quote the full line from their blog post:

> Currently our production load peaks at around 700K events per second. To account for the future growth and possible disaster recovery scenarios, we settled on a test load of 2M events per second.

In the absence of any other numbers, using the 700k number seems reasonable otherwise you would start to get into modelling without any real data to back up your assumptions. This at least gives you a maximum they would be paying under current production conditions.

dmytton··on How much is Spotify paying Google Cloud?
It'll be interesting to see what other numbers Spotify release in subsequent blog posts for the other services they're using. Hopefully some cost calculations can be run for those, too.
dmytton··on How much is Spotify paying Google Cloud?
Batching doesn't make any difference to the pricing:

> Batched calls are accounted for as multiple operations.

https://cloud.google.com/pubsub/pricing

dmytton··on Now Open – AWS Asia Pacific (Seoul) Region
Google added their 4th region - US East - in October last year[1] but it's true that this is poor effort compared to Azure and AWS.

I wrote an article for InfoWorld about this at the end of last year: http://www.infoworld.com/article/3008617/cloud-computing/glo... and have a map of locations at https://www.google.com/maps/d/viewer?mid=zV7qz2PQ6Sz4.kJHcuw...

[1] http://googlecloudplatform.blogspot.co.uk/2015/10/Bringing-G...

dmytton··on Next generation of Google Cloud SQL
I wonder if you've actually submitted a ticket to Google Cloud support at all? At the Gold level of $400/m I've had excellent experiences including:

a) First responders actually being engineers who know the code well, can reference the open source repositories pointing to where issues are and suggesting workarounds.

b) Full workarounds being coded by the first responder (in Python for our use but they officially support other languages too) which are simple drop in replacement to fix the problem temporarily until the root cause is addressed.

c) Fast troubleshooting of critical issues (within minutes) with helpful debugging advice to resolve the issue.

d) A single contact owning the issue by default, with handover to alternative timezones on requst e.g. when an issue is long running or (in my case) you were in one location for a few weeks then change to another.

e) Consistently achieving the above over 1+ years worth of tickets, including responses handled from Google support teams worldwide to fit into the timezone e.g. Tokyo based support answering questions UK time Sunday night (Monday in Tokyo).

f) The same level of support offered for alpha and beta products even though the official word is "best effort/no SLA"

Where it has been more frustrating is with bugs discovered in the products. These have tended to be escalated to the engineering teams which take a while to fix the root cause, although acceptable workarounds are provided in the meantime. Their support ticket UI is also pretty rubbish.

Google might have a poor reputation for support when you're a consumer, but then you're not paying them anything. For both Google Cloud and Google Apps, their support has been excellent.

dmytton··on Google loses data as lightning strikes
This is S3 which isn't comparable to Google's persistent disks. S3 is equivalent to Google Cloud Storage which has "99.999999999%" durability as per https://cloud.google.com/storage/

To accurately compare them you'd need to look at AWS EBS: "Amazon EBS volumes are designed for an annual failure rate (AFR) of between 0.1% - 0.2%, where failure refers to a complete or partial loss of the volume, depending on the size and performance of the volume."

dmytton··on Global Outage of AWS CloudFront CDN on Nov 26 2014
The Google Cloud status page is at https://status.cloud.google.com/
dmytton··on OnMetal: The Right Way To Scale
This is an interesting announcement for a few reasons:

1) It's true that VMs generally provide bad performance for high i/o applications, particularly databases. There are various ways to mitigate this such as using high IOPS drives like AWS EBS PIOPS or SSD backed storage from Google/Digital Ocean.

2) This is using OpenCompute and is all open source and going to be released, so they can take advantage of the efficiencies of that hardware architecture.

3) It gives you quick access to physical servers connected to your cloud environment. Benefits of flexibility/scalability with the benefits of hardware. This seems to be how they're differentiating against Softlayer who have bare metal servers available via API, but they're billed monthly and take a few hours to provision (which is still pretty impressive).

However, there is no pricing announced - this will be key.

Also, will this start to eat into their managed services perhaps? Anticipating disruption?

dmytton··on Introducing HTTP Load Balancing and SSD Persistent Disk
It's routing traffic to a pool of Compute Engine instances: https://developers.google.com/compute/docs/load-balancing/ht...
dmytton··on Introducing HTTP Load Balancing and SSD Persistent Disk
This is Google using the power of their private network. A single IP routed to your closest data center which can be done because your traffic enters their network at the closest POP and they can then route internally. Currently, you have to do this using anycast DNS – Google is making this a really simple product to get completely automated geographic redundancy/local serving of content. Dyn charge a lot for this product.

Softlayer has a similar product called Global IP where traffic enters their extensive network at the local POP. You can then reroute the IP to any of your servers but it’s not load balanced and is really just another IP. Still a good product which we use to power all our services for automated failover. But Google just one-upped them.

Think AWS elastic IP, but Softlayer allow you to route to any server in any DC. And Google allow you to automatically route to a pool of servers in a location closest to the user.

dmytton··on Don't be evil: Moving everything off of Google
http://dbpmail.net/essays/2013-06-29-hackers-replacement-for... is a good tutorial on replacing GMail.

GMail is excellent at search and spam filtering, everything else is just a standard e-mail service. The tutorial points out how you can at least try and match the level of spam filtering with the community based Pyzor.

dmytton··on A Year of MongoDB
A few comments on the problems:

* CPU bottleneck. The mongod is no usually bound by CPU except for building indexes on existing data (which shouldn't really happen in production). The issue he's talking about is contention between the web server (or workers) and the mongos. This isn't anything unexpected. It's recommended to put the mongos onto the application server and then you scale this by adding CPUs initially but then by adding multiple application nodes and using a load balancer. Or perhaps splitting the mongos onto dedicated nodes?

* Virtualisation: VMs are notorious for having variable performance because you have the overhead of the hypervisor but more importantly, are sharing resources with others. We run performance critical apps on dedicated servers and reserve VMs for tools or things which aren't high throughput e.g. a MongoDB arbiter.

* EBS: This has had known performance issues for years. It's fine as a basic file store but should never be used for databases. PIOPs are the way around this but local instance storage is also an option.

* No transactions: MongoDB has never had them. This is known.

* Schemaless != no schema design. It makes it easy to play around but you still need to think through things carefully. See http://blog.serverdensity.com/mongodb-schema-design-pitfalls...

* No joins. Again, it has never had joins. This is known.

dmytton··on If you love automation, why is your development environment manual?
We do this using Vagrant and Puppet and actually just blogged about it yesterday: http://blog.serverdensity.com/many-projects-with-vagrant-and...
dmytton··on MongoDB 2.4 Released: Text Search, Security, Hash-based Sharding
Using the new working set analyser is going to make figuring out how much memory you need significantly easier. Giving MongoDB enough memory for your working set is the easiest way to get performance but it was quite difficult to figure it out: you had to know both data and index sizes across your most common usage patterns.
dmytton··on Pull Request: Launch GOV.UK
This is an example of how to run a government IT project. Extremely well run team, excellent programmers and designers and using a lot of interesting technology. That's cool for the tech community but for the majority of the users the biggest benefit is a single location for everything. Direct.gov.uk was still a collection of links to lots of external sites but the goal with gov.uk is to have everything on one site. I think this is a great project.

Some background at http://digital.cabinetoffice.gov.uk/2012/10/16/directgov-a-q...

dmytton··on Does everyone hate MongoDB?
Flexibility is useful in development. Once you deploy then things will change less frequently so the real benefit is when you do have to change something, you're not running a big ALTER statement.
dmytton··on Braintree’s Payments Platform Launches Across Europe And Canada
Server Density has recently finished integrating into Braintree having had awful experiences with Netbanx (aka Optimal Payments) in the UK.

Our existing provider, Netbanx, have terrible customer support often taking days to reply to e-mails and refusing to answer questions over the phone. They have no out of hours support for critical issues and continually change their backend and API without informing customers. This meant we often had failed payments because changes Netbanx made which were undocumented and broke our integration. Having a stable and trusted payment processing partner is essential and Netbanx failed to get this right.

Braintree support has been excellent for the few times we've needed it (their dev docs are great, as are the libraries for each language) and it means we now combine our merchant account and payment API in a single provider, one control panel, one set of rates and one location for support.

dmytton··on MongoDB 2.2.0 Released
Not true. We're processing 12TB of data each month with thousands of writes per second and sub-millisecond response times. You simply have to understand how to use MongoDB correctly which isn't difficult with the out of the box settings and by reading some of the documentation. Figuring out your working set is the most difficult part but that doesn't take long to calculate based on understanding the queries you're doing and creating the appropriate indexes.

Since you seem to be fairly negative about MongoDB but light on details, perhaps you should write up your experiences so others can learn from what you did wrong.

dmytton··on MongoDB 2.2.0 Released
It might "feel" like a bad solution not granular enough but if you actually test the implementation then locking is no longer a real concern. There are still improvements to be made and this is the first step towards that. Changing from a global to database lock required huge amounts of work which sets the foundations for the next releases with collection and document level locking.

In my experience replication has been extremely stable and reliable for several point releases so I'm not sure what problems you had.

dmytton··on MongoDB 2.2.0 Released
It's inaccurate that there's no difference between the global lock and database level locking. Whilst it's true that locking is now down at the database level and you get benefits from splitting into multiple databases, the real benefit is from the new PageFaultException architecture. Even using a single database you will see significant performance improvements.

I benchmarked this at http://blog.serverdensity.com/goodbye-global-lock-mongodb-2-... and there is a video explaining how this works at http://www.10gen.com/presentations/concurrency-internals-mon...

dmytton··on Doubling SaaS Revenue By Changing The Pricing Model
It depends if you're "getting rid" of existing customers by forcing them into the new pricing. I don't think that's ever a good idea - they signed up at a specific price and shouldn't be forced to change if they don't want to. That's how Server Density have implemented this change - it's for new signups only. Existing customers can move if they wish but they'll never be forced to.
dmytton··on Doubling SaaS Revenue By Changing The Pricing Model
This is a quote from Patrick and not how Server Density label the customers who are priced out. His observation is that, typically, customers only willing to pay a small amount tend to be higher maintenance.
dmytton··on Doubling SaaS Revenue By Changing The Pricing Model
We request a credit card when the trial expires.
dmytton··on Doubling SaaS Revenue By Changing The Pricing Model
Depends if you're maximising for revenue or signups. The number of signups decreased but the revenue increased significantly, making up for the loss in signups. Of course if you can increase signups now with the new pricing then that would also show revenue increase.
dmytton··on Doubling SaaS Revenue By Changing The Pricing Model
We had a few people pick up on this test (colleagues browsing on different machines) and they understood when we explained we were running a test.

I think so long as you don't try and cover it up and offer those who discover the choice between the new/old pricing, it shouldn't cause any problems.

dmytton··on Mongo: Mr Right... or Mr Right Now?
What version was this? 2.0 (which has been out almost a year) has journaling enabled by default which almost always provides complete single server durability for these situations.

Any database can suffer corruption if you shut down the server uncleanly - it depends on the state of the filesystem when it comes back up. That's why you have replication to protect against failure of a single node.

Having run Mongo in production for over 3 years, I've never lost any data despite numerous failure scenarios (including unclean shutdown).

dmytton··on Amon - server monitoring, simplified logging and error tracking for web apps
As this is self hosted, it leaves you to deal with all the management of things like alert delivery (e.g. ensuring delivery of e-mails, working with SMS gateways) and data retention.

The MongoDB installation that Amon sets up is not in a replica set nor sharded, so it leaves you to deal with redundancy, failover and scaling. We use MongoDB for Server Density and I can tell you that managing huge quantities of data is not easy.

dmytton··on Online services our startup subscribes to
We're using Assistly for our support site and they've had a number of outages in the last few days.

Auto deployment is something we're working on improving but right now you can use auto copy[1], which bases new instance detection on the hostname, our auto deploy script[2] or our API[3] to add devices automatically.

[1] http://support.serverdensity.com/customer/portal/articles/72...

[2] http://support.serverdensity.com/customer/portal/articles/72...

[3] http://developer.serverdensity.com/docs

dmytton··on Our holiday / vacation policy – unlimited with bonuses
The 5th bullet point says 30 days is a reasonable guideline with the bonus after 25 days there to encourage that further.
← PreviousPage 2 of 4Next →