Comment by kyledrake | Hacker News Reader

Comment by kyledrake | Hacker News Reader

Parent Full thread

This x 1000. Even when you throw your own CDN on top of this, the bandwidth markup is nasty. Neocities would be unsustainable if we used GCS for our hosting.

I'm paying about $0.01/GB right now, and I've seen market rate at half that. And that's not even directly using IP transit providers. You can get a gigabit unmetered for $450-1000/mo which you can shove a theoretical 324TB through every month. The difference in the numbers is so staggering I sometimes wonder if I'm even doing the math right.

Perhaps their bandwidth is better somehow (prove it), but 12-18x better it is probably not, and having truffle shavings added to your IP transit really adds up when you're hauling a lot of traffic. If you're doing something with heavy BW usage and low margins, be careful with stuff like this. It quickly becomes much more expensive than doing it yourself.

I'd love to be wrong here. I'm sitting next to 60 pounds of storage servers I'm setting up for a data center, they're taking up my entire living room. I would love to get out of the data persistence business forever. But at these BW rates, it's never going to happen.

You can get gigabit + a full cabinet of colocation from Hurricane Electric for $400/mo so you know: https://he.net/colocation.html

I'm sharing it with a few other people and it's great.

I've looked at this. Easily the best deal in town. A few notes for people considering it:

- It's IP transit and a DC from HE, which means it's possibly not BGP peering through other transit providers? If this is true, if HE's network fails, so does your server's internet connection. Typically you want to be multi-homed to ensure redundancy, unless you're doing something like an Anycast CDN and you don't care if it craps out for a few hours.

- HE is the cheapest transit for reasons. It may not be a big deal for what you're doing, but be aware of the differences https://news.ycombinator.com/item?id=5348624

- They only provide a 15A circuit (likely only 80% usable) for a 42U rack, which is pretty ridiculous. You'll blow through that pretty quickly if you're using dual Xeon servers. It's cheaper under their pricing to get a second 42U rack than it is to get the proper amount of power needed for a full cabinet.

- If you're doing Anycast, be aware that HE doesn't provide any BGP communities except blackholing, which could make it hard to tune your network.

(If any of this sounds annoying to deal with and think about, you know exactly why I'd prefer that the cloud providers had better BW costs so I could not have to do this anymore. Again, if the business model works with GCS, congratulations, use it.)

> It's IP transit and a DC from HE, which means it's possibly not BGP peering through other transit providers?

The gigabit transit that comes with the cabinet is HE-only, yep. You can however buy transit and an interconnect from any other carrier in the datacenter (there are quite a few in FMT2 at least).

> They only provide a 15A circuit for a 42U rack, which is pretty ridiculous.

Yeah, it's pretty shitty. I ended up going with Xeon D for the power efficiency. Works pretty well.

I sent them a quote request, and was asked to send in the master order form via fax...Sales rep did not offer another option.

At our highest egress tier, our CDN offering around GCE (or GCS in Alpha), starts being competitive [1]. I agree though that if you just want some bandwidth, no major public cloud provider comes close to the kind of transit rates you'll find for "this isn't VOIP or gaming...". How much egress does Neocities do a month? Feel free to ping me if you'd prefer to discuss offline.

Disclosure: I work for Google Cloud.

[1] https://cloud.google.com/cdn/pricing#cache_egress_pricing_ta...

$0.02 for North America is a rate where I start to go "okay, that could make sense" for entry level. I would call it priced right if it was extremely high quality bandwidth, as in you made some good peering arrangements with the eyeball ISPs and had some strong low-saturation IP transit in the mix. You would have to convince me of it's quality. My traceroutes should show a lot of direct regional peering to the major ISPs and Level 3 caliber transit.

I really think you could differentiate a from your competition hugely by just using that $0.02/GB point for all your customers, regardless of the amount of egress they're using. It's a big chicken-and-egg problem to have those bandwidth costs initially as a startup, but then be able to handle them as a bigger organization. It made us move to our own infrastructure, and once we're here, I already have my infrastructure in a datacenter at that point, so why bother with the cloud?

Even better (for me anyways) would be to provide an option for the "this isn't VOIP or gaming" bandwidth. Not everybody needs that. Actually, probably most people don't. For what I would use GCS for I only need transit to other datacenters. That bandwidth is much cheaper than major ISP peering bandwidth because there aren't monopolies like Comcast extorting everybody for peering.

out of curiosity, what storage do you use?

any experience using backblaze b2 as hot storage, and glacier or google coldline as backup

I'm planning to build a CephFS cluster at this point. I've given up on finding a cloud storage provider that will work well for us. This will require a fairly high fixed monthly cost to get space and transit at a datacenter, but after ~10-30TB BW you start to save a lot of money.

There are trade offs. Aside from more upfront costs and a fixed monthly, Ceph is ridiculously complicated. Interface wise, they need some much better abstractions.

B2 was a strong candidate, their BW rates are a little high still but approaching reasonable. The one issue is that they seem to have inconsistent latency. Not a problem for most use cases, but I need nearly all requests to come back in <100ms consistently, as I'm using this for web hosting. Their use case seems more focused on "hot standby backup" than on high availability ATM. I'm strongly considering them as the backup provider for my storage cluster.

FWIW, GCS is not winning any best-of-show awards for latency either. S3 at the time I ran tests was doing a better job.

I read a lot of good stuff on CephFS, but never tried it.

What makes Ceph complicated? The underlying idea and how Ceph works is fairly simple.

This is the "quick installation": http://docs.ceph.com/docs/master/start/

It really shouldn't be this complex. I would love to just be able to boot an executable with a simple config file and be done with it. SeaweedFS shines a light on how this could be improved: https://github.com/chrislusf/seaweedfs

... which naturally brings the question: why choose Ceph and not SeaweedFS ? I've read a bit about the latter (especially the Facebook paper), and the design and operability seem simple enough that it might be a good starting point

Ceph is more production-ready, and has a lot of things SeaweedFS I believe doesn't yet have (such as consistency checking and rebuilding of failed nodes). It also has FS layer capability. SeaweedFS can use a filer but it currently doesn't support indexing (so you can't do subdirectory lookups for example, you would need to know the full filename).

SeaweedFS is fundamentally trying to do a different thing than Ceph is, and it benefits certain use cases. Reading the Facebook Haystack paper will give you an idea of the differences.

That said, I'm extremely impressed with how simple the interface is. It's very easy to get started.

> Backblaze B2 was a strong candidate... The one > issue is that they seem to have inconsistent latency.

I work at Backblaze, and B2 is our second product (Online Backup was our first product). We are less experienced at serving up data that has fixed latency requirements, but we're learning and actively improving this area all the time. I'm curious when you last tried it and I would be interested in hearing about your experiences (in a personal message if you like).

In a nutshell, the first time we read a file we build it from the vaults (vaults are our lowest layer which is reliable but slowest) and that should be fairly consistent with some caveats (see below) and then for at least the next 24 hours it should be extremely fast and consistent being served out of our SSD cache layer.

Caveats: If you upload a ton of files it is getting loaded into the most recent vault we have deployed. So for three or four days you and everybody else are uploading tons and tons of files to this exact vault causing great loads. After ten days the vault will be "full" and we'll deploy another vault and suddenly serving up your files will become a lot more consistent and easier and faster because the vault is almost entirely idle. What we just started doing is deploying twice as many vaults at once to lower their load in half while they "fill up" making the response in the first 10 days faster and more consistent.

hurstdog can speak to this more, but we absolutely don't scream on latency (particularly time to first byte). Internally, we rely heavily on caching, and that's just now been made available (in Alpha) for Cloud CDN to wrap GCS. Does <100ms for cache fill matter, or just aggregate p95 < 100ms including caching?

I'd also like to point everyone at Zach Bjornson's wonderful blogpost from last year [1], which was both thorough and independent!

Disclosure: I work on Google Cloud.

[1] http://blog.zachbjornson.com/2015/12/29/cloud-storage-perfor...

I am currently using the Hetzner SX131 in RAID 1 and http://minio.io. For about 200€ you get (2x) 30 TB and 50 TB traffic included. Thats about 0.007€ per GB with "free" traffic.

is minio.io a service or server like cephfs?

It is similar to cephfs, but I found it to be more performant and easier to set up.

Minio is basically a local implementation of the S3 API so you can access your stuff remotely and integrate with S3 compatible services using it.

reading the documentation, it looks like more of an NFS replacement with S3 API; I did not see any mention of clustering and redundancy

Reply on news.ycombinator.com