I didn't read all their math but I expect their final result to be off by a factor of 2-5x. Hard drives are a surprisingly low percentage of the cost of a storage system.
I didn't read all their math but I expect their final result to be off by a factor of 2-5x. Hard drives are a surprisingly low percentage of the cost of a storage system.
Sia systems don't need a ton of networking. I ran the networking buildout costs by some networking people, and again it comes down to cutting corners. If you only need 10 gbps per rack, if you don't mind having extra milliseconds added, etc, you can get away with very scrappy setups. The whole point is that it's not a highly reliable facility.
Tying two bricks together doesn't make them float.
To me a big part of computer science is abstracting away unreliable details to make them seem reliable.
Now of course that assumes they are uncorrelated, but since Sia nodes are distributed across the internet, that's likely as opposed to multiple servers at a few data centers like AWS.
Turns out mortgage bonds tend to be correlated.
It is when you just start bundling everything and then saying the entire thing is AAA that the trouble starts.
Talking about the disk-space pricing is, IMO, disingenuous. Talk about the math for an all-in use-case of the service. I appreciate what the OP is going for, however, hand-waving around uncomfortable critical items like bandwidth cost for the Sia network is not a good look.
You really need to back that math up.
Here's my math: 10mbps is over 3TB per month at max utilization, so let's say it's good enough for either 2TB throughput with semi-even use, or 1TB throughput if the use is really focused on part of the day.
When buying transit in bulk in the US, Google tells me that 10mbps was very roughly $8.50 a month in 2017, $4.50 a month in 2018, and $2.50 a month in 2019.
Are you expecting the data to be rebuilt every single day or something?
Definitely not worth it.
To get out of Amazon entirely, they definitely want to gouge you, but it's not the end of the world. If $24 a year is an acceptable storage price, then Snowball export costing somewhere around $36/TB in bulk isn't too awful. (And if you don't have enough data to fill up a snowball, you can probably smuggle it out through lightsail.)
A good is example are the Apple iPhone screens. If Apple wants a new competing supplier, they have to invest in new competitors.
This is bad for customers in the long term. Like how Amazon is slowly destroying brick and mortar stores, or having Amazon Basics at the front undercutting other sellers. Monopolies mean you don’t have an open fair market anymore.
I'm explaining a common business move called pay to play, in industries that require very high investments for new competitors. If customers want B to compete, they have to pay B first to ensure B does not make a loss, so everyone wins a little in the end.
So in the case of OP. He needs to finds paying customers first, who are willing to pre-order his service.
so many options
Because of AWS scale it cannot be the cheapest. Amazon cannot use odd lots or small lots, which means that it has very few possible suppliers and those suppliers will never be the cheapest.
Same goes for the network infrastructure. While it is possible buy 100Gbit/sec at a throw away price to because some sales person needs to make his numbers and neither he nor his director of sales thinks Kmod Hosting would be able to fill the pipe it is buying, Amazon would need 900Gbit/sec for their exit and no one at the vendor is going to blink at making Amazon pay higher price than Kmod Hosting.
If you are able to take advantage of sales and clearance deals here and there, you are very likely to get a lower price than even the biggest buyers.
From operational experience reports of S3 I've seen out there and discussions [1], including on HN [2], once you reach a large enough object count, S3 is "sorta-reliable". Objects will disappear. The durability claim is not SLA-enforced [3], which is uptime-oriented. The only remedy is a service credit.
[1] https://www.theregister.co.uk/2018/07/19/data_durability_sta...
I didn’t think much about it, but I guess I was really unlucky.
This post is intended to address people who do not feel that decentralization can be cost effective at scale.
https://support.sia.tech/article/dk91b0eibc-welcome-to-sia - this support article is also a good introduction to the network and why it was built
Also take a look at Skynet, a file sharing protocol built on top of Sia: https://siasky.net/
Reliability, redundancy, disaster recovery, cost cutting, and absolute focus on execution!
Amazon's goal is to get people to pay a premium to get access to their entire ecosystem of services. They don't optimize the price of each individual service. There's a lot of take it or leave it for their specific offerings. Look at how they used to have reduced redundancy as a cost saver, but don't anymore. If you want corner-cutting storage from Amazon, you get funneled into Glacier now.
So let's look at how glacier deep archive is $.99 per TB per month, with a waiting time. And Google's weird offering where it costs $1.23 per TB per month, with instant access, but you pay a bunch of money to access. That means they can't store it on tapes, but they expect to profit anyway. And they're probably not running that as a loss leader that depends on the archival data being accessed.
Actually, looking closer, is one-zone even storing fewer copies? It's offering the same durability, except in the case of "availability zone destruction". So if they're selling three copies for $10 a TB, then the equivalent price for 64-of-96 would be $5 a TB. Cutting that down with cheaper worse hardware would go a long way to get you toward the $2 goal.
Glacier Deep Archive is .99/TB/month in US East and EU Ireland and several others, plus $2.50/TB bulk retrieval.
https://cloud.google.com/storage/pricing
Archive Storage is $1.20/TB/month in us-east1 and europe-west1 and several others. But (instant) retrieval is $50/TB, with no options...
If instead they decide it's the cash cow, then it wouldn't be a stretch to predict that they price it a bit above the competition. The customer has the impression that they can amortize the cost across the other value-adds in the ecosystem. Whether the customer is right or not barely matters to Amazon from a purely financial aspect. All that matters is that they believe it to be true.
There's also the short-term versus the long term strategy. Short term they could do either and I wouldn't be surprised. Long term, I think I would expect the latter.
As someone who has worked with a couple of market leaders in their respective fields i want to disssaude anyone who would listen of this notion. NO, the market leaders aren't doing things optimally. Some things are downright stupid.
E.g bandwidth costs on AWS are high enough that if you actually serve up lots of data from S3 you can typically afford to rent servers to cache all of it 'in front' of AWS and still save a ton of money.
S3 only gets close to competitive if you never access the data from outside of AWS.
Which gets to the point: If you use an AWS service like S3, you pretty much has to use other AWS services if you want the cost to be even somewhat reasonable in aggregate.
They don't need to compete on cost, because once they get you to buy into one set of services, moving any one set of services off AWS gets more painful and/or costly, and a full migration looks too scary for most people.
S3 will never be priced to undercut anything but big players for that reason. They need to be competitive with Google and Azure, because those guys can offer to offset transitioning costs and generally aggressively target AWS customers.
A small player isn't the same threat even if substantially cheaper.
But secondly, offline doesn't usually mean dead and gone, with unstable datacenters like this they are usually back online before the user has lost a full 25% of their redundancy.
Row level and rack level outages are handled by data randomization. The entire Sia system heavily depends on probabilistic techniques, both on the renting and hosting side. Row level failures will take out some of your data, but nobody should be disproportionately impacted by a cluster failure.
On Sia, each piece is at a different site. So 64-of-96 implies that each chunk of data (96 pieces to a chunk) is located in 96 different places. This doesn't help with the geo-bandwidth, but as discussed above there are other techniques to handle that.
Surprisingly, bandwidth pricing on the Sia network is even cheaper than storage pricing relative to centralized competition. That's a lot harder to model at scale though, so we aren't as confident the Sia bandwidth pricing will hold up at $1 / TB in the long term.
And technically, most of this stuff is customizable per-customer. If your particular use case has a different optimal parameterization, it's fairly easy to tune your client to suit your particular needs.
Reed-Solomon encoding adds 50%, of you want 3 block per 2 data blocks. Replicated encoding (not relevant here since this is allow throughput usecase, but necessary if you want to sustain high read throughput) is adding at least 200% (if you want a 3x replication, which I think should be the minimum).
I mean this is basically the moment where I would expect every systems designer on HN coming out of the woods and crushing Sia into the ground, if there were, in fact, any ground at all to crush Sia into.
Is this actually legit? If so, where is the rejoicing? What am I missing?
Engineering is about tradeoffs. I could build a network as good as Google's with infinite money, infinite time, and infinite help. I could design a product as beautify as Apple's with the same lack of limitations. Unfortunately for me, I have limited money, limited time, and limited help. Every systems designer understands that, innately, so isn't rushing out of the woodword because Sia and Dropbox have merely chosen different tradeoffs. That one has IPO'd is uninteresting in the abstract. It's just money after all.
At this point the cautious skeptic will be thinking "hmm, maybe there's something to this", not necessarily full on rejoicing.
That said, I agree it does seem promising. If you ever find yourself in need of cheap cloud storage it wouldn't hurt to look into Sia as a possible option.
That puts the cost of 192 TB at more like $6240, not $4945. Could be less if you find a good deal on mini-SAS PCIE cards, but still going to be substantially higher than $4945.
In theory something like an Epyc 3000-series with lots of PCIe or onboard SATA that supports port multipliers would work great, but I don't think anyone actually makes that.
Can't be more than 2.5 because Backblaze B2 already gives you $5/TB/Mo.
Well it can be, if they have a lot of inefficiencies. Backblaze could have more experienced engineers who overcame these. I assure you, I can accidentally design a very expensive storage system as I’m not that smart ;)
That is also not the only service that Backblaze offers and wasn't their first. It could be that B2 is simply a way for them to offset their cost for extra capacity and are running it effectively near-cost for them.
About a third of the users are under 100GB.
Another third is under 500GB.
13% from 500GB to 1TB.
9% from 1TB to 2TB.
8% from 2TB to 5TB.
5% above that.
And they cited raw costs of over $3.50 per TB. But of course that's with real datacenters and non-shoddy equipment.
Somewhat of a random question, can you point me to some state of the art research?
We build clusters of low PB scale that have a TCO with everything from labor to hardware, from financing to electric, from routers to cables and can be run below 3€/TB. For that you can store data in Block(rbd, iscsi), Objekt(S3, swift) or Filestorage(CephFS, NFS, SMB), high available on Supermicro hardware in any datacenter worldwide.
Feel free to contact us or use our free community edition to start your own cluster.
With a standard fileshare, you need to walk the filesystem to retrieve your file - this incurs a ton of metadata overhead. It also means when you've got potentially billions of files in a directory, it can be slooooowww. All the metadata requests also make it very chatty - so doing it over a WAN link tends to be extremely painful if it works at all. Newer versions of SMB and NFS have done a lot to batch the metadata requests but they are still protocols meant to happen at extremely low latency inside a datacenter.
However, you touched a key point: object stores are all about throughput, not latency. You can store at a GB/s (if you have the pipes), but even checking if an object exists will cost you a few milliseconds.
Which is not that hard to do.
And all those protections in a datacenter are far from free.
I looked at their parts list and it's obvious they aren't serious. CPU is missing, memory is missing, SAS to SATA cables, but no SAS controller, no mounting for the system board. Low effort at best.
The price was $4700 something, now it's $4945.
They simply smooth over it with this:
"So we will be using a rig cost of $4500 in our spreadsheet."
That way their overall math doesn't change. Should get banned for doing things like that. These guys are up to no good.
THIS! There's no use of a 2TB storage if you can't upload/download this amount each month