Without a reasonable description of hardware, any ideas would be just hand waving.
365 karma · joined August 24, 2007
Without a reasonable description of hardware, any ideas would be just hand waving.
OTOH, an industry maven told me recently that "they don't listen" (to industry feedback), which I hope, is not entirely true.
I hate Prototype and its ilks with a passion.
Yes, you can certainly implement MapReduce with MPI partially, but typical MapReduce framework like Hadoop also handles fault tolerance transparently to users.
The most interesting thing about Sector/Sphere is that it doesn't use TCP/IP for transport but their own UDT stuff, which has its own pluses (more efficient on high latency links) and minuses (stability issues due to lack of wide usage/testing)
Real RAM clouds already exist, as Bigtable etc have durable in-memory table that can take advantage of SSD as well.
>> they don't support range scan efficiently either > Cassandra does.
Only if you call order preserving hashing efficient when all the keys can be easily hashed in to one bucket, no matter what hash function you choose :) Cassandra's range query is an occasionally useful hack, which is not comparable to Bigtable like implementations, in terms of efficiency, scalability and robustness.
The Bigtable's single-master with standbys is never a problem in practice. The AppEngine outage was a due to a bug in GFS master protocol decoding. If this bug is in every node, all nodes would be crashing instead of one with some service (reads and some writes) still available. The so called "fully" distributed model is only better when you assume that you don't have bugs in your code and that node failure is random. I argue that separating different responsibilities/functionalities into different components is good for fault isolation, a sound software engineering practice. Naive fully distributed model is much more brittle in practice due to code bugs.
BTW, although Hypertable is inspired by Bigtable in design, the implementation and features set (support a dozen languages via Thrift and full read isolation via MVCC) are different enough that I wouldn't call it a clone :)
Technically, I found CouchDB more interesting as it's more "different" from other approaches.
Of course, they don't support range scan efficiently either. A proper Bigtable implementation is much harder, I guess :)
Anyway, the standard way (to handle protocols) is to parse the thing not making any assumption of the struct layout.
Backups are essential when you're doing major upgrades or data migrations/mangling that can fail.
OTOH, data export is not an inherent problem of distributed database either. If enough people want the feature, they can be built without too much fuss.
Service is the only thing that matters to users, which can be built more efficiently upon non-hardware-virtualized distributed computing/storage platforms, where OS will be evolved into mainly security containers.
I suspect that erlang is successful despite of the language itself, mostly because it's also a mature product: a well tested production quality actor system with decades of tweaking.
I liked the Q&A especially whether a better type system would catch the "horrible bugs" they had :)
SSD for developers is usually waste of money from technical POV, as the working set usually fits in the fs page cache.
30" IPS panels, OTOH, would be money well spent.
The OP is an example of wheel reinvented, poorly.
The quote you mentioned actually meant that if you use Bigtable on top of gfs1, the single master failure is more apparent due to the low latency requirement of the application that use the Bigtable.
I'm glad that I'm not crazy :)