HNHacker News
TopNewBestAskShowJobs

vicaya

365 karma · joined August 24, 2007

hn@vicaya.com
submissionscomments
vicaya··on 80legs Web-Scale Apps Competition
Instead of a thinly veiled advertisement, how about some information on the capability of the hardware: histograms of RAM size, hard drives, CPU units (equivalents of some known CPU) and network speed/latency for these "50,000" machines?

Without a reasonable description of hardware, any ideas would be just hand waving.

vicaya··on Sector/Sphere: High Performance Distributed File System and Parallel Data Engine
One caveat: Sector/Sphere is not used by anyone (notable) in the industry. I hope they become a viable alternative to Hadoop as they do have some strong points on paper.

OTOH, an industry maven told me recently that "they don't listen" (to industry feedback), which I hope, is not entirely true.

vicaya··on NoSQL: Distributed and Scalable Non-Relational Database Systems
He probably equates NoSQL to KV stores that has no chance of having SQL implemented, as they have no efficient range scan capabilities. It's totally feasible to have full SQL (and more) implementation on top over Hypertable.
vicaya··on MooTools: The Open Source Magic Behind Cloudera Desktop UI
Any reasons why they picked MooTools instead of JQuery? MooTools is a Prototype derivative. Prototype derivatives are huge pains, as they litter the global namespace like they own it, which makes sane mashup implementations practically impossible.

I hate Prototype and its ilks with a passion.

vicaya··on Three big myths about MapReduce
MapReduce is about hiding explicit parallel primitives like threads and MPI from users (programmers): You only need to implement two functions map and reduce in a typical framework.

Yes, you can certainly implement MapReduce with MPI partially, but typical MapReduce framework like Hadoop also handles fault tolerance transparently to users.

vicaya··on Sector/Sphere is significantly faster than Hadoop
I'd been a fan of Sector/Sphere and Dr. Gu for a while. But here is the biggest problem of the benchmark: "replication is disabled" for maximum performance, which means these are not realistic benchmarks for large scale clusters (> 200 nodes), where node failure is the norm. I suspect that Sector's replication algorithm is not up to par with HDFS', which uses fairly sophisticated replication pipelining for maximum throughput.

The most interesting thing about Sector/Sphere is that it doesn't use TCP/IP for transport but their own UDT stuff, which has its own pluses (more efficient on high latency links) and minuses (stability issues due to lack of wide usage/testing)

vicaya··on RAMClouds: Scalable High-Performance Storage Entirely in DRAM
Ousterhout and gang are credible guys, but this is completely back of the envelope vaporware. Some numbers are a bit dubious as well, like 1M request/s per server. Let's say the requests is simple messages like twitter with average length of 200 bytes (including message overhead), it would need cross sectional switch bandwidth of 2Gbit/s on small frames; anything more interesting say 20KB web pages, would need 200GBit/s switch bandwidth, which is not gonna happen any soon. It's no big deal to do 1M/s in process, but I'd love to see a real implementation that can do that over a commodity network.

Real RAM clouds already exist, as Bigtable etc have durable in-memory table that can take advantage of SSD as well.

vicaya··on A C++ Challenge
I'd argue that the integer overflow inside new implementation is a compiler bug.
vicaya··on Riak - web-shaped data storage system
1000 nodes are nothing if you crawl the entire web and keep a history. I'm sure someone already has a modified version of an OSS bigtable cluster at 1PB+ and close to 1k nodes :)

>> they don't support range scan efficiently either > Cassandra does.

Only if you call order preserving hashing efficient when all the keys can be easily hashed in to one bucket, no matter what hash function you choose :) Cassandra's range query is an occasionally useful hack, which is not comparable to Bigtable like implementations, in terms of efficiency, scalability and robustness.

The Bigtable's single-master with standbys is never a problem in practice. The AppEngine outage was a due to a bug in GFS master protocol decoding. If this bug is in every node, all nodes would be crashing instead of one with some service (reads and some writes) still available. The so called "fully" distributed model is only better when you assume that you don't have bugs in your code and that node failure is random. I argue that separating different responsibilities/functionalities into different components is good for fault isolation, a sound software engineering practice. Naive fully distributed model is much more brittle in practice due to code bugs.

BTW, although Hypertable is inspired by Bigtable in design, the implementation and features set (support a dozen languages via Thrift and full read isolation via MVCC) are different enough that I wouldn't call it a clone :)

vicaya··on Riak - web-shaped data storage system
They serve different purposes. Riak is a DHT (for key sharding) layer intended to run on top of different backends, just like Dynamo. CouchDB has an integrated storage engine. Riak doesn't support range queries (nature of DHT). I'm pretty sure CouchDB does, which is easy for a single node. CouchDB doesn't scale in the sense that it doesn't do any automatic sharding yet. It's quite possible to use CouchDB as a backend of Riak, though it implies crippling some interesting features in CouchDB.

Technically, I found CouchDB more interesting as it's more "different" from other approaches.

vicaya··on Riak - web-shaped data storage system
Yet another Dynamo/DHT. Now I've seen implementations in Java (Dynamo, Cassandra, Voldemort), Perl (Blekko) and now Erlang. I wonder if any of these has been used on >1000 node cluster? I haven't seen any mention of checksums (besides those in TCP) to ensure that switch/router would not corrupt data (including cluster states), like Amazon learned the hard way.

Of course, they don't support range scan efficiently either. A proper Bigtable implementation is much harder, I guess :)

vicaya··on Ask HN: What was the worst bug you've ever solved ?
Were you using GCC's __attribute__((__packed__))?

Anyway, the standard way (to handle protocols) is to parse the thing not making any assumption of the struct layout.

vicaya··on Judy, an efficient sparse dynamic array implementation
According to Nikolas Askitis' PhD thesis, Judy Array actually underperforms Burst trie (also a sorted collection) by significant margins in both space and time. Burst trie is much easier to implement (a few hundred lines of C++ or Java code) and tune but he has so far refused to release his code for people to independently verify his claims.

cf. http://www.naskitis.com/

vicaya··on Poll: number of monitors attached to your development machine
I get surprising (to me) amount of stuff done on my 10" netbook (a $200 dell mini 10v) with vim and git, while the kids are asleep, though my "main" dev setup is a 18.4" laptop + a 24".
vicaya··on The dark side of NoSQL
Replication != backup, if you/someone accidentally/maliciously screw up your data with the API, the screw ups are replicated as well, especially with Cassandra, where there is no builtin data versioning.

Backups are essential when you're doing major upgrades or data migrations/mangling that can fail.

OTOH, data export is not an inherent problem of distributed database either. If enough people want the feature, they can be built without too much fuss.

vicaya··on The dark side of NoSQL
Well, you can get both with Hypertable, where you can alter table to add/drop column families in O(1) time, while still have schema free benefits with column qualifiers.
vicaya··on The dark side of NoSQL
Well, ad-hoc querying can be implemented in Hypertable with HQL, it's just not a priority at the moment, so features are a bit lacking. I mean, it's not an inherent 'darkside' of distributed database. OTOH, I've done cap shell with HQL and grep/sort/whatnot a few times on a few TB of data and not exactly painful :)
vicaya··on The dark side of NoSQL
With completely schema-free db, you have to encode 'column' in a key, which means you cannot delete or rename a column easily (in O(1) time).
vicaya··on Virtualization: Solution or Problem?
Virtualization is the duct tape.

Service is the only thing that matters to users, which can be built more efficiently upon non-hardware-virtualized distributed computing/storage platforms, where OS will be evolved into mainly security containers.

vicaya··on Newegg’s IPO reveals the numbers behind A $2B Electronics Retailer
They need to move to Nevada like Zappo did to avoid incurring the exorbitant CA retail tax and electronics recycle fee. I often picked Amazon even if Newegg is slightly cheaper and faster in shipping.
vicaya··on Why Stylesheet Abstractions Matter
It seems that most of the benefits listed can also be achieved using any decent template system to serve the css files.
vicaya··on An Introduction to JavaScript's "this"
The behavior of 'this' is from DOM bindings and not from the language per se. Javascript can set 'this' of any calling context to any object with 'apply' or 'call'. The convention for on* handlers is that 'this' is set to the DOM element when the handlers are called. I only learned to appreciate this, when I wrote my own little JS DOM framework a la JQuery or Dojo.
vicaya··on Functional Programming at Facebook. (30 min video, CUFP talk.)
I found 1M messages/day rather too low. So I checked the video, sure enough they have 1B messages/day using close to 10Gbps at peak, with 100+ servers. A correctly built (epoll/reactor based) C/C++ based system should be able to handle that with ~10 servers.

I suspect that erlang is successful despite of the language itself, mostly because it's also a mature product: a well tested production quality actor system with decades of tweaking.

I liked the Q&A especially whether a better type system would catch the "horrible bugs" they had :)

vicaya··on Why You Should Quit Your Job and Travel around the World
That's what noscript is for.
vicaya··on The SSD Relapse: Understanding and Choosing the Best SSD
According to his benchmarks, compilation of C code (Pidgin at least) doesn't seem to benefit from SSD, even the 5400RPM drive performs the same.

SSD for developers is usually waste of money from technical POV, as the working set usually fits in the fs page cache.

30" IPS panels, OTOH, would be money well spent.

vicaya··on Ask HN: Best code editor?
Actually, I found vim quite discoverable, you can pretty much type :help <topic> on anything and you'll see ton of stuff around that are interesting.
vicaya··on Ask HN: Why is concurrent programming so hard ?
General concurrent programming is hard because of the shared mutable states and their interdependencies, which are ripe grounds for deadlocks and race conditions. However, there are many successful attempts to solve large chunks of problems by avoiding shared states in user/application code: e.g., actor models (popularized by Erlang) and map-reduce frameworks, which can be used to solve a large class of parallel programming problems.
vicaya··on Display your current git branch as part of your bash prompt
It also shows the correct state of merge/rebase in the prompt as well. In many distributions, it's installed as /etc/bash_completion.d/git when you install git from a package management system, you just need to source /etc/bash_completion to get all the benefits.

The OP is an example of wheel reinvented, poorly.

vicaya··on GFS: Evolution on Fast-forward
gfs1 is still single master, but the workload is much simpler in this case: it serves the gfs2 master bigtable cluster exclusively. Most of the documented gfs master failures are due to misbehaved map-reduce clients. Also the gfs1 master can be down for extended period of time without affecting the master operations, due to the nature of the cluster (you're unlikely to create a million files per second resulting in much compaction and splits in metadata tablets)

The quote you mentioned actually meant that if you use Bigtable on top of gfs1, the single master failure is more apparent due to the low latency requirement of the application that use the Bigtable.

vicaya··on GFS: Evolution on Fast-forward
I think I independently came up with the same idea to solve the distributed gfs master problem: use a separate bigtable/gfs1 cluster for master metadata for gfs2.

I'm glad that I'm not crazy :)

← PreviousPage 6 of 9Next →