700 karma · joined August 7, 2013
[1] http://15721.courses.cs.cmu.edu/spring2018/papers/02-inmemor...
The GC problem is not limited to C*. This shit(virtual machine) is hitting the whole Hadoop stack: HDFS, Hive, Spark, Flink, Pig...
Immense number of tickets in any fairly large cluster is related somewhat to GC and JVM behavior.
What is the benefit of that? Who on earth runs a DB written in Java on windows? Any useful server software will end up using platform native features, be it SQL server, MySQL, HBase, ...
There is nothing wrong with hospitals in EU.
I would extend this to cloud as well. The more prevalent cloud becomes, the more ignorant developers become. It's like: I have Mathematica license, who cares how to calculate function derivative?
[0] https://www.quora.com/What-are-some-good-books-on-Distribute...
[1] https://www.quora.com/What-is-a-reading-list-for-distributed...
Long live DistributedLog. One of the great aspects of DistributedLog is the read scaling. Start reading from any replica. Even read proxy node caches records for all consumers. Launch as many read proxies as you need, since read proxies are stateless.
Raft/multi-paxos are already the cheapest communication bound consensus. Theoretically you can't achieve any better with current network guarantees.
> The replica that registers first becomes the new leader. The new leader chooses its LEO as the new HW.
Isn't it a better idea to elect a process who has the largest offset? Basically during leader election, a process should get confirmation of majority of correct processes. If another process observes that it has larger offset in the log, rejects the election and starts another round of election. Eventualy a correct process who has largest offset will become leader. Regarding correctness, this sounds to me as a safer election.
1. Leader replicates to followers
2. Followers get the messages and send ACK to leader
3. Leader gets the confirmation from followers, increments high watermark (HW) and replies client that messages is commited.
4. NOW: Leader fails before it could piggyback to followers that HW has been incremented.
5. The question is: Since potential leader is not aware of the fact that HW has been incremented, it becomes the new leader and truncates the log to old HW, which means we have lost an entry in the log that has been confirmed by previous leader.
As a result, client has been confirmed that the message has been successfully written, but it has been lost.
For CPU, RAM there is no other option. you have to replicate stuff.
OF COURSE every thing is going to work in the lab, BUT MAY BE there is some other corner case in the production that you haven't considered yet. --- Louis C.K. after a sleepless night of switching primary DB to secondaries.
You should try really hard to be unproductive with SQL and any relational system.
- rebalancing: Bookkeeper (which is the storage backend of Pulsar) manages rebalancing way better than kafka.
- Read scaling: Bookkeeper uses consensus to store log records, which means you can read from whatever shard you want. Whereas in kafka, reader has to connect to partition master.