HNHacker News
TopNewBestAskShowJobs

shenli3514

1,056 karma · joined September 7, 2015

Blog: https://medium.com/@shenli3514
submissionscomments
shenli3514··on Building a Large-Scale Distributed Storage System Based on Raft
We also introduced another role (Raft-learner) to the Raft group and make a columnar extension. More details can be found here: https://pingcap.com/blog/delivering-real-time-analytics-and-...
shenli3514··on Jepsen: TiDB 2.1.7
Thanks for your excellent work!
shenli3514··on Progress in performance and scalability with CockroachDB
“Between 90 and 135 16 vCPU nodes depending on cloud hardware ” How many nodes did you use in the CRDB 2.0 TPC-C 10k benchmark? Could I say that the "5x increment" is on the same hardware condition? Thanks!
shenli3514··on MySQL High Availability at GitHub
TiDB developer here. There are more than one hundred companies who use TiDB in their production environment. The number keeps growing. The link in sanxiyn's post shows part of them. I think it is safe to say that TiDB is mature enough.
shenli3514··on What I wish I knew when I became CTO
>Of the list, AngularJS and MySQL have been the only ones to give us scaling problems. Our monolithic AngularJS code-bundle has got too big and the initial download takes quite a while and the application is a bit too slow. MySQL (in RDS) crashes and restarts due to growing BI query complexity and it’s been hard to fix this.

Maybe they should try TiDB(https://github.com/pingcap/tidb). It is a MySQL drop-in replacement that scales.

shenli3514··on TiDB – a global scale distributed DB
We use different isolation levels and priorities for OLAP and OLTP workload.
shenli3514··on TiDB – a global scale distributed DB
We will release a new benchmark result soon.
shenli3514··on TiDB – a global scale distributed DB
I mean the raft/mvcc/transaction layer which are on top of RocksDB and below the SQL layer.
shenli3514··on TiDB – a global scale distributed DB
There are some key differences between TiDB and Cockroach.

1. User interface and eco-system Despite that TiDB and CockroachDB both support SQL, TiDB is compatible with MySQL protocol while Cockroach chooses PostgreSQL. You can directly connect to TiDB server with any MySQL client.

2. Architecture The whole TiDB project is logically divided into two parts: the stateless SQL layer (TiDB) and distributed storage layer (TiKV). As TiDB is built on top of TiKV, developers have the freedom to choose to use TiDB or TiKV, depending on their own business. If you only want a distributed Key-Value database, you can just use TiKV alone for higher performance and lower latency.

In a word, our system is highly-layered and modularized while CockroachDB is a P2P system. The design of our system results in the fact that we use two programming languages: Go for TiDB and Rust for TiKV to improve the storage performance.

And benefit by the highly-layered architecture, we build another project[1] to run Apache Spark to on top of TiDB/TiKV to answer the complex OLAP queries. It takes advantages of both the Spark platform and the distributed TiKV cluster.

3. Transaction model Even though CockroachDB and TiDB both support ACID transaction, TiDB uses a model introduced by Google’s Percolator. The key feature of this model is that it needs an independent timestamp allocator. Like Spanner, each transaction in TiDB will have a timestamp to isolate different transactions.

The model that CockroachDB uses is similar to the TrueTime API that Google described in its paper. However, unlike Google, CockroachDB didn’t build the atomic clocks and GPS receivers to keep the time consistent across different data centers. Instead, it uses NTP for clock synchronization, which leads to the problem of uncertain errors. To solve this problem, CockroachDB adapts the Hybrid Logical Clocks (HLC) algorithm.

4. Programming Language TiDB uses Go for the SQL layer and Rust for the storage engine layer. As Go has a Garbage Collector (GC) and runtime, we think it will cost us days to tune the performance. Therefore, we use Rust, a static language, for TiKV. Its performance is much better. CockroachDB only uses Go.

[1] Spark on TiKV: https://github.com/pingcap/tispark

shenli3514··on Show HN: TiDB 1.0 is production ready
We have a document for Docker deployment: https://github.com/pingcap/docs/blob/master/op-guide/docker-...

And we will provide a Kubernetes based deployment solution soon.

shenli3514··on CockroachDB 1.1 Released: Production Made Easy
Hi xmichael99,

I'm a lead developer from TiDB, and thank you for voicing your experience regarding reporting bugs in our community. I can assure you that our team member would never ever offend you, or any other developer contributing to TiDB, in such a way. We are grateful for your participation and support.

Could you send us a link to the issue you are referring to, so we can look into it more and make sure it's resolved? If you have any other issue, feel free to email me directly as well: shenli@pingcap.com

Thank you for your feedback. TiDB's progress would not be possible without people like you.

shenli3514··on Distributed SQL database (TiDB) source code explained
Thanks for your attention.
shenli3514··on Distributed SQL database (TiDB) source code explained
Here is two examples about how to use TiKV's KV API: https://github.com/pingcap/tidb/blob/master/cmd/benchkv/main... https://github.com/pingcap/tidb/blob/master/cmd/benchraw/mai...
shenli3514··on TiDB – A distributed NewSQL database compatible with MySQL protocol
Please feel free to discuss in this google group or raise a new issue on github.
shenli3514··on TiDB – A distributed NewSQL database compatible with MySQL protocol
Yes, TiDB is a Relational database.
shenli3514··on TiDB – A distributed NewSQL database compatible with MySQL protocol
We use RocksDB as the single-machine storage engine and build TiKV above RocksDB as a distributed storage engine. At first we consider use Go to build TiKV, but the cgo overhead between TiKV and Rocksdb is considerable. So we turn to Rust. The network communication overhead between TiDB and TiKV has nothing to do with cgo.
shenli3514··on TiDB – A distributed NewSQL database compatible with MySQL protocol
There are two more things: 1. Go has high development efficiency. So we could build complex SQL logic easily. 2. Go is easy to write concurrent code. So we could enjoy the benefit of multi-core cpu. For example, we could use multiple goroutine to scan data, do join, do aggregation.
shenli3514··on TiDB – A distributed NewSQL database compatible with MySQL protocol
Go code communicate with Rust code through network. So there is no overhead. Go has interface, which could be used for expressing the AST, plan tree. We define some interface for AST node and plan tree node. It is convenient to apply some rules (predict-pushdown/column-pruning/cost-computing) on the tree. You could refer to the code in https://github.com/pingcap/tidb/blob/master/plan/plan.go
shenli3514··on TiDB – A distributed NewSQL database compatible with MySQL protocol
Yes, Go has higher development efficiency than Rust, especially for new comers. For a storage engine, the most important thing is stable. We focus on write right and stable codes. Rust's compiler is quite strict. Which is good for write right codes. We have many good programmers who is very good at writing Rust. So we do not suffer from the low development efficiency. About the libraries, we could get most of we need. But some were missing when we begin to develop TiKV (GRPC/HTTP2 for example). But thanks to Rust Team/Community, they give us great supports. GRPC for Rust is available now. We also contribute back to the community. We develop prometheus client for Rust. The Rust community is active and helpful. So we do not worry about the lack of libraries.
shenli3514··on TiDB – A distributed NewSQL database compatible with MySQL protocol
Go is good at concurrent concurrency and has great development efficiency. We could easily develop complex SQL logical and parallelize many operators. But its GC and cgo overhead is not suitable for developing storage engine. So we choose Rust to develop TiKV.
shenli3514··on Ingesting MySQL data at scale – Part 1
If there is a RDMS with great scale capacity, it will save the effort of data migration. Spark/Hadoop could read data from the RDMS directly.
← PreviousPage 2 of 2