748 karma · joined August 17, 2009
However one of the perks of being a male is that society only ever expects me to take a shower to be presentable. So I'm totally cool if a colleague wants to leave the camera off even if it's a 1 on 1.
It's actually quite hard to fix bugs in charset/collations, because any changes to the sort order implicitly could affect the on-disk format (indexes are sorted).
(It makes use of skeema, which allows you to track your schema in a declarative way vs. ALTER TABLE statements.)
The article is MariaDB specific.
I was the product manager, and we did have complaints about the size. So it was removed:
https://mysqlserverteam.com/mysql-8-0-retiring-support-for-l...
I don't believe there have been any regrets.
Redefining `utf8` to mean 4-byte would break the upgrade since existing tables would not be able to join against newly created tables.
This is discussed here: https://mysqlserverteam.com/sushi-beer-an-introduction-of-ut...
https://mysqlserverteam.com/new-defaults-in-mysql-8-0/
https://dev.mysql.com/doc/refman/5.7/en/added-deprecated-rem...
https://dev.mysql.com/doc/refman/5.6/en/server-default-chang...
One detail that is not always obvious is how much work goes into limiting regressions. The work to switch to utf8mb4 really started in MySQL 5.6 by not allocating the sort buffer in full (and then further improved in 5.7). 8.0 then added a new temptable storage engine for variable length temp tables.
These are not small cases either: When you compare to latin1 because the _profile_ of queries could change from all in memory to on disk, we could be talking about 10x regressions. In MySQL 8.0 it is more like 11% https://www.percona.com/blog/2019/02/27/charset-and-collatio...
Edit: Also forgot to mention, switching the default character set broke over 600 tests. It's not as easy as it sounds!
I believe SqlAlchemy defaults to READ-COMMITTED, whereas TiDB defaults to (and recommends) MySQL's default of REPEATABLE-READ.
I agree that column and row store have very different characteristics, but what I think is worth mentioning is that some hybrid solutions actually store as both row and columnar and have a query optimizer that can pick between them. For example: Oracle DB In-Memory, SQL Server Columnstore index.
At the same event as this announcement, we also announced that we are working on TiFlash which will do similar. Stay tuned for a blog post with more details :-)
w.r.t. nested transactions, this is not something that MySQL currently offers (TiDB is MySQL 5.7 compatible). Sometimes this is emulatable via savepoints, which is a feature we plan to add in the future.
(We must have missed a spot, and I will follow up and make sure it is addressed).
We try to be transparent about the differences from MySQL. On the compatibility page, there are a few cases described such as large transactions, small transactions and single threaded workloads:
Just to expand a little bit on this comment:
> The expected data volume for TiDB is larger than memory.
There is both total data volume, and working set (workload dependent). This point was in regards to some databases use data structures which require the total data to be memory resident.
The working set is a longer discussion - it could just be a few percent of the total volume, and some working sets have a "long tail" of cache misses vs. short tail. And it can vary for a B+tree vs. an LSM tree too. RocksDB uses compression which helps a lot.
It is not clear in this benchmark if the working set fits entirely in memory. Thanks for bringing it to my attention - I will follow up and make sure we do a better job of communicating what is being tested.
In the interim though, when comparing TiDB to (application) sharded systems, it is important to clarify that FOREIGN KEYS will only be available locally to a single server. So it is a limitation that some of the large deployments we encounter are already familiar with.
The community has added a Redis protocol on top of TiKV with Titan: https://medium.com/@shafreeck/titan-a-distributed-redis-prot...
I expect to see more, including native language drivers directly to TiKV.
For a quick answer, I believe Daniel Abadi's followup comment addresses it:
> This post did not discuss a central oracle. It discusses the difference between partitioned vs global consensus.
In TiDB the pd leader is a central oracle.
- TiDB is Open Source (Apache 2.0 license). Several others that you mention here are commercial offerings.
- The expected data volume for TiDB is larger than memory. I believe MemSQL, for example, is memory-only.
- The architecture of TiDB is inspired by Google Spanner.
- We try to be transparent on less-suited cases. See large+small transactions, single-threaded workloads from: https://www.pingcap.com/docs/sql/mysql-compatibility/
In regards to the analytical piece:
- We suggest you use TiDB for "adhoc OLAP", and Spark for more complicated cases. While parallel, the data is still stored in a row-format (more on that next year!), so an OLAP-only solution may still have performance advantages. TiDB also supports hash joins/aggregation/sort merge joins etc. So compared to MySQL for example, you should see quite a performance improvement.
Hope this helps!
However, I want to mention mitigating circumstances which have helped TiDB stabilize:
- We use RocksDB as the underlying storage library, which was developed by Facebook, and has been battle-tested by many others.
- The architecture itself has been inspired (and proven) by Google Spanner
- Correctness has been proven with TLA+ https://github.com/pingcap/tla-plus
- We have 300+ customers storing 15 PiB+
It is a relational DB (not time series). To describe a couple of differentiators from its peers:
- It aims to optimize both OLTP and OLAP workloads (aka HTAP)
- It uses a component-based architecture (the TiDB server is stateless and speaks the MySQL protocol. TiKV is the distributed storage layer. Thus, you can scale either independently. You can also connect to tikv directly from Spark).
Ensuring compatibility from a clean slate is hard, but to make MySQL distributed requires more than the current storage engine API provides. Thus, you would be likely be creating a fork and no longer getting the benefits of having an upstream[1].
The surgery you need to perform on the code would also reduce you from full compatibility (switching from pessimistic to optimistic locking means some statements no longer work). There are some great side benefits to a clean slate too: In TiDB we're able to use modern languages such as Go for the TiDB Server, and Rust for TiKV.
([1] Phrasing this question to upstream: I think they'd like to offer such functionality, but introducing this level of backward incompatibility is hard. They are also very concerned about performance regression bugs, which would be very hard to prevent with all the refactoring required.)
Window functions & CTEs are only very recent features in MySQL 8.0 (TiDB is 5.7 compatible). None the less, they are important for HTAP workloads, and I'm looking forward to seeing them too.
The gap of features missing is documented here: https://www.pingcap.com/docs/sql/mysql-compatibility/
I would rate compatibility as actually pretty good: all but one SQL mode is supported (which is a feat in itself), and most of the SQL functions are supported.
There are some exceptions though, some which are addressable (missing functions) and some that are not (often a property of being an optimistic system).
We try to be as transparent as possible on this, which might be part of the reason why you feel there is a lot missing?
If you have specific examples, I would be happy to clarify. We also have a course designed for MySQL DBAs, which is designed to make the adoption easier: https://www.pingcap.com/tidb-academy
Many Percona patches did make it into MySQL 5.7 and MySQL 8.0.
That's not to say that I'm not a fan of it, but it's for a smaller sized deployment than GitHub. You can see this for yourself on the customer list here: