Hacker News Reader

Comment by base | Hacker News Reader

Is there an easy way to have ClickHouse Cloud ingest data from MySQL hosted in Amazon RDS?

There are a few options for migrating or synchronizing data from MySQL - I'd recommend starting with this page in the ClickHouse Docs - there is a nice video there that explains some of those options:

https://clickhouse.com/docs/en/integrations/migration/

Depending on what you are trying to achieve, you could use clickhouse-local with the MySQL engine to move data, or could use an ETL/ETL tool to migrate/sync

Good point. If you just want to copy data far and away the easiest way to transfer data is using MySQLDatabaseEngine. You can even copy table definitions. Watch for issues with Decimal datatypes if you do this.

Be careful with this engine, it's easy to accidentally expose the password as you only need table read permissions if it wasn't set up using an external credential file.

https://github.com/ClickHouse/ClickHouse/issues/3311

I also had some pretty bad join performance (CH table joined to MySQL table), the quick solution to both of these is that we instead use the table function (https://clickhouse.com/docs/en/sql-reference/table-functions...) to copy the data periodically.

Use Named Collections to protect credentials. They are very handy. Here's an article that discusses use in the JDBC Bridge.

https://altinity.com/blog/connecting-clickhouse-to-external-...

Disclaime: I work for Altinity.

Check out the Altinity Sink Connector for ClickHouse [0]. This is advancing quite quickly and already has prod deployments. Please feel free to try it out.

[0] https://github.com/Altinity/clickhouse-sink-connector

Reply on news.ycombinator.com