38 karma · joined December 18, 2023
1. A holistic solution for Database Replication/Consolidation for numerous embedded databases used in several instances of edge/desktop applications, into centralized databases/dw/data lake etc.
2. A holistic data streaming solution offering a Kafka Producer and SQL API designed for final destination data delivery.
Few code samples are shown here: https://github.com/syncliteio/SyncLite and https://github.com/syncliteio/SyncLite/tree/main/synclite-co...
A jsp-servlet sample web app is here: https://github.com/syncliteio/synclite-sample-web-app
SyncLite logger transactionally captures the exact sql statements (DDLs and DMls) as executed on the underlying embedded databases, into sqllog files.
The log files for each device (i.e. database) are shipped to a configured staging storage(Local FS/SFTP/S3/MinIO/Kafka etc).
SyncLite consolidator continuously looks for new devices and new logs for each devices.
The received raw sql logs are not directly applicable on final destination as they are logical statements, may involve complex dependencies/joins e.g. insert into select <cols> from t1 JOIN t2 ...
Consolidator applies the incoming sql logs transactionally to a native embedded database (SQLite) and in the process generates record level CDC logs by identifying exact changed records and tables
The CDC logs are record-level INSERT, UPDATE, DELETE statements, DDL statements : CREATE TABLE, ALTER TABLE, DROP TABLE, RENAME TABLE.
The CDC log files as generated are applied to final destination databases.
A summary of this is here : https://www.synclite.io/synclite/sync-ready-apps
Thank you for the feedback and we are improving the documentation.
- SyncLite Logger, a wrapper JDBC driver, supporting popular embedded databases (SQLite, DuckDB, Apache Derby, H2, HyperSQL), creates transactional sql logs for embedded dbs, ships them to configured staging storage ( S3/MinIO/SFTP/Kafka/OneDrive/GoogleDriver etc)
- Several application instances each creating multiple embedded databases/devices are synchronized in real-time on the staging storage.
- SyncLite consolidator, a centralized tool, continuously processes device logs, generates CDC logs, replicates and consolidates staged devices into a wide range of industry leading database, data warehouses and data lakes.
Look forward to feedback and suggestions.
-Build Sync-Ready Applications with Zero-Coding
-Build Streaming Applications For Last Mile Data Integration
-Deploy Smart Database ETL/Replication/Migration Pipelines
-Setup Rapid IoT Data Connectors and more...