apecloud / ape-dts

English | 中文 Introduction • ape-dts is a data migration tool enabling any-to-any data transfers. • It also provides data subscription and data processing. • It is lightweight, efficient and standalone, requiring no third-party components or extra storage. • Designed for cloud-native stateless component scenarios. • In Rust. Key features • Supports data migration between various databases, both homogeneous and heterogeneous. • Supports snapshot and cdc tasks with resume from breakpoint. • Supports checking and revising data. • Supports filtering and routing at the database, table, and column levels. • Implements different parallel algorithms for different sources, targets, and task types to improve performance. • Allows loading user-defined Lua scripts to modify the data. Supported task types | | mysql -> mysql | pg -> pg | mongo -> mongo | redis -> redis | mysql -> kafka | pg -> kafka | mysql -> starrocks | mysql -> clickhouse | mysql -> tidb | pg -> starrocks | pg -> clickhouse | mysql -> doris | pg -> doris | | :----------------------- | :------------- | :------- | :------------- | :------------- | :------------- | :---------- | :----------------- | :------------------ | :------------ | :-------------- | :--------------- | :------------- | :---------- | | Snapshot | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | | CDC | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | | Data check/revise/review | ✔ | ✔ | ✔ | | | | | | ✔ | | | | | | Structure migration | ✔ | ✔ | | | | | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | Advanced Crate features The dt-main crate provides several optional components which can be enabled via : • : Enable Prometheus format task metrics HTTP service interface. After enabling this feature, you can customize the metrics service with the following configuration: • TBD Quick starts Tutorial • prerequisites • mysql -> mysql • pg -> pg • mongo -> mongo • redis -> redis • mysql -> starrocks • mysql -> doris • mysql -> clickhouse • mysql -> tidb • mysql -> kafka -> consumer • pg -> starrocks • pg -> doris • pg -> clickhouse • pg -> kafka -> consumer • snapshot + cdc without data loss • modify data by lua Run tests Refer to test docs for details. More docs • Configurations • config details • Structure tasks • migration • check • check by Liquibase • Snapshot tasks • data migration • data check • data revise • data review • resume at breakpoint • multiple tables in parallel • CDC tasks • data sync • heartbeat to source database • two-way data sync • generate sqls from CDC • resume at breakpoint • Custom consumers • mysql/pg -> kafka -> consumer • Data processing • modify data by lua • Monitor • monitor info • position info • Task templates • mysql -> mysql • pg -> pg • mongo -> mongo • redis -> redis • mysql/pg -> kafka • mysql -> starrocks • mysql -> doris • mysql -> clickhouse • pg -> starrocks • pg -> doris • pg -> clickhouse Benchmark • MySQL -> MySQL, Snapshot | Method | Node Specs | RPS(rows per second) | Source MySQL Load (CPU/Memory) | Target MySQL Load (CPU/Memory) | | :------- | :--------- | :------------------- | :----------------------------- | :----------------------------- | | ape_dts | 1c2g | 71428 | 8.2% / 5.2% | 211% / 5.1% | | ape_dts | 2c4g | 99403 | 14.0% / 5.2% | 359% / 5.1% | | ape_dts | 4c8g | 126582 | 13.8% / 5.2% | 552% / 5.1% | | debezium | 4c8g | 4051 | 21.5% / 5.2% | 51.2% / 5.1% | • MySQL -> MySQL, CDC | Method | Node Specs | RPS(rows per second) | Source MySQL Load (CPU/Memory) | Target MySQL Load (CPU/Memory) | | :------- | :--------- | :------------------- | :----------------------------- | :----------------------------- | | ape_dts | 1c2g | 15002 | 18.8% / 5.2% | 467% / 6.5% | | ape_dts | 2c4g | 24692 | 18.1% / 5.2% | 687% / 6.5% | | ape_dts | 4c8g | 26287 | 18.2% / 5.2% | 685% / 6.5% | | debezium | 4c8g | 2951 | 20.4% / 5.2% | 98% / 6.5% | • Image size | ape_dts:2.0.25-alpha.1 | debezium/connect:2.7 | | :--------------------- | :------------------- | | 71.4 MB | 1.38 GB | • more benchmark details Contributing Structure Modules • dt-main: program entry • dt-precheck: pre-check, to minimize interruptions during subsequent data operations by identifying issues early for fast failure • dt-connector: extractors + sinkers for databases • dt-pipeline: pipeline to connect extractors and sinkers • dt-parallelizer: parallel algorithms • dt-task: create extractors + sinkers + pipelines + parallelizers according to configurations • dt-common: common utils, basic data structures, metadata management • dt-tests: integration tests • related sub module: mysql binlog connector in rust Build • Minimum supported Rust version (MSRV) The current minimum supported Rust version (MSRV) is 1.85.0. • cargo build • build images Checklist • run fix all clippy issues. Community If you have any questions, you can reach out to us through: • ApeDTS GitHub Discussions • ApeDTS Wechat Account with note **ape-dts**: