tiflow v6.5.1

Improvements

• Support batch UPDATE statements to improve TiCDC replication performance #8084 @amyangfei
• Enable pull-based sink to optimize system throughput #8232 @hi-rustin
• Support storing redo logs to GCS-compatible or Azure-compatible object storage #7987 @CharlesCheung96
• Implement MQ sink and MySQL sink in the asynchronous mode to improve the sink throughput #5928 @amyangfei @CharlesCheung96

Bug fixes

• Fix the issue that changefeed might get stuck in special scenarios such as when scaling in or scaling out TiKV or TiCDC nodes #8174 @hicqu

• Fix the issue that precheck is not performed on the storage path of redo log #6335 @CharlesCheung96

• Fix the issue of insufficient duration that redo log can tolerate for S3 storage failure #8089 @CharlesCheung96

• Fix the issue that transaction_atomicity and protocol cannot be updated via the configuration file #7935 @CharlesCheung96

• Fix the issue that the checkpoint cannot advance when TiCDC replicates an excessively large number of tables #8004 @overvenus

• Fix the issue that applying redo log might cause OOM when the replication lag is excessively high #8085 @CharlesCheung96

• Fix the issue that the performance degrades when redo log is enabled to write meta #8074 @CharlesCheung96

• Fix a bug that the context deadline is exceeded when TiCDC replicates data without splitting large transactions #7982 @hi-rustin

• Fix the issue that pausing a changefeed when PD is abnormal results in incorrect status #8330 @sdojjy

• Fix the data inconsistency that occurs when replicating data to a TiDB or MySQL sink and when CHARACTER SET is specified on the column that has the non-null unique index without a primary key #8420 @asddongmen

• Fix the panic issue in table scheduling and blackhole sink #8024 #8142 @hicqu

• Fix the issue that the binlog-schema delete command fails to execute #7373 @liumengya94

• Fix the issue that the checkpoint does not advance when the last binlog is a skipped DDL #8175 @D3Hunter

• Fix a bug that when the expression filters of both "update" and "non-update" types are specified in one table, all UPDATE statements are skipped #7831 @lance6716

tiflow v6.1.5

Bug fixes

• Fix the issue that applying redo log might cause OOM when the replication lag is excessively high #8085 @CharlesCheung96
• Fix the issue that the performance degrades when redo log is enabled to write meta #8074 @CharlesCheung96
• Fix the issue that the binlog-schema delete command fails to execute #7373 @liumengya94
• Fix the issue that the checkpoint does not advance when the last binlog is a skipped DDL #8175 @D3Hunter

tiflow v6.6.0

Improvements

• Support batch UPDATE DML statements to improve TiCDC replication performance #8084 @amyangfei

• Implement MQ sink and MySQL sink in the asynchronous mode to improve the sink throughput #5928 @hicqu @hi-rustin

• Optimize DM alert rules and content #7376 @D3Hunter

Previously, alerts similar to "DM_XXX_process_exits_with_error" were raised whenever a related error occured. But some alerts are caused by idle database connections, which can be recovered after reconnecting. To reduce this kind of alerts, DM divides errors into two types: automatically recoverable errors and unrecoverable errors:

    * For an error that is automatically recoverable, DM reports the alert only if the error occurs more than 3 times within 2 minutes.
    * For an error that is not automatically recoverable, DM maintains the original behavior and reports the alert immediately.

• Optimize relay performance by adding the async/batch relay writer #4287 @GMHDBJD

Bug fixes

• Fix the issue that transaction_atomicity and protocol cannot be updated via the configuration file #7935 @CharlesCheung96
• Fix the issue that precheck is not performed on the storage path of redo log #6335 @CharlesCheung96
• Fix the issue of insufficient duration that redo log can tolerate for S3 storage failure #8089 @CharlesCheung96
• 修复 changefeed 在 tikv、CDC 节点扩缩容特殊场景下卡住的问题。 #8197 @hicqu
• Fix the issue of too high traffic among TiKV nodes #14092 @overvenus
• Fix the performance issues of TiCDC in terms of CPU usage, memory control, and throughput when the pull-based sink is enabled #8142 #8157 #8001 #5928 @hicqu @hi-rustin
• Fix the issue that the binlog-schema delete command fails to execute #7373 @liumengya94
• Fix the issue that the checkpoint does not advance when the last binlog is a skipped DDL #8175 @D3Hunter
• Fix a bug that when the expression filters of both "update" and "non-update" types are specified in one table, all UPDATE statements are skipped #7831 @lance6716
• Fix a bug that when only one of update-old-value-expr or update-new-value-expr is set for a table, the filter rule does not take effect or DM panics #7774 @lance6716

tiflow v6.1.4

Improvements

• Add the DML batch operation mode to improve the throughput when SQL statements are generated in batches #7653 @asddongmen
• Support storing redo logs to GCS- or Azure-compatible object storage #7987 @CharlesCheung96

Bug fixes

• Fix the issue that the checkpoint cannot advance when TiCDC replicates an excessively large number of tables #8004 @asddongmen
• Fix the issue that transaction_atomicity and protocol cannot be updated via the configuration file #7935 @CharlesCheung96
• Fix the issue that TiCDC mistakenly reports an error when the version of TiFlash is later than that of TiCDC #7744 @overvenus
• Fix the issue that OOM occurs when TiCDC replicates large transactions #7913 @overvenus
• Fix a bug that the context deadline is exceeded when TiCDC replicates data without splitting large transactions #7982 @hi-rustin
• Fix the issue that sasl-password in the changefeed query result is not masked #7182 @dveeden
• Fix the issue that data is lost when a user quickly deletes a replication task and then creates another one with the same task name #7657 @overvenus
• Fix a bug that DM might raise an error during precheck when the downstream database name in SHOW GRANTS contains a wildcard ("*") #7645 @lance6716
• Fix the issue that DM prints too many logs caused by "COMMIT" in binlog query events #7525 @liumengya94
• Fix the issue that the DM task fails to start when only ssl-ca is configured for SSL #7941 @liumengya94
• Fix a bug that when the expression filters of both "update" and "non-update" types are specified in one table, all UPDATE statements are skipped #7831 @lance6716
• Fix a bug that when only one of update-old-value-expr or update-new-value-expr is set for a table, the filter rule does not take effect or DM panics #7774 @lance6716

tiflow v6.5.0

For new features, improvements, and bug fixes released in 6.5.0 for ticdc and dm, see TiDB 6.5.0 release notes.

tiflow v5.1.5

Bug fixes

• Fix data loss that occurs in special incremental scanning scenarios #5468
• Fix the issue that there are no sorter metrics #5690
• Fix excessive memory usage by optimizing the way DDL schemas are buffered #1386

tiflow v6.1.3

Bug fixes

• Fix data loss occurred in the scenario of executing DDL statements first and then pausing and resuming the changefeed #7682 @asddongmen
• Fix the issue that when collation_compatible is set to "strict", DM might generate SQL with duplicated collations #6832 @lance6716
• Fix the issue that DM tasks might stop with an Unknown placement policy error #7493 @lance6716
• Fix the issue that relay logs might be pulled from upstream again in some cases #7525 @liumengya94
• Fix the issue that data is replicated for multiple times when a new DM worker is scheduled before the existing worker exits #7658 @GMHDBJD

Improvements

• Enable transaction split and disable the safe mode of a changefeed in TiCDC by default to improve performance #7505 @asddongmen
• Improve the performance of Kafka protocol encoder #7540, #7532, #7543 @sdojjy @3AceShowHand

tiflow v6.4.0

Improvements

• Remove the useless operate-source update command from dmctl #7246 @buchuitoudegou
• Fix the issue that DM full import fails if the upstream database uses DDL statements that are incompatible with TiDB. You can create the schema of target tables in TiDB manually in advance using DDL statements supported by TiDB to ensure successful import #37984 @lance6716
• Support replicating the exchange partition DDL statements #639 @asddongmen
• Improve non-batch sending performance for the MQ sink module #7353 @hi-rustin
• Improve performance of TiCDC puller when a table has a large number of Regions #7078 #7281 @sdojjy
• Support reading historical data in the downstream TiDB by using the tidb_enable_external_ts_read variable when Syncpoint is enabled #7419 @asddongmen

Bug fixes

• Fix the issue that DM WebUI generates the wrong allow-list parameter #7096 @zoubingwu
• Fix the issue that a DM-worker has a certain probability of triggering data race when it starts or stops #6401 @liumengya94
• Fix the issue that when DM replicates an UPDATE or DELETE statement but the corresponding row data does not exist, DM silently ignores the event #6383 @GMHDBJD
• Fix the issue that the secondsBehindMaster field is not displayed after you run the query-status command #7189 @GMHDBJD
• Fix the issue that updating the checkpoint may trigger a large transaction #5010 @lance6716
• Fix the issue that in full task mode, when a task enters the sync stage and fails immediately, DM may lose upstream table schema information #7159 @lance6716
• Fix the issue that deadlock may be triggered when the consistency check is enabled #7241 @buchuitoudegou
• Fix the issue that task precheck requires the SELECT privilege for the INFORMATION_SCHEMA table #7317 @lance6716
• Fix the issue that an empty TLS configuration causes an error #7384 @liumengya94
• Fix the issue that sasl-password in the changefeed query result is not masked #7182 @dveeden
• Fix the issue that TiCDC might become unavailable when too many operations in an etcd transaction are committed #7131 @asddongmen
• Fix the issue that redo logs might be deleted incorrectly #6413 @asddongmen
• Fix performance regression when replicating wide tables in Kafka Sink V2 #7344 @hi-rustin
• Fix the issue that checkpoint ts might be advanced incorrectly #7274 @hi-rustin
• Fix the issue that too many logs are printed due to improper log level of the mounter module #7235 @hi-rustin
• Fix the issue that a TiCDC cluster might have two owners #4051 @asddongmen

tiflow v6.1.2

Improvements

• Enhance the performance of the region worker by handling resolved TS in a batch #7078 @sdojjy

Bug fixes

• Fix the issue that the cdc server might panic if it receives an HTTP request before the cdc server fully starts #6838 @asddongmen

• Fix the log flooding issue during upgrade #7235 @hi-rustin

• Fix the issue that changefeed's redo log files might be deleted by mistake #6413 @hi-rustin

• Fix the issue that TiCDC might become unavailable when too many operations in an etcd transaction are committed #7131 @hi-rustin

• Fix the issue that data inconsistency might occur when non-reentrant DDL statements in redo logs are executed twice #6927 @hicqu

• Fix the issue that upstream table structure information is lost when DM tasks enter the sync unit and are interrupted #7159 @lance6716

• Fix large transaction errors by splitting SQL statements when saving checkpoints #5010 @lance6716

• Fix the issue that DM precheck requires the SELECT privilege on INFORMATION_SCHEMA #7317 @lance6716

• Fix the issue that DM-worker triggers a deadlock error after running DM tasks with fast/full validators #7241 @buchuitoudegou

• Fix the issue that DM reports the Specified key was too long error #5315 @lance6716

• Fix the issue that latin1 data might be corrupted during replication #7028 @lance6716

tiflow v6.3.0

Improvements

• Improve compatibility with MySQL 8.0 as data source #6448 @lance6716

• Optimize DDL by executing DDL asynchronously when encounter "invalid connection" #4689 @lyzx2001

• Improve compatibility for MySQL 8.0 upstream #6506 @lance6716

• Support logging start ts of DML statements when MySQL sink gets an error #6460 @overvenus

• Enhance the api/v1/health API to return a more accurate health state of a TiCDC cluster #4757 @overvenus

• Implement MQ sink and MySQL sink in the asynchronous mode to improve the sink throughput #5928 @hicqu @hi-rustin

• Delete the deprecated pulsar sink #7087 @hi-rustin

• Improve replication performance by discarding DDL statements that are irrelevant to a changefeed #6447 @asddongmen

Bug fixes

• Fix the issue that DM reports the Specified key was too long error #5315 @lance6716

• Fix goroutine leak when relay meets an error #6193 @lance6716

• Fix the issue that when collation_compatible is set to "strict", DM might generate SQL with duplicated collations #6832 @lance6716

• Reduce the appearance of the warning message "found error when get timezone from binlog status_vars" in DM-worker log #6628 @lyzx2001

• Fix the issue that latin1 data might be corrupted during replication #7028 @lance6716

• Fix the issue that TiCDC returns an inaccurate error for a wrong PD address with a grpc service #6458 @crelax

• Fix the issue that the cdc cause cli changefeed list command does not return failed changefeeds #6334 @asddongmen

• Fix the issue that TiCDC is unavailable when changefeed initialization fails #6859 @asddongmen