[FLINK-34864][cdc-connector-mysql] Add the IgnoreNoPrimaryKeyTable parameter to skip tables without primary keys in multi-table synchronization #4027
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rameter to skip tables without primary keys in multi-table synchronization
lvyanquan
reviewed
May 27, 2025
...test/java/org/apache/flink/cdc/connectors/mysql/source/assigners/MySqlChunkSplitterTest.java
Outdated
Show resolved
Hide resolved
...c/src/main/java/org/apache/flink/cdc/connectors/mysql/debezium/reader/BinlogSplitReader.java
Outdated
Show resolved
Hide resolved
...mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/MySqlDataSourceOptions.java
Outdated
Show resolved
Hide resolved
| "Whether to ignore tables without primary key. When enabled, the connector will skip tables " | ||
| + "that don't have a primary key. By default these tables will be processed, but for some " | ||
| + "scenarios it may be desirable to ignore them since tables without primary keys " | ||
| + "might cause performance issues during incremental snapshot."); |
There was a problem hiding this comment.
What would happen if we met the records of these tables in incremental reading phase? Skip it or emit it?
There was a problem hiding this comment.
Can you add an ITCase test to verify it during snapshot and incremental reading phases?
There was a problem hiding this comment.
The code logic content has been newly added and verified through testing.
lvyanquan
reviewed
May 27, 2025
...src/main/java/org/apache/flink/cdc/connectors/mysql/source/assigners/MySqlChunkSplitter.java
Show resolved
Hide resolved
added 2 commits
June 3, 2025 10:36
…c table creation of tables without primary keys and the issue of incremental data synchronization
# Conflicts: # docs/content.zh/docs/connectors/flink-sources/mysql-cdc.md # docs/content/docs/connectors/flink-sources/mysql-cdc.md
added 3 commits
June 10, 2025 17:44
…c table creation of tables without primary keys and the issue of incremental data synchronization
|
@lvyanquan Could you please let me know when you can take a look at this PR and merge it? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/FLINK-34864
Previously, in the scenario of multi-table synchronization such as from MySQL to Doris, if some tables had no primary keys, it would directly result in an error, which affected the synchronization of other tables. Therefore, relevant development work was carried out. Incidentally, I found that this issue had already been recorded on Jira. As a result, I added unit tests and other content and submitted this pull request (PR). The specific usage is to set the ignoreNoPrimaryKeyTable parameter to "true" in the pipeline of the YAML file, and then this function can be achieved.