[pipeline-connector/paimon] Support table specific table.properties configuration #4078
Conversation
nickdelnano
changed the title
| String keyWithoutPrefix = key.substring(PREFIX_TABLE_PROPERTIES.length()); | ||
| // General table option: table.properties.<property> | ||
| tableOptions.put(keyWithoutPrefix, value); | ||
| } else if (key.startsWith(PREFIX_SPECIFIC_TABLE_PROPERTIES)) { |
There was a problem hiding this comment.
Shouldn't we flip these so that PREFIX_SPECIFIC_TABLE_PROPERTIES take priority of PREFIX_TABLE_PROPERTIES?
There was a problem hiding this comment.
Actually… I just re-read it and realised it's in the for-each in parsing, not when they're actually applied.
There was a problem hiding this comment.
That is the behavior. Test case testTableSpecificOptionsPrecedenceOverTableOptions demonstrates it.
The function here parses the configuration. Then here the table specific overrides are applied when a table is created and its properties are set.
|
@lvyanquan can you review this change? |
|
We can modify the table options through transform, but I agree to provide more configurations in the sink as alternative solutions, refer to https://nightlies.apache.org/flink/flink-cdc-docs-release-3.4/docs/core-concept/transform/#specify-table-creation-configuration. Compared to transform, the current configuration requires too many parameters to be set for situations where many tables need to be parameterized(users usually synchronize a large number of tables in one job). Can we provide a solution for parameter configuration based on regular expressions (Selectors)? |
This PR adds a new format for defining table properties as
specific.table.properties.<database_name>.<table_name>.<property>. This allows defining properties that are specific to a table. The table specific property takes precedence.Paimon sink supports defining table properties as
table.properties.*. They apply to ALL tables. In some cases it is warranted to set table properties based on the table use case. Some examples are compaction settings, Iceberg compatibility settings, etc. This PR allows this.