In the iceberg equality delete as join solution, why do we need to use the partition field as the join key?

[squalud]

I noticed that in this implementation, in addition to using data_sequence_number and equality_ids, the partition field is also used as the join key. Why is that?

[hantangwangd]

Follow the Iceberg specification, an equality delete file must be applied to a data file when all of the following are true:

1. The data file's data sequence number is strictly less than the delete's data sequence number
2. The data file's partition (both spec id and partition values) is equal to the delete file's partition or the delete file's partition spec is unpartitioned

So we should add partition fields as the join key as well to ensure that the data files and the delete files applied to them located in the same partition.

Referring to here. Hope this can help you.

[squalud]

oh, yes, you do me a big favor. Thanks