Replication Method not available in Target

[joaopamaral]

We’re currently working on developing a new Singer target and came across a question regarding the replication_method. Is there a specific reason why this property isn’t typically defined or used on the target side?

Our expectation was that the replication_method could help the target determine whether to overwrite all table data (Full Table) or to insert/upsert based on incremental replication. However, we’ve noticed that some targets seem to infer this behavior solely from the presence or absence of key_properties— not just to determine insert vs. upsert behavior, but even to decide whether to recreate the table — which doesn’t always align with the intended replication strategy as key_properties could be define to FULL_TABLE as we can see in this example.

@edgarrmondragon, do you happen to know if there’s a standard around this, or any rationale for targets ignoring replication_method?

[joaopamaral]

cc @jcellary

[edgarrmondragon]

Hey @joaopamaral 👋

Is there a specific reason why this property isn’t typically defined or used on the target side?

Probably no good reason other than the Singer spec not covering it initially.

Our expectation was that the replication_method could help the target determine whether to overwrite all table data (Full Table) or to insert/upsert based on incremental replication.

I think this is kind of what ACTIVATE_VERSION¹ messages accomplish:

The ACTIVATE_VERSION message type is a mechanism that is used in full table sync mode which tells the target to disregard all previously received records and only consider the current version as “active”.

I've got a WIP PR², which I've had a hard time putting time into so more eyes are welcome in case there are gaps in the implementation.

Footnotes

1. https://hub.meltano.com/singer/docs/#activate-version ↩

2. https://github.com/meltano/sdk/pull/2686 ↩

[joaopamaral]

I think this is kind of what ACTIVATE_VERSION1 messages accomplish:

Nice! Thank you for the reply @edgarrmondragon, I'll check the ACTIVATE_VERSION if it will work for us.

I've got a WIP PR2, which I've had a hard time putting time into so more eyes are welcome in case there are gaps in the implementation.

Sure! I'll take a look 👀 . Thanks