[Bug Fix] Bring back fivetran_deleted for gl_detail models
#20
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CHANGELOG.md
Outdated
|
|
||
| ## Breaking Changes (`--full-refresh` required) | ||
| - Reintroduced `_fivetran_deleted` from the `gl_detail` source table, as the field was not fully deprecated within in the connector. | ||
| It is null in normal incremental syncs, but can populate true in historical resyncs. |
There was a problem hiding this comment.
Suggested change
| It is null in normal incremental syncs, but can populate true in historical resyncs. | |
| **It is null in normal incremental syncs, but can populate true (or false) in historical resyncs.** |
CHANGELOG.md
Outdated
| ## Breaking Changes (`--full-refresh` required) | ||
| - Reintroduced `_fivetran_deleted` from the `gl_detail` source table, as the field was not fully deprecated within in the connector. | ||
| It is null in normal incremental syncs, but can populate true in historical resyncs. | ||
| - This ensures the general ledger models in the downstream `dbt_sage_intacct` transform package excludes deleted records from `stg_sage_intacct__gl_detail`. |
There was a problem hiding this comment.
Suggested change
| - This ensures the general ledger models in the downstream `dbt_sage_intacct` transform package excludes deleted records from `stg_sage_intacct__gl_detail`. | |
| - This ensures the general ledger models in the downstream `dbt_sage_intacct` transform package exclude deleted records from `stg_sage_intacct__gl_detail`. |
models/stg_sage_intacct.yml
Outdated
| @@ -413,6 +413,10 @@ models: | |||
| description: When transaction is modified | |||
| - name: paid_at | |||
| description: When transaction was paid | |||
| - name: _fivetran_deleted | |||
There was a problem hiding this comment.
Suggested change
| - name: _fivetran_deleted | |
| - name: is_detail_deleted |
We should also update the analogous is_batch_deleted column in the staging gl_batch model
models/stg_sage_intacct.yml
Outdated
Comment on lines
417
to
419
| description: The timestamp of when the data was deleted by Fivetran. Although the `_fivetran_deleted` column is no | ||
| longer being actively synced in incremental runs in the connector in `gl_detail`, | ||
| it can populate with `true/false` values in historical resyncs. |
There was a problem hiding this comment.
Something more like this
Suggested change
| description: The timestamp of when the data was deleted by Fivetran. Although the `_fivetran_deleted` column is no | |
| longer being actively synced in incremental runs in the connector in `gl_detail`, | |
| it can populate with `true/false` values in historical resyncs. | |
| description: Boolean field indicating whether the record was deleted. Due to source limitations, this field is currently populated with `nulls` in incremental syncs in Fivetran. In historical resyncs, it may be populated with `true` or `false`. |
There was a problem hiding this comment.
Can you apply the same update to the description of stg_sage_intacct__gl_batch._fivetran_deleted (which should be renamed to is_batch_deleted there)
There was a problem hiding this comment.
Oh could you also update the descriptions in the source yml?
There was a problem hiding this comment.
Wasn't this behavior just for gl_detail? or was it for other models like gl_batch as well?
There was a problem hiding this comment.
Confirmed. Did update the _fivetran_deleted descriptions to reference them as booleans rather than timestamps.
CHANGELOG.md
Outdated
| # dbt_sage_intacct_source v0.4.0 | ||
| [PR #20](https://github.com/fivetran/dbt_sage_intacct_source/pull/20) includes the following updates: | ||
|
|
||
| ## Breaking Changes (`--full-refresh` required) |
There was a problem hiding this comment.
Suggested change
| ## Breaking Changes (`--full-refresh` required) | |
| ## Breaking Changes |
The package doesn't actually have any incremental models
fivetran-avinash
commented
Feb 5, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Overview
This PR will address the following Issue/Feature: Internal ticket
This PR will result in the following new package version: v0.4.0
Bringing back in a new field.
Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:
Breaking Changes (
--full-refreshrequired)_fivetran_deletedfrom thegl_detailsource table, as the field was not fully deprecated within in the connector.It is null in normal incremental syncs, but can populate true in historical resyncs.
dbt_sage_intaccttransform package excludes deleted records fromstg_sage_intacct__gl_detail._fivetran_deletedto:is_batch_deletedinstg_sage_intacct__gl_batchis_detail_deletedinstg_sage_intacct__gl_detailint_sage_intacct__active_gl_detailin thedbt_sage_intacctpackage.Under the Hood
gl_detailyml and seed files with_fivetran_deleteddocumentation and records for testing.PR Checklist
Basic Validation
Please acknowledge that you have successfully performed the following commands locally:
Before marking this PR as "ready for review" the following have been applied:
Detailed Validation
Please share any and all of your validation steps:
See transform package
If you had to summarize this PR in an emoji, which would it be?
🧪