[Bug Fix] Bring back fivetran_deleted for gl_detail models

CHANGELOG.md

@@ -1,3 +1,21 @@
+# dbt_sage_intacct_source v0.4.0
+[PR #20](https://github.com/fivetran/dbt_sage_intacct_source/pull/20) includes the following updates:
+
+## Breaking Changes (`--full-refresh` required)
+- Reintroduced `_fivetran_deleted` from the `gl_detail` source table, as the field was not fully deprecated within in the connector. 
+It is null in normal incremental syncs, but can populate true in historical resyncs. 
+- This ensures the general ledger models in the downstream `dbt_sage_intacct` transform package excludes deleted records from `stg_sage_intacct__gl_detail`.
+- Renamed `_fivetran_deleted` to: 
+    - `is_batch_deleted` in `stg_sage_intacct__gl_batch` 
+    - `is_detail_deleted` in `stg_sage_intacct__gl_detail`
+    - This ensures column name uniqueness when they are joined into `int_sage_intacct__active_gl_detail` in the `dbt_sage_intacct` package.
+
+## Documentation
+- Corrected references to connectors and connections in the README. ([#19](https://github.com/fivetran/dbt_sage_intacct_source/pull/19))
+
+## Under the Hood
+- Updated `gl_detail` yml and seed files with `_fivetran_deleted` documentation and records for testing.
+
 # dbt_sage_intacct_source v0.3.1
 [PR #16](https://github.com/fivetran/dbt_sage_intacct_source/pull/16) includes the following updates:
 

README.md

@@ -22,7 +22,7 @@
 ## How do I use the dbt package?
 ### Step 1: Prerequisites
 To use this dbt package, you must have the following:
-- At least one Fivetran Sage Intacct connector syncing data into your destination.
+- At least one Fivetran Sage Intacct connection syncing data into your destination.
 - A **BigQuery**, **Snowflake**, **Redshift**, **PostgreSQL**, or **Databricks** destination.
 
 #### Databricks Dispatch Configuration
@@ -39,7 +39,7 @@ If you are **not** using the [Sage Intacct transformation package](https://githu
 ```yaml
 packages:
   - package: fivetran/sage_intacct_source
-    version: [">=0.3.0", "<0.4.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.4.0", "<0.5.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 ### Step 3: Define database and schema variables
 By default, this package runs using your destination and the `sage_intacct` schema. If this is not where your Sage Intacct data is (for example, if your Sage Intacct schema is named `sage_intacct_fivetran`), add the following configuration to your root `dbt_project.yml` file:

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'sage_intacct_source'
-version: '0.3.1'
+version: '0.4.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'sage_intacct_source_integration_tests'
-version: '0.3.1'
+version: '0.4.0'
 config-version: 2
 
 profile: 'integration_tests' 

integration_tests/seeds/sage_intacct__gl_detail.csv

@@ -1,11 +1,11 @@
-recordno,accountno,accounttitle,amount,batch_date,batch_no,batch_title,batchkey,bookid,creditamount,debitamount,customerid,customername,description,docnumber,entry_date,entry_state,entrydescription,line_no,recordid,recordtype,totaldue,totalentered,totalpaid,tr_type,trx_amount,trx_creditamount,trx_debitamount,vendorid,vendorname,whencreated,whendue,whenmodified,whenpaid
-YGy2nQ7YHphQv5+5AdQTyQ==,15030,A+5CoJT/HDmjwrGYGL6RUg==,2492.5,2021-12-01,74885,El+ECK1x7a25SuLqaMwODA==,506019,/8yT7c/TpIKmxe0K8MmSNA==,0,2492.5,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,06m0www4a75rDMeUgh3o0w==,2021-12-01,P,06m0www4a75rDMeUgh3o0w==,3,20760,1lkow/+J96atpaza+bjZwQ==,2492.5,2492.5,0,1,2492.5,0,2492.5,V-005638,NJa+iTk8TGFrCNzU4GaQMg==,2021-11-16,2021-11-16,2021-12-10 11:47:03,2021-12-10
-GA/oILgpDw/6ay94J1jgLA==,20000,HtcR/HtKU0hM5hLuZcjjuA==,-2492.5,2021-12-01,74885,El+ECK1x7a25SuLqaMwODA==,506019,/8yT7c/TpIKmxe0K8MmSNA==,2492.5,0,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,06m0www4a75rDMeUgh3o0w==,2021-12-01,P,1B2M2Y8AsgTpgAmY7PhCfg==,2,20760,1lkow/+J96atpaza+bjZwQ==,2492.5,2492.5,0,-1,-2492.5,2492.5,0,V-005638,NJa+iTk8TGFrCNzU4GaQMg==,2021-11-16,2021-11-16,2021-12-08 16:00:57,2021-12-10
-aA1UkdHgMiUM9APkcv3MXA==,20000,HtcR/HtKU0hM5hLuZcjjuA==,-6825,2021-12-01,75155,SNfyN9QHUE4npoS72Hd1Bw==,506509,/8yT7c/TpIKmxe0K8MmSNA==,6825,0,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,2021-12-01,P,1B2M2Y8AsgTpgAmY7PhCfg==,2,20239,1lkow/+J96atpaza+bjZwQ==,6825,6825,0,-1,-6825,6825,0,V-002809,fYhwLN+cpy+roem/6mSQXg==,2021-11-04,2021-11-04,2021-12-09 18:22:58,2021-12-10
-geRi+pxC0L/61P7xBaJdPA==,58300,GbTAkJACGOjrVzQghLqbWg==,6825,2021-12-01,75155,SNfyN9QHUE4npoS72Hd1Bw==,506509,/8yT7c/TpIKmxe0K8MmSNA==,0,6825,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,2021-12-01,P,vItcRirL0sTcu0nfSOts+w==,3,20239,1lkow/+J96atpaza+bjZwQ==,6825,6825,0,1,6825,0,6825,V-002809,fYhwLN+cpy+roem/6mSQXg==,2021-11-04,2021-11-04,2021-12-10 11:31:18,2021-12-10
-SYe36P3OgdN7+/ocUIlQJw==,20000,HtcR/HtKU0hM5hLuZcjjuA==,378.38,2021-12-10,40933,SNfyN9QHUE4npoS72Hd1Bw==,506993,/8yT7c/TpIKmxe0K8MmSNA==,0,378.38,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,2021-12-10,P,j13Eh8tAgyBPUNi2GaU8ow==,51,,SYjPoEyQZSe8RTfgN62BvA==,0,14000,14000,1,378.38,0,378.38,V-003017,Fik0+VTeuZo+2L4rH/Y5GA==,2021-12-10,2021-12-10,2021-12-10 12:05:54,2021-12-10
-RfwHVmMm2yi5pzPHaHgfwA==,20000,HtcR/HtKU0hM5hLuZcjjuA==,378.38,2021-12-10,40933,SNfyN9QHUE4npoS72Hd1Bw==,506993,/8yT7c/TpIKmxe0K8MmSNA==,0,378.38,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,2021-12-10,P,j13Eh8tAgyBPUNi2GaU8ow==,63,,SYjPoEyQZSe8RTfgN62BvA==,0,14000,14000,1,378.38,0,378.38,V-003017,Fik0+VTeuZo+2L4rH/Y5GA==,2021-12-10,2021-12-10,2021-12-10 12:05:54,2021-12-10
-R+r/suM2XEm3pV4OpKEUAA==,20000,HtcR/HtKU0hM5hLuZcjjuA==,378.38,2021-12-10,40933,SNfyN9QHUE4npoS72Hd1Bw==,506993,/8yT7c/TpIKmxe0K8MmSNA==,0,378.38,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,2021-12-10,P,j13Eh8tAgyBPUNi2GaU8ow==,15,,SYjPoEyQZSe8RTfgN62BvA==,0,14000,14000,1,378.38,0,378.38,V-003017,Fik0+VTeuZo+2L4rH/Y5GA==,2021-12-10,2021-12-10,2021-12-10 12:05:54,2021-12-10
-8eKBstZUwA0YkRjpTduzxw==,20000,HtcR/HtKU0hM5hLuZcjjuA==,378.38,2021-12-10,40933,SNfyN9QHUE4npoS72Hd1Bw==,506993,/8yT7c/TpIKmxe0K8MmSNA==,0,378.38,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,2021-12-10,P,j13Eh8tAgyBPUNi2GaU8ow==,25,,SYjPoEyQZSe8RTfgN62BvA==,0,14000,14000,1,378.38,0,378.38,V-003017,Fik0+VTeuZo+2L4rH/Y5GA==,2021-12-10,2021-12-10,2021-12-10 12:05:54,2021-12-10
-LbAdQij3k+nNd4i0+vLJLw==,20000,HtcR/HtKU0hM5hLuZcjjuA==,378.38,2021-12-10,40933,SNfyN9QHUE4npoS72Hd1Bw==,506993,/8yT7c/TpIKmxe0K8MmSNA==,0,378.38,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,2021-12-10,P,j13Eh8tAgyBPUNi2GaU8ow==,37,,SYjPoEyQZSe8RTfgN62BvA==,0,14000,14000,1,378.38,0,378.38,V-003017,Fik0+VTeuZo+2L4rH/Y5GA==,2021-12-10,2021-12-10,2021-12-10 12:05:54,2021-12-10
-lztVVJ9OI+1P5nTsHkyJOg==,20000,HtcR/HtKU0hM5hLuZcjjuA==,378.38,2021-12-10,40933,SNfyN9QHUE4npoS72Hd1Bw==,506993,/8yT7c/TpIKmxe0K8MmSNA==,0,378.38,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,2021-12-10,P,j13Eh8tAgyBPUNi2GaU8ow==,49,,SYjPoEyQZSe8RTfgN62BvA==,0,14000,14000,1,378.38,0,378.38,V-003017,Fik0+VTeuZo+2L4rH/Y5GA==,2021-12-10,2021-12-10,2021-12-10 12:05:54,2021-12-10
+recordno,accountno,accounttitle,amount,batch_date,batch_no,batch_title,batchkey,bookid,creditamount,debitamount,customerid,customername,description,docnumber,entry_date,entry_state,entrydescription,line_no,recordid,recordtype,totaldue,totalentered,totalpaid,tr_type,trx_amount,trx_creditamount,trx_debitamount,vendorid,vendorname,whencreated,whendue,whenmodified,whenpaid,_fivetran_deleted
+YGy2nQ7YHphQv5+5AdQTyQ==,15030,A+5CoJT/HDmjwrGYGL6RUg==,2492.5,2021-12-01,74885,El+ECK1x7a25SuLqaMwODA==,506019,/8yT7c/TpIKmxe0K8MmSNA==,0,2492.5,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,06m0www4a75rDMeUgh3o0w==,2021-12-01,P,06m0www4a75rDMeUgh3o0w==,3,20760,1lkow/+J96atpaza+bjZwQ==,2492.5,2492.5,0,1,2492.5,0,2492.5,V-005638,NJa+iTk8TGFrCNzU4GaQMg==,2021-11-16,2021-11-16,2021-12-10 11:47:03,2021-12-10,FALSE
+GA/oILgpDw/6ay94J1jgLA==,20000,HtcR/HtKU0hM5hLuZcjjuA==,-2492.5,2021-12-01,74885,El+ECK1x7a25SuLqaMwODA==,506019,/8yT7c/TpIKmxe0K8MmSNA==,2492.5,0,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,06m0www4a75rDMeUgh3o0w==,2021-12-01,P,1B2M2Y8AsgTpgAmY7PhCfg==,2,20760,1lkow/+J96atpaza+bjZwQ==,2492.5,2492.5,0,-1,-2492.5,2492.5,0,V-005638,NJa+iTk8TGFrCNzU4GaQMg==,2021-11-16,2021-11-16,2021-12-08 16:00:57,2021-12-10,FALSE
+aA1UkdHgMiUM9APkcv3MXA==,20000,HtcR/HtKU0hM5hLuZcjjuA==,-6825,2021-12-01,75155,SNfyN9QHUE4npoS72Hd1Bw==,506509,/8yT7c/TpIKmxe0K8MmSNA==,6825,0,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,2021-12-01,P,1B2M2Y8AsgTpgAmY7PhCfg==,2,20239,1lkow/+J96atpaza+bjZwQ==,6825,6825,0,-1,-6825,6825,0,V-002809,fYhwLN+cpy+roem/6mSQXg==,2021-11-04,2021-11-04,2021-12-09 18:22:58,2021-12-10,FALSE
+geRi+pxC0L/61P7xBaJdPA==,58300,GbTAkJACGOjrVzQghLqbWg==,6825,2021-12-01,75155,SNfyN9QHUE4npoS72Hd1Bw==,506509,/8yT7c/TpIKmxe0K8MmSNA==,0,6825,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,2021-12-01,P,vItcRirL0sTcu0nfSOts+w==,3,20239,1lkow/+J96atpaza+bjZwQ==,6825,6825,0,1,6825,0,6825,V-002809,fYhwLN+cpy+roem/6mSQXg==,2021-11-04,2021-11-04,2021-12-10 11:31:18,2021-12-10,FALSE
+SYe36P3OgdN7+/ocUIlQJw==,20000,HtcR/HtKU0hM5hLuZcjjuA==,378.38,2021-12-10,40933,SNfyN9QHUE4npoS72Hd1Bw==,506993,/8yT7c/TpIKmxe0K8MmSNA==,0,378.38,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,2021-12-10,P,j13Eh8tAgyBPUNi2GaU8ow==,51,,SYjPoEyQZSe8RTfgN62BvA==,0,14000,14000,1,378.38,0,378.38,V-003017,Fik0+VTeuZo+2L4rH/Y5GA==,2021-12-10,2021-12-10,2021-12-10 12:05:54,2021-12-10,FALSE
+RfwHVmMm2yi5pzPHaHgfwA==,20000,HtcR/HtKU0hM5hLuZcjjuA==,378.38,2021-12-10,40933,SNfyN9QHUE4npoS72Hd1Bw==,506993,/8yT7c/TpIKmxe0K8MmSNA==,0,378.38,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,2021-12-10,P,j13Eh8tAgyBPUNi2GaU8ow==,63,,SYjPoEyQZSe8RTfgN62BvA==,0,14000,14000,1,378.38,0,378.38,V-003017,Fik0+VTeuZo+2L4rH/Y5GA==,2021-12-10,2021-12-10,2021-12-10 12:05:54,2021-12-10,FALSE
+R+r/suM2XEm3pV4OpKEUAA==,20000,HtcR/HtKU0hM5hLuZcjjuA==,378.38,2021-12-10,40933,SNfyN9QHUE4npoS72Hd1Bw==,506993,/8yT7c/TpIKmxe0K8MmSNA==,0,378.38,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,2021-12-10,P,j13Eh8tAgyBPUNi2GaU8ow==,15,,SYjPoEyQZSe8RTfgN62BvA==,0,14000,14000,1,378.38,0,378.38,V-003017,Fik0+VTeuZo+2L4rH/Y5GA==,2021-12-10,2021-12-10,2021-12-10 12:05:54,2021-12-10,FALSE
+8eKBstZUwA0YkRjpTduzxw==,20000,HtcR/HtKU0hM5hLuZcjjuA==,378.38,2021-12-10,40933,SNfyN9QHUE4npoS72Hd1Bw==,506993,/8yT7c/TpIKmxe0K8MmSNA==,0,378.38,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,2021-12-10,P,j13Eh8tAgyBPUNi2GaU8ow==,25,,SYjPoEyQZSe8RTfgN62BvA==,0,14000,14000,1,378.38,0,378.38,V-003017,Fik0+VTeuZo+2L4rH/Y5GA==,2021-12-10,2021-12-10,2021-12-10 12:05:54,2021-12-10,FALSE
+LbAdQij3k+nNd4i0+vLJLw==,20000,HtcR/HtKU0hM5hLuZcjjuA==,378.38,2021-12-10,40933,SNfyN9QHUE4npoS72Hd1Bw==,506993,/8yT7c/TpIKmxe0K8MmSNA==,0,378.38,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,2021-12-10,P,j13Eh8tAgyBPUNi2GaU8ow==,37,,SYjPoEyQZSe8RTfgN62BvA==,0,14000,14000,1,378.38,0,378.38,V-003017,Fik0+VTeuZo+2L4rH/Y5GA==,2021-12-10,2021-12-10,2021-12-10 12:05:54,2021-12-10,FALSE
+lztVVJ9OI+1P5nTsHkyJOg==,20000,HtcR/HtKU0hM5hLuZcjjuA==,378.38,2021-12-10,40933,SNfyN9QHUE4npoS72Hd1Bw==,506993,/8yT7c/TpIKmxe0K8MmSNA==,0,378.38,,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,1B2M2Y8AsgTpgAmY7PhCfg==,2021-12-10,P,j13Eh8tAgyBPUNi2GaU8ow==,49,,SYjPoEyQZSe8RTfgN62BvA==,0,14000,14000,1,378.38,0,378.38,V-003017,Fik0+VTeuZo+2L4rH/Y5GA==,2021-12-10,2021-12-10,2021-12-10 12:05:54,2021-12-10,TRUE

macros/get_gl_detail_columns.sql

@@ -45,7 +45,8 @@
     {"name": "whencreated", "datatype": "date"},
     {"name": "whendue", "datatype": "date"},
     {"name": "whenmodified", "datatype": dbt.type_timestamp()},
-    {"name": "whenpaid", "datatype": "date"}
+    {"name": "whenpaid", "datatype": "date"},
+    {"name": "_fivetran_deleted", "datatype": "boolean"}
 ] %}
 {{ return(columns) }}
 {% endmacro %}

models/stg_sage_intacct.yml

@@ -413,6 +413,10 @@ models:
         description: When transaction is modified
       - name: paid_at
         description: When transaction was paid
+      - name: _fivetran_deleted
+        description: The timestamp of when the data was deleted by Fivetran. Although the `_fivetran_deleted` column is no 
+          longer being actively synced in incremental runs in the connector in `gl_detail`, 
+          it can populate with `true/false` values in historical resyncs. 
 
   - name: stg_sage_intacct__gl_batch
     description: Records batch updates of records originating from the gl_detail table

models/stg_sage_intacct__gl_batch.sql

@@ -20,7 +20,7 @@ fields as (
 final as (
 
     select 
-        _fivetran_deleted,
+        _fivetran_deleted as is_batch_deleted,
         _fivetran_synced,
         baselocation as base_location,
         baselocation_no as base_location_no,

models/stg_sage_intacct__gl_detail.sql

@@ -69,7 +69,8 @@ final as (
         whencreated as created_at,
         whendue as due_at,
         whenmodified as modified_at,
-        whenpaid as paid_at
+        whenpaid as paid_at,
+        _fivetran_deleted as is_detail_deleted
 
 
         --The below script allows for pass through columns.