feat: add columns lineage to Dataform dependency graph

[Cir02]

hi,

awsoome extesion for work with dataform. it should be posible to add column level lineage inside the Dataform dependency graph?

thx!

[ashish10alex]

Hi @Cir02 , thank you for the kind words of appreciation 🙂

Column level lineage will be a good thing to have. However, I don't think it will be a trivial thing to implement something that is useable. Also, the examples of other implementations I have seen quickly get out of hand when the graph gets complicated which is often the case in real world data products. I will do some research on existing implementation types and how feasible it would be to implement. Meanwhile if you have any specific ideas please feel free to put them here. Also, if you would like to contribute, I would suggest checkout out CONTRIBUTING.md

[Cir02]

Hi, thank you for answering. The idea is to have something similar to dbt out of the box or open metada, which can get data column lineage from bigquery querys using data plex/data lineage on gcp. i will check also others examples. thanks!