`run_results.json` on DAG level

[dinjazelena98]

Hi guys, so i have Airflow with source ingestion DAG which triggers DBT models with Astronomer Cosmos. Finally, at the end i want to push all of the metadata into OpenMetadata. Currently after source ingestion is done, it updates assets, and dbt models are triggered on that asset. Once the dbt models are done, my DAG for updating metadata is triggered. I am writing conf files for both dbt and our warehouse (Clickhouse) and then i just use BashOperator where i use metadata-ingest CLI to trigger push of metadata to OpenMetadata.

Now, my problem is that Cosmos parses my dbt by running dbt ls, and it creates a task per each model which just runs isolated dbt run . On each dbt run it generates run_results.json , which OpenMetadata uses to parse some additional metadata, do u have experience how can i do that on whole dag level not per task per model which is how is happening in Cosmos, because each model/task is separate dbt run command.

[pankajkoti]

hi @dinjazelena98 . Would you like to have tasks per model, or is it okay to have just one Airflow task for your dbt pipeline? If the latter is okay, instead of using DbtDag (or DbtTaskgroup), you could use a single Cosmos operator like DbtRunLocalOperator as shown in the example DAG at

astronomer-cosmos/dev/dags/example_operators.py

Line 70 in a225515

run_operator = DbtRunLocalOperator(

Additionally, you can use a callback function for the operators that will allow you to process the run_results.json and using your custom Python function can forward it to OpenMetadata (e.g. as shown in

astronomer-cosmos/dev/dags/example_operators.py

Line 46 in a225515

callback=upload_to_aws_s3,

)

[dinjazelena98]

Well, i would like to have task per model, it helps with debugging and allows us to play with parallelism, it has an amazing view as well. But it also generate run_results.json for each of the models, so i think i need extra step to merge all run_results.json in order to push it OpenMetadata.