End-to-End Data Pipeline Project with Change Data Capture Using Postgres, Airflow, and Snowflake!

[@KAshIf0o7]

Thanks George and Manmeet for this. Very well explained 🙌

[@thedataguygeorge]

Thank you, will pass the message along to Manmeet!

[@rasmusandreasson1548]

Niccce! BIG thumbs up for some EndToEnd-projects!

[@thedataguygeorge]

Thanks Rasmus, this one was for you!

[@rasmusandreasson1548]

@@thedataguygeorge really appreciate it!!

[@ManishJindalmanisism]

Hey.. Thanks for putting this. Lot of action. is this committed on github? This will be a learning project for people who are starters in airflow.
Additionally I have one question : Why have we hardcoded customer in get_dag_conf, should that not be dynamic basis your list of 3 tables?
Also in the tutorial, I think you didnt show customer.sql which you are using in extract task, and modifying to add where clause. Can you show how it looks like?

[@thedataguygeorge]

It sure is! github.com/manmeetkaur/etl-airflow/tree/main
Can see the customer.sql file there, and in practice you wouldn't want to hardcode, I just did that for ease of use for this example

[@ManishJindalmanisism]

@@thedataguygeorge Thanks for replying. Very kind of you

[@StephenRayner]

Why snowflake for OLAP? Postgres, airflow, dbt are all open source (and cosmos). But snowflake isn't. I don't get why everyone uses it. Why not an open source alternative? What about Clickhouse?

[@thedataguygeorge]

Snowflake requires almost 0 set up or management expertise which is a big value add for a lot of people