Data Engineering with the Open Source Modern Data Stack (From MDS Fest '23)

The Modern Data Stack gives us a wide range of free technology for data ingestion, data storage, data transformation, data orchestration, and data visualization.
Pedram Navid, Head of Data Engineering and DevRel at Dagster, walks us through one data pipeline design using a number of open-source solutions, including Dagster, dbt, PopSQL, and the Mastodon API, DuckDB, dbt-duckdbt, Evidence, Sling and Steampipe.
The pipeline is used to analyze bird observation data from the Cornell Lab.
Github Repo: github.com/dagster-io/mdsfest...
Source Data: feederwatch.org/explore/raw-d...
Try Dagster today with a 30-day free trial: dagster.io/lp/dagster-cloud-t...

[@Robay146]

Great presentation.

[@VaibhavPatil-rx7pc]

Excellent information thanks

[@maxisqt]

This is the coolest tech demo I've ever seen. I have wanted for so long to see an end-to-end analytics stack demo, or tutorial, and never found it. You just did it in 15 minutes, using free, open source tools I can run locally on my laptop. Absolutely incredible!

[@dagsterio]

Thanks. The Dagster capabilities are expanding with each new release.

[@gauravlotekar660]

Excellent demo

[@pnavid]

Great job❤

[@simondelorean]

Awesome, you just saved me a ton of work. This tool set looks great. One thing I noticed: I had to turn up the volume quite a bit.

[@allthatyouare]

Trying this out! This was a great overview. Thank you!!

[@alexzir]

Awesome!