Fix Your Data Pipeline...From The Start

[@KahanDataSolutions]

Looking for help with your team's data strategy? → www.kahandatasolutions.com
Looking to improve your data engineering skillset?→ bit.ly/more-kds

[@adityamathur2284]

We don't use a different database for landing zone (with naming convention raw_) rather we use separate schema with raw_ naming convention and put the source tables there.

[@s_m_exist]

Hello, I have a question here on using dbt Cloud. If fivetran is injecting data to only one env, say Dev, Is there a way to maintain single codebase for dev, staging and prod by pointing source at dev raw datasets while staging raw and prod raw are views created on top of it? Each env transformation is based on their respective raw layers (but raw of staging and prod are just views on top of dev raw)

[@Tepeee]

then what's the difference between the "raw" db and staging? technically we can use staging db as a landing zone right?

[@k0sumkind356]

Raw Zone is where the raw data is stored. It is the target of your EL tool like Fivetran. The staging zone is importing from the raw zone and minor changes are applied. That can be column names, date format but no joints.

[@KahanDataSolutions]

Here's another video that breaks down the Staging Layer - ua-cam.com/video/z_TrtkJKXoA/v-deo.htmlsi=im91zuze1V1Zs5Tw

[@Milhouse77BS]

Yes, our "raw" data from AS400/iSeries comes in with Integers as Decimal 38.8

[@altaneryilmaz3356]

Pfff anything new?