What Is A Reverse ETL - And Why Is The Modern Data Stack Obsessed With It?
Вставка
- Опубліковано 5 вер 2024
- There have been a lot of companies in the modern data stack adding in a reverse ETL component.
It makes a lot of sense as they try to create stickier products that manage more data flows.
But what is a reverse ETL?
And why do you care?
Also, special thanks to Rudderstack for sponsoring this video. If you want to read more about their Reverse ETL product check out the link below.
www.rudderstac...
If you enjoyed this video, check out some of my other top videos.
Top Courses To Become A Data Engineer In 2022
• Top Courses To Become ...
What Is The Modern Data Stack - Intro To Data Infrastructure Part 1
• What Is The Modern Dat...
If you would like to learn more about data engineering, then check out Googles GCP certificate
coursera.pxf.io/rn1rGv
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy...
Or check out my blog
www.theseattle...
And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy...
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.
I know you mentioned that this has been a "thing" for quite some time. But, it also has had a name. It was always called harmonization in the past. After data has been merged, cleaned, and enhanced it was common to harmonize it back to the source. Beside giving the source access to this enhanced data it would also allow for more efficient ETL as it would not need to be transformed over and over again. A lot of times these harmonization engines were tied to master data management and data stewardship. So, it was often a manual process to approve the harmonization. But, the process itself was automatic - once approved. I know that I am old but it is a bit shocking how much of modern data engineering is just reinventing the wheel.
Thanks a lot for this comments. As a Junior Data Engineer I found it quite useful to hear about "Harmonization".
What has changed over the past 5-10 years is the analysts having more control over their data stacks and being less reliant on data engineering to manage ETLs. With more modern tooling, the implementation of technology gets closer to the user, and will limit the investments needed to bring data closer to user actions with more impact.
Very, very informative. Also love these types of videos Ben.
Glad you enjoyed it!
Ok, so would an example of an R-ETL tool be something that takes in raw data, processes data with BI logic transformations, and then output something like a chart? so all you would see in the UI is basically you drop in data, and you can click a button that says "generate chart" ?
Not so much a chart. Thats more standard etl to data warehouse to reporting layer to dashboard. Instead, its more like, from data warehouse back to salesforce and add in something that makes it clear to target customer.
@@SeattleDataGuy any ETL tool could work in both directions
I was so confused when I first heard this term, only to discover I'd been doing this with various products for decades 🤣🤣🤣 honestly this is quite simply marketing genius, its a way for newer smaller products to "ok boomer" the mature products like Informatica that have done this for decades. A lot of the sales teams will not know what this term means and hence will get caught out in front of a customer when they ask "can you do reverse etl?". Its a bit like DBT used to push this story they were a no code when in fact dbt is all about code!
Hi Ben, one question, do I want to specialize in a cloud. Are there AWS, GCP, Azure... which one do you recommend to specialize as a Data Engineer?
Which one has better demand, tools, also in which one do you recommend me to get certified as a beginner.
AWS and Azure have the largest market share. I would say just pick one of the big three and you should be good
Thanks, very beneficial 👍
Glad it was helpful!
Great video as always. Noticed you sounded just every so slightly less energetic than usual, wondering if you were tired or just had a long day the same day of recording?
Maybe a little tired. I can't fully recall. I did film it the same day I did looker vs tableau
Instead of calling it “Reverse ETL” , WE should call it CRYSTALLIZATION
This just makes me think of crystal reports...
@@SeattleDataGuy I really admire your width to cover so much in such diverse DATA-STACK environment.
With that huge combinations of choices for different scenarios, I can see why you change to be a "Full Stack" DATA SOLUTION CONSULTANT. There is definitely a huge need from a spectrum of different organizations in different size, legacy background & their position in their data-legacy journey to be a fully data-driven organization.
Next time please don't use an analogy from a movie or TV series. Not all of us watch those
You must be fun at parties