Data Engineering Pipeline / ETL Process Task & Source Audit Basics using Python - Design Walkthrough

Data Engineering / ETL / ELT
ETL - Extract, Transform, and Load
ELT - Extract, Load, and Transform
In this video, I have explained how to write the code for Source Data Audit and Task Audit using Python.
In the previous video, I explained the Data Engineering / ETL Concepts • Data Engineering ETL V... .
The following essential concepts have been covered in the previous video:
Incremental Extract vs Full Extract
How to design the Incremental Extract
Source and Task Audit Table
Staging Area and its importance
Data Warehouse / Data Lake
Data Mart
Data Transformation and Aggregation
Defining the granularity of the data storage
Hierarchies
I have also covered how we can use Python for ETL or Data Integration tools like Pentaho Data Integration (PDI) / SQL Server Services for ETL at a conceptual level.
I hope this video would serve as a good starting point for anyone wanting to understand the Data Engineering / ETC Concepts.
Website: k2analytics.co.in
Email: [e-mail]
Mobile: [phone]

[@kamalgopalsingh244]

Great content

[@hasanmougharbel8030]

Hey dear, god bless your efforts in this channel.
I have a general enquiry as a new sql learner.
How could i create a pipeline to extract and load data from existing accounting program into our SQL server instances.
How can i know if the export mechanism in the software permits me to undertake this extraction process, and how can i know if an application have an api?
Thanks for taking care of my enquires.
Looking forward to gain more knowledge from you

[@RajeshJakhotiaAIML]

You will have to use the API of accounting software to extract data.

[@Velben]

I always extract, transform and create the staging tables from within the python script. Is there a performance benefit in having the tables created prior?

[@ksspqf6016]

Part of my degree was data analytics but didn't cover stuff like this whatsoever. I didn't know what etl, data modelling excels power query nor powerbi was when I left university. I feel ripped off as when I can search for a video online and it will teach me a semester's worth of content in a single video

[@sksarifulislamtech]

can u plz provide the code