- 21
- 1 596
Data Eng Safari
Приєднався 20 січ 2024
PROJECT 4 - AWS End-To-End Data Engineering in 15 min | EC2 - Snowflake - Airflow - VS Code
In this video, you'll learn as simply as possible what AWS EC2 is and carry out your fourth project. You will combine AWS EC2, Snowflake, Airflow, VS Code to create a simple data pipeline.
⚠️ Linux CLI commands
sudo apt update
sudo apt install python3-pip
sudo apt install python3-venv
python3 -m venv airflow_snow_venv
source airflow_snow_venv/bin/activate
pip install apache-airflow
pip install apache-airflow-providers-snowflake==2.1.0
pip install snowflake-connector-python
pip install snowflake-sqlalchemy
pip uninstall pydantic==2.0
pip install pydantic==1.10.10
airflow standalone
⚠️ Linux CLI commands
sudo apt update
sudo apt install python3-pip
sudo apt install python3-venv
python3 -m venv airflow_snow_venv
source airflow_snow_venv/bin/activate
pip install apache-airflow
pip install apache-airflow-providers-snowflake==2.1.0
pip install snowflake-connector-python
pip install snowflake-sqlalchemy
pip uninstall pydantic==2.0
pip install pydantic==1.10.10
airflow standalone
Переглядів: 21
Відео
PROJECT 3 - AWS End-To-End Data Engineering in 45 min | EC2 - Postgres RDS - S3 - Airflow - VS Code
Переглядів 449 годин тому
In this video, you'll learn as simply as possible what AWS EC2 is and carry out your third project. You will combine AWS EC2, S3, Postgres RDS, Airflow, VS Code to create a parallel data pipeline to retrieve data, process it and store it in S3 and Postgres RDS. ⚠️ LINK TO the us_city.csv and python code github.com/mdesafari/AWS_Series/tree/main/001_Postgres1 ⚠️ LINK TO Postgres CLI and AWS S3 c...
PROJECT 2 - AWS End-To-End Data Engineering in 30 min | EC2 - S3 - Airflow - VS Code
Переглядів 62День тому
In this video, you'll learn as simply as possible what AWS EC2 is and carry out your second project. You will combine AWS EC2, S3, Airflow, VS Code to create a data pipeline to retrieve data, process it and store it in S3. #azure #dataengineering #databricks #ec2instance #aws #airflow
Project 1 to introduce AWS EC2 | Cloud Data Eng Projects (AWS Series)
Переглядів 20День тому
In this video, you'll learn as simply as possible what AWS EC2 is and carry out your first small project to familiarize yourself with the essentials of how AWS EC2 works. #azure #dataengineering #databricks #ec2instance #aws
Part 2 - Azure End-To-End Data Engineering in 10 min | Project Olympic Data Analytics
Переглядів 5814 днів тому
In this video, you will query your data (from the part 1 of this serie) using Azure Synapse Analytics. #azure #dataengineering #databricks
Part 1 - Azure End-To-End Data Engineering in 30 min | Project Olympic Data Analytics
Переглядів 26614 днів тому
In this video, you will analyze Olympic data using various tools and technologies, including Azure Data Factory, Data Lake Gen 2, Synapse Analytics, and Azure Databricks. #azure #dataengineering #databricks
#Airflow 3: Build a complete data pipeline with Apache Airflow and store in a PostgreSQL Database
Переглядів 489 місяців тому
Build a complete data pipeline with Apache Airflow and store in a PostgreSQL Database Link to all codes and the Jupyter notebook: github.com/mdesafari/ETL/tree/main/002-ETL-Orchestration-Apache-Airflow #airflow #dataengineering #dataengineer #ai #machinelearning #pipeline #datapipeline
#Airflow 2: Build a complete data pipeline with Apache Airflow - The weather data problem
Переглядів 769 місяців тому
Build a complete data pipeline with Apache Airflow - The weather data problem Link to all codes (dockerfile, docker-compose.yml, weather_dag.py, transform_load_weather_funcs.py) and the Jupyter notebook: github.com/mdesafari/ETL/tree/main/002-ETL-Orchestration-Apache-Airflow #airflow #dataengineering #dataengineer #ai #machinelearning #pipeline #datapipeline
#Airflow 1: How to install Docker Desktop for Windows + Complete configuration + Connect to VS Code
Переглядів 779 місяців тому
How to install Docker Desktop for Windows Complete configuration Connect to VS Code #docker #dataengineering #ai #machinelearning #howtoinstall #how #howto
How to install Power BI Desktop + Example of ETL and visualization processes
Переглядів 189 місяців тому
How to install Power BI Desktop Example of ETL (Extract Transform Load) and visualization processes Link to data: github.com/mdesafari/PowerBI/tree/main/Data #powerbi #etl #dataengineering #dataengineer #machinelearning
Learn some data formats for data engineering
Переглядів 139 місяців тому
Learn some data formats for data engineering. I'll be updating regularly to add more. Link to the Data format Repository: github.com/mdesafari/DataFormat #dataengineering #dataformat #machinelearning #apache #hadoop
Learn Data Structures and Algorithms with Python
Переглядів 219 місяців тому
Learn Data Structures and Algorithms with Python Link to the repository DSA: github.com/mdesafari/DSA-Python #python #datastructures #algorithm #vscode #dataengineering #machinelearning #ai
PostgreSQL #2 Learn PostgreSQL in 1 HOUR with VS Code and CMD - No talking
Переглядів 7410 місяців тому
Learn PostgreSQL in 1 hour with VS Code and CMD - Tutorial Windows 11 & No talking Link to data: see in the first comment. #postgresql #dataengineering #vscode #mysql #machinelearning #ai #cmd
PostgreSQL #1 How to install PostgreSQL, connect it to VSCode and build your 1st PostgreSQL project
Переглядів 21710 місяців тому
How to install PostgreSQL, connect it to VSCode and build your first PostgreSQL project on Windows #postgresql #installation #database #dataengineering
Install Python, connect it to VS Code, create a virtual environment in VS Code
Переглядів 4810 місяців тому
Install Python, connect it to VS Code, create a virtual environment in VS Code #python #dataengineering #datastructures #algorithm #machinelearning #vscode #virtualenvironment
Complete MongoDB Tutorial #2 - Learn MongoDB in 1 Hour 30 min - No Talking
Переглядів 12010 місяців тому
Complete MongoDB Tutorial #2 - Learn MongoDB in 1 Hour 30 min - No Talking
Complete MongoDB Tutorial #1 - Download and Install MongoDB, connect it to VS Code
Переглядів 22310 місяців тому
Complete MongoDB Tutorial #1 - Download and Install MongoDB, connect it to VS Code
Complete MySQL project on VS Code in 1 Hour - No Talking
Переглядів 4011 місяців тому
Complete MySQL project on VS Code in 1 Hour - No Talking
Connect to MySQL from Visual Studio Code and Run SQL Queries (French voice / English Subtitles)
Переглядів 2311 місяців тому
Connect to MySQL from Visual Studio Code and Run SQL Queries (French voice / English Subtitles)
How to download and install the latest version of MySQL 8.0.36 Server and Workbench on Windows 11
Переглядів 8011 місяців тому
How to download and install the latest version of MySQL 8.0.36 Server and Workbench on Windows 11
Comment installer la dernière version de MySQL 8.0.36 Server et Workbench sur Windows 11
Переглядів 5211 місяців тому
Comment installer la dernière version de MySQL 8.0.36 Server et Workbench sur Windows 11
Thanks for sharing!
You're welcome! I'm glad you found it helpful!
Binance red Packet = BPLHLNBS3D
Link to download the dataset (GitHub): github.com/mdesafari/AzureProject/tree/main/dataset
Link to all codes (dockerfile, docker-compose.yml, weather_dag. py, transform_load_weather_funcs.py) and the Jupyter notebook: github.com/mdesafari/ETL/tree/main/002-ETL-Orchestration-Apache-Airflow
i am not getting edit option while being in environmental variable (path)..please help
Sorry to answer a little too late, but if you can't find the "Edit" button, you have several options: 1) double-click on "path" and once in "Edit environment variable", also double-click on an empty line to paste the path. 2) Consider opening the "Edit environment variable" window with administrator rights and follow the steps in option 1. Don't hesitate if you have any concerns.
If you have any questions or suggestions, don't hesitate to post them as comments. I'll do my best to answer them.
really helpfull, kepp going
Thank you and I'm glad it helped.
Link to data: github.com/mdesafari/PostgreSQL/tree/main/data
Link to data: github.com/mdesafari/MongoDB/blob/main/001-Data/data.mongodb Link to codes: github.com/mdesafari/MongoDB Link to cheat sheet: github.com/mdesafari/MongoDB/blob/main/readme.md
CSV Files: github.com/mdesafari/MySQL/tree/main/000-Data All codes are downloadable: github.com/mdesafari/MySQL