- 70
- 208 343
Techno Devs with Saurabh
India
Приєднався 6 бер 2011
This channel is devoted to helping you to learning about database, python and cloud concepts.
My name is Saurabh and I have over 10 years of experience working in tech as a professional software engineer.
I make educational videos.
If you want to join me, hit that subscribe button now.
My name is Saurabh and I have over 10 years of experience working in tech as a professional software engineer.
I make educational videos.
If you want to join me, hit that subscribe button now.
Databricks Run Pyspark Code and Schedule a Job
Databricks Run Pyspark Code and Schedule a Job
Переглядів: 99
Відео
Spark Architecture |Spark Tutorial |Pyspark (Hindi)
Переглядів 13221 день тому
In thi video , I have covered below topics in detail. About Spark Spark architecture Spark session Spark Config Calculate Executor Spark Memory Management Spark cluster manager
Apache Spark Installation in Windows local | Run pyspark
Переглядів 9021 день тому
In this video on Spark installation you will learn below topics how to download Apache Spark on Windows. how to download java and python how to set class path for java ,python and spark how to run pyspark code in window.
Creating a free Databricks Account
Переглядів 11521 день тому
In this video, I have covered below topics. Creating a free Databricks Account Creating a workspace with AWS Using clouformation template to create workspace
Databricks Tutorial with AWS
Переглядів 13021 день тому
In this video cover below topics in details. Workspace Metastore Unity Catalog Managed Table (internal table) Create Dataframe using s3 Read and write data into s3 usimg spark Read data from DBFS FileStore Create External location Create Catalog External Table Mount S3 Bucket Delta Lake conecpt in detail
Running EMR jobs with Airflow| Create EMR cluster and Submit a job on EMR using AWS MWAA (Part3)
Переглядів 1,4 тис.6 місяців тому
create emr cluster and submit pyspark job with Amazon Managed Workflows for Apache Airflow. GitHub link for code github.com/saurabhgarg013/Airflowcode.git
AWS Airflow Demo| Python code |AWS MWAA SETUP (part2)
Переглядів 2,4 тис.6 місяців тому
In this video, we have covered below topics. Create Amazon Managed Workflows for Apache Airflow. Create DAG python code Run python code by scheduling job using Airflow understand airflow environment GitHub link for code github.com/saurabhgarg013/Airflowcode.git
Apache Airflow Tutorial in Hindi | AWS Airflow (part1)
Переглядів 7 тис.6 місяців тому
Amazon Managed Workflows for Apache Airflow (MWAA). In this video , you will learn below topics. I will create another video for demo. About Apache airflow and AWS managed airflow . Airflow use DAG Operators dag and operator example AIRFLOW COMPONENT Amazon MWAA environment Amazon MWAA execution role EMR job by Airflow Airflow cost Below video link for demo lab for airflow ua-cam.com/video/6apd...
AWS RDS PostgreSQL Database Tutorial with Demo
Переглядів 6817 місяців тому
In this video , I have covered below topics. About PostgreSQL and its Features PostgreSQL vs MySQL AWS RDS Aurora PostgreSQL vs AWS RDS PostgreSQL Create AWS PostgreSQL database Connect with PostgreSQL using cmd and pgAdmin Create Database, User , Schema and Tables Used Datatype like json, uuid and array. insert and create table with json and uuid data type Execute create table and insert scrip...
Create EMR cluster and add steps | Submit job
Переглядів 2 тис.7 місяців тому
In this video covered below topics: 1.How to submit EMR job in cluster mode. 2.Add a Spark step - Amazon EMR 3.Run PySpark code in EMR master node terminal Learn how to adding steps to a cluster with the help of master node and cli
Amazon EMR Studio | Launch Jupyter notebooks
Переглядів 1,1 тис.7 місяців тому
Amazon EMR Studio is a web-based integrated development environment (IDE) for fully managed Jupyter notebooks that run on Amazon EMR clusters. In this video I have cover below topics 1.create EMR cluster 2.Create EMR Studio 3.Create a workspace 4.Run Pyspark code Use Pyspark with a Jupyter Notebook in an AWS EMR. Amazon EMR Studio - Creating a new Studio Workspace
AWS Glue Catalog With AWS EMR | Spark SQL in EMR
Переглядів 1,9 тис.9 місяців тому
Use the AWS Glue Data Catalog as the metastore for Spark SQL Amazon EMR access to AWS Glue Data Catalog
Launch Jupyter Notebook With AWS EMR| Run Pyspark Code
Переглядів 2 тис.9 місяців тому
In this video, You get to know how to setup Jupyter Notebook and run Pyspark code Jupyter Notebook (formerly IPython) is one of the most popular user interfaces for running Python Use JupyterHub on Amazon EMR to host multiple instances of a single-user Jupyter notebook server for multiple users.
AWS Glue Studio Join and Custom transformation | Joining Datasets in AWS Glue ETL Job
Переглядів 2 тис.Рік тому
In this video , I explain how o join two datasets and also apply custom transformation in AWS Glue. AWS Glue Studio which joins two datasets, transforms the joined dataset and than apply custom transformation and then put down data into destination bucket. AWS Glue Studio is a graphical interface that makes it easy to create, run, and monitor extract, transform, and load (ETL) jobs in AWS Glue....
AWS Kinesis theory with demo| ETL Hands on demo using lambda and Kinesis
Переглядів 4 тис.Рік тому
AWS - Kinesis Data Stream and data firehose practical lab with theory. Topics that we will cover in this video are as follows: what is Amazon Kinesis Data Streams? what is Amazon kinesis Data firehose? diff between Data Streams vs Data firehose Shards and partition key concept Demo ETL using Aws lambda with Kinesis data stream Demo ETL using Kinesis data stream and data firehose · Request you t...
AWS Code Pipeline with Elastic Beanstalk |CICD AWS Code pipeline
Переглядів 673Рік тому
AWS Code Pipeline with Elastic Beanstalk |CICD AWS Code pipeline
AWS Elastic Beanstalk Tutorial with Demo
Переглядів 4,6 тис.Рік тому
AWS Elastic Beanstalk Tutorial with Demo
AWS CodePipeline |Deploy Glue ETL script using CI/CD Pipeline on AWS
Переглядів 5 тис.Рік тому
AWS CodePipeline |Deploy Glue ETL script using CI/CD Pipeline on AWS
Create Redshift Cluster and Load Data using Python| Connect redshift using Python
Переглядів 3 тис.Рік тому
Create Redshift Cluster and Load Data using Python| Connect redshift using Python
Building ETL Pipeline using AWS Glue and Step Functions
Переглядів 5 тис.Рік тому
Building ETL Pipeline using AWS Glue and Step Functions
Create Amazon Elastic Container Service (ECS) with a Load Balancer Demo | Step by Step
Переглядів 1,6 тис.Рік тому
Create Amazon Elastic Container Service (ECS) with a Load Balancer Demo | Step by Step
AWS ECR| Create Docker Image and AWS push docker image to ECR|AWS ECS
Переглядів 1,4 тис.Рік тому
AWS ECR| Create Docker Image and AWS push docker image to ECR|AWS ECS
AWS ECS Fargate Demo for Beginner | Deploying Containers on ECS Fargate
Переглядів 3,6 тис.Рік тому
AWS ECS Fargate Demo for Beginner | Deploying Containers on ECS Fargate
Complete Docker Tutorial with Demo |Docker installation| Deploy image
Переглядів 1,2 тис.Рік тому
Complete Docker Tutorial with Demo |Docker installation| Deploy image
Create AWS EMR Cluster Using AWS CLI and Submit job
Переглядів 1,6 тис.Рік тому
Create AWS EMR Cluster Using AWS CLI and Submit job
DIFF AWS GLUE VS EMR| AWS EMR VS AWS Glue
Переглядів 1,5 тис.Рік тому
DIFF AWS GLUE VS EMR| AWS EMR VS AWS Glue
AWS EMR Tutorial | Amazon EMR Architecture
Переглядів 6 тис.Рік тому
AWS EMR Tutorial | Amazon EMR Architecture
AWS EMR Cluster Create using AWS Console | Submitting Spark Jobs in AWS EMR Cluster
Переглядів 15 тис.Рік тому
AWS EMR Cluster Create using AWS Console | Submitting Spark Jobs in AWS EMR Cluster
AWS Ec2 VS Amazon Lambda|Whats the difference in 5 minute
Переглядів 694Рік тому
AWS Ec2 VS Amazon Lambda|Whats the difference in 5 minute
What is AWS Lambda and AWS Glue? |Difference between AWS Glue and AWS lambda explained in 5 mins
Переглядів 2,3 тис.Рік тому
What is AWS Lambda and AWS Glue? |Difference between AWS Glue and AWS lambda explained in 5 mins
Very usefull video ❤
Do you have any videos for sparks development?
We’ll explain
Hi Sir.. can you provide 6 pillars of AWS well-architected Framework… video
please follow him guys
your teach very well
Bhai ❤ thank you so much you have explained so nicely even a noobie can understand and work on emr.....
This is a great tutorial. Absolutely loved it
make a playlit on the real world project on spark
Great Video!! If followed word-to-word, it will definitely work. Thanks!!
your explanation is simple and easy to understand. aur kya chahiye. Just keep giving us these great stuff.
Thank you bhaiya,much needed for data engineer job
Hi Saurabh, In my case after second commit the deploy stage(using elastic beanstalk) fails in codepipeline. Can you help me regarding this?
sir ye playlist me aap pura spark sikhaoge ?
Sir, i watch your videos and it's always a great learning experience. I would recommend deleting the last part of the video (1.06.00).
Noted
Hello, i followed your vedio, i got this error, "although the task says it's queued. (Info: None) Was the task killed externally?", could you help me out please
Super good content saurabh, you explain really nice, I would love to see more videos on Data Engineering tools and technologies :)
Mast Vdeo
Nice
hello sir....i did not understand one thing that...once we read the data from s3 and convert it into parquet format....then, our requirement is to insert the data into redshift then, why we need to create another table in AWS data catalog sir ?
provide the code
Excellent explanation.want more videos
Why are we crawling redshift when it is empty?
Excellent explanation
🔥🔥🔥thank you so much. can you please provide the ppt🙂
lambda is a big pain
can we practice glue in free tier or it will be chargable?
Bro great Job. BC just 131 likes on this video, Yaha kisi ne apni ek baar G##D hila di hoti to 1 million like aa jate. In Bharat there is no value of science, hardwork and technical work.
Do we need to create 10 code pipeline for 10 glue jobs..can we use same pipeline for 10 glue jobs deployed on different times
Great work
In AWS Redshift cluster, what is zero ETL and how does it work, sir?
In AWS Redshift cluster, what is zero ETL and how does it work, sir?
Why need two crawler, can't we directly read parquet and load to Redshift
Bro love you videos.. especially long form videos where you explain each and every option or a feature of a service
Hello sir, video is too good. I have cleared lots of things from this video. 🥳🎉👌👌
Thank you so much
Nicely done Sir. Thank you.
Nice one Sir. Thank you.
Hi sir, could you please share the slides that you used in this video?
Very Good
This is the best video which cleared my doubts and i completely understood the concepts.
excellent
Very good explanation
Great work, Saurabh!
Sir bohut badiya smjhaate ho par pls gala thoda saaf krke videos banaye kabhi kabhi irritating ho jata hai ye
Well streamlined video and content.!
Very informative 🎉🎉 Thank you
Thankyou so much....with your help I got role in jpmc... however I am scared as I don't belongs to coding background ...how I will survive as data engineer
Sir you are great and No. 1
Excellent video . Can not thank you enough for giving me the confidence of mastering redshift foundation from both your redshift theory and this practical tutorial.