Techno Devs with Saurabh
Techno Devs with Saurabh
  • 70
  • 208 343

Відео

Spark Architecture |Spark Tutorial |Pyspark (Hindi)
Переглядів 13221 день тому
In thi video , I have covered below topics in detail. About Spark Spark architecture Spark session Spark Config Calculate Executor Spark Memory Management Spark cluster manager
Apache Spark Installation in Windows local | Run pyspark
Переглядів 9021 день тому
In this video on Spark installation you will learn below topics how to download Apache Spark on Windows. how to download java and python how to set class path for java ,python and spark how to run pyspark code in window.
Creating a free Databricks Account
Переглядів 11521 день тому
In this video, I have covered below topics. Creating a free Databricks Account Creating a workspace with AWS Using clouformation template to create workspace
Databricks Tutorial with AWS
Переглядів 13021 день тому
In this video cover below topics in details. Workspace Metastore Unity Catalog Managed Table (internal table) Create Dataframe using s3 Read and write data into s3 usimg spark Read data from DBFS FileStore Create External location Create Catalog External Table Mount S3 Bucket Delta Lake conecpt in detail
Running EMR jobs with Airflow| Create EMR cluster and Submit a job on EMR using AWS MWAA (Part3)
Переглядів 1,4 тис.6 місяців тому
create emr cluster and submit pyspark job with Amazon Managed Workflows for Apache Airflow. GitHub link for code github.com/saurabhgarg013/Airflowcode.git
AWS Airflow Demo| Python code |AWS MWAA SETUP (part2)
Переглядів 2,4 тис.6 місяців тому
In this video, we have covered below topics. Create Amazon Managed Workflows for Apache Airflow. Create DAG python code Run python code by scheduling job using Airflow understand airflow environment GitHub link for code github.com/saurabhgarg013/Airflowcode.git
Apache Airflow Tutorial in Hindi | AWS Airflow (part1)
Переглядів 7 тис.6 місяців тому
Amazon Managed Workflows for Apache Airflow (MWAA). In this video , you will learn below topics. I will create another video for demo. About Apache airflow and AWS managed airflow . Airflow use DAG Operators dag and operator example AIRFLOW COMPONENT Amazon MWAA environment Amazon MWAA execution role EMR job by Airflow Airflow cost Below video link for demo lab for airflow ua-cam.com/video/6apd...
AWS RDS PostgreSQL Database Tutorial with Demo
Переглядів 6817 місяців тому
In this video , I have covered below topics. About PostgreSQL and its Features PostgreSQL vs MySQL AWS RDS Aurora PostgreSQL vs AWS RDS PostgreSQL Create AWS PostgreSQL database Connect with PostgreSQL using cmd and pgAdmin Create Database, User , Schema and Tables Used Datatype like json, uuid and array. insert and create table with json and uuid data type Execute create table and insert scrip...
Create EMR cluster and add steps | Submit job
Переглядів 2 тис.7 місяців тому
In this video covered below topics: 1.How to submit EMR job in cluster mode. 2.Add a Spark step - Amazon EMR 3.Run PySpark code in EMR master node terminal Learn how to adding steps to a cluster with the help of master node and cli
Amazon EMR Studio | Launch Jupyter notebooks
Переглядів 1,1 тис.7 місяців тому
Amazon EMR Studio is a web-based integrated development environment (IDE) for fully managed Jupyter notebooks that run on Amazon EMR clusters. In this video I have cover below topics 1.create EMR cluster 2.Create EMR Studio 3.Create a workspace 4.Run Pyspark code Use Pyspark with a Jupyter Notebook in an AWS EMR. Amazon EMR Studio - Creating a new Studio Workspace
AWS Glue Catalog With AWS EMR | Spark SQL in EMR
Переглядів 1,9 тис.9 місяців тому
Use the AWS Glue Data Catalog as the metastore for Spark SQL Amazon EMR access to AWS Glue Data Catalog
Launch Jupyter Notebook With AWS EMR| Run Pyspark Code
Переглядів 2 тис.9 місяців тому
In this video, You get to know how to setup Jupyter Notebook and run Pyspark code Jupyter Notebook (formerly IPython) is one of the most popular user interfaces for running Python Use JupyterHub on Amazon EMR to host multiple instances of a single-user Jupyter notebook server for multiple users.
AWS Glue Studio Join and Custom transformation | Joining Datasets in AWS Glue ETL Job
Переглядів 2 тис.Рік тому
In this video , I explain how o join two datasets and also apply custom transformation in AWS Glue. AWS Glue Studio which joins two datasets, transforms the joined dataset and than apply custom transformation and then put down data into destination bucket. AWS Glue Studio is a graphical interface that makes it easy to create, run, and monitor extract, transform, and load (ETL) jobs in AWS Glue....
AWS Kinesis theory with demo| ETL Hands on demo using lambda and Kinesis
Переглядів 4 тис.Рік тому
AWS - Kinesis Data Stream and data firehose practical lab with theory. Topics that we will cover in this video are as follows: what is Amazon Kinesis Data Streams? what is Amazon kinesis Data firehose? diff between Data Streams vs Data firehose Shards and partition key concept Demo ETL using Aws lambda with Kinesis data stream Demo ETL using Kinesis data stream and data firehose · Request you t...
AWS Code Pipeline with Elastic Beanstalk |CICD AWS Code pipeline
Переглядів 673Рік тому
AWS Code Pipeline with Elastic Beanstalk |CICD AWS Code pipeline
AWS Elastic Beanstalk Tutorial with Demo
Переглядів 4,6 тис.Рік тому
AWS Elastic Beanstalk Tutorial with Demo
AWS CodePipeline |Deploy Glue ETL script using CI/CD Pipeline on AWS
Переглядів 5 тис.Рік тому
AWS CodePipeline |Deploy Glue ETL script using CI/CD Pipeline on AWS
Create Redshift Cluster and Load Data using Python| Connect redshift using Python
Переглядів 3 тис.Рік тому
Create Redshift Cluster and Load Data using Python| Connect redshift using Python
Building ETL Pipeline using AWS Glue and Step Functions
Переглядів 5 тис.Рік тому
Building ETL Pipeline using AWS Glue and Step Functions
Create Amazon Elastic Container Service (ECS) with a Load Balancer Demo | Step by Step
Переглядів 1,6 тис.Рік тому
Create Amazon Elastic Container Service (ECS) with a Load Balancer Demo | Step by Step
AWS ECR| Create Docker Image and AWS push docker image to ECR|AWS ECS
Переглядів 1,4 тис.Рік тому
AWS ECR| Create Docker Image and AWS push docker image to ECR|AWS ECS
AWS ECS Fargate Demo for Beginner | Deploying Containers on ECS Fargate
Переглядів 3,6 тис.Рік тому
AWS ECS Fargate Demo for Beginner | Deploying Containers on ECS Fargate
Complete Docker Tutorial with Demo |Docker installation| Deploy image
Переглядів 1,2 тис.Рік тому
Complete Docker Tutorial with Demo |Docker installation| Deploy image
Create AWS EMR Cluster Using AWS CLI and Submit job
Переглядів 1,6 тис.Рік тому
Create AWS EMR Cluster Using AWS CLI and Submit job
DIFF AWS GLUE VS EMR| AWS EMR VS AWS Glue
Переглядів 1,5 тис.Рік тому
DIFF AWS GLUE VS EMR| AWS EMR VS AWS Glue
AWS EMR Tutorial | Amazon EMR Architecture
Переглядів 6 тис.Рік тому
AWS EMR Tutorial | Amazon EMR Architecture
AWS EMR Cluster Create using AWS Console | Submitting Spark Jobs in AWS EMR Cluster
Переглядів 15 тис.Рік тому
AWS EMR Cluster Create using AWS Console | Submitting Spark Jobs in AWS EMR Cluster
AWS Ec2 VS Amazon Lambda|Whats the difference in 5 minute
Переглядів 694Рік тому
AWS Ec2 VS Amazon Lambda|Whats the difference in 5 minute
What is AWS Lambda and AWS Glue? |Difference between AWS Glue and AWS lambda explained in 5 mins
Переглядів 2,3 тис.Рік тому
What is AWS Lambda and AWS Glue? |Difference between AWS Glue and AWS lambda explained in 5 mins

КОМЕНТАРІ

  • @isharkpraveen
    @isharkpraveen 3 години тому

    Very usefull video ❤

  • @raviyadav-dt1tb
    @raviyadav-dt1tb 5 днів тому

    Do you have any videos for sparks development?

  • @raviyadav-dt1tb
    @raviyadav-dt1tb 5 днів тому

    We’ll explain

  • @amitjoshi8621
    @amitjoshi8621 5 днів тому

    Hi Sir.. can you provide 6 pillars of AWS well-architected Framework… video

  • @ShubhamAlsunde
    @ShubhamAlsunde 8 днів тому

    please follow him guys

  • @ShubhamAlsunde
    @ShubhamAlsunde 8 днів тому

    your teach very well

  • @WolfmaninKannada
    @WolfmaninKannada 11 днів тому

    Bhai ❤ thank you so much you have explained so nicely even a noobie can understand and work on emr.....

  • @prasannaakolka
    @prasannaakolka 12 днів тому

    This is a great tutorial. Absolutely loved it

  • @abhishekn2048
    @abhishekn2048 15 днів тому

    make a playlit on the real world project on spark

  • @lakshya-q2q
    @lakshya-q2q 21 день тому

    Great Video!! If followed word-to-word, it will definitely work. Thanks!!

  • @tinkuchanchal1238
    @tinkuchanchal1238 21 день тому

    your explanation is simple and easy to understand. aur kya chahiye. Just keep giving us these great stuff.

  • @jaat9039
    @jaat9039 23 дні тому

    Thank you bhaiya,much needed for data engineer job

  • @debarghyasharma6910
    @debarghyasharma6910 23 дні тому

    Hi Saurabh, In my case after second commit the deploy stage(using elastic beanstalk) fails in codepipeline. Can you help me regarding this?

  • @cretive549
    @cretive549 24 дні тому

    sir ye playlist me aap pura spark sikhaoge ?

  • @vijaybisht4205
    @vijaybisht4205 26 днів тому

    Sir, i watch your videos and it's always a great learning experience. I would recommend deleting the last part of the video (1.06.00).

  • @YashKumarJain-v5c
    @YashKumarJain-v5c 29 днів тому

    Hello, i followed your vedio, i got this error, "although the task says it's queued. (Info: None) Was the task killed externally?", could you help me out please

  • @aayushchourasia9224
    @aayushchourasia9224 Місяць тому

    Super good content saurabh, you explain really nice, I would love to see more videos on Data Engineering tools and technologies :)

  • @thesuccessfactor383
    @thesuccessfactor383 Місяць тому

    Mast Vdeo

  • @lalitkumar129
    @lalitkumar129 Місяць тому

    Nice

  • @rahulkumar-qt5yy
    @rahulkumar-qt5yy Місяць тому

    hello sir....i did not understand one thing that...once we read the data from s3 and convert it into parquet format....then, our requirement is to insert the data into redshift then, why we need to create another table in AWS data catalog sir ?

  • @prabhatgupta6415
    @prabhatgupta6415 Місяць тому

    provide the code

  • @nadianizam6101
    @nadianizam6101 Місяць тому

    Excellent explanation.want more videos

  • @Sunnykhatnani
    @Sunnykhatnani Місяць тому

    Why are we crawling redshift when it is empty?

  • @nadianizam6101
    @nadianizam6101 Місяць тому

    Excellent explanation

  • @paurashdewangan7349
    @paurashdewangan7349 Місяць тому

    🔥🔥🔥thank you so much. can you please provide the ppt🙂

  • @ssalgia
    @ssalgia Місяць тому

    lambda is a big pain

  • @prabhatgupta6415
    @prabhatgupta6415 Місяць тому

    can we practice glue in free tier or it will be chargable?

  • @ssrakeshsharma
    @ssrakeshsharma Місяць тому

    Bro great Job. BC just 131 likes on this video, Yaha kisi ne apni ek baar G##D hila di hoti to 1 million like aa jate. In Bharat there is no value of science, hardwork and technical work.

  • @deepakkaushik4192
    @deepakkaushik4192 2 місяці тому

    Do we need to create 10 code pipeline for 10 glue jobs..can we use same pipeline for 10 glue jobs deployed on different times

  • @deepakkaushik4192
    @deepakkaushik4192 2 місяці тому

    Great work

  • @hafizadeelarif3415
    @hafizadeelarif3415 2 місяці тому

    In AWS Redshift cluster, what is zero ETL and how does it work, sir?

  • @hafizadeelarif3415
    @hafizadeelarif3415 2 місяці тому

    In AWS Redshift cluster, what is zero ETL and how does it work, sir?

  • @thelifehackerpro9943
    @thelifehackerpro9943 2 місяці тому

    Why need two crawler, can't we directly read parquet and load to Redshift

  • @420nyk
    @420nyk 2 місяці тому

    Bro love you videos.. especially long form videos where you explain each and every option or a feature of a service

  • @apurbarana8731
    @apurbarana8731 2 місяці тому

    Hello sir, video is too good. I have cleared lots of things from this video. 🥳🎉👌👌

  • @vandung6025
    @vandung6025 2 місяці тому

    Thank you so much

  • @thapasujan07
    @thapasujan07 2 місяці тому

    Nicely done Sir. Thank you.

  • @thapasujan07
    @thapasujan07 2 місяці тому

    Nice one Sir. Thank you.

  • @hafizadeelarif3415
    @hafizadeelarif3415 2 місяці тому

    Hi sir, could you please share the slides that you used in this video?

  • @venkataramanamurthypasumar4542
    @venkataramanamurthypasumar4542 2 місяці тому

    Very Good

  • @DiptiprakashChhanda
    @DiptiprakashChhanda 2 місяці тому

    This is the best video which cleared my doubts and i completely understood the concepts.

  • @nadianizam6101
    @nadianizam6101 2 місяці тому

    excellent

  • @venkataramanamurthypasumar4542
    @venkataramanamurthypasumar4542 2 місяці тому

    Very good explanation

  • @vinodvashistha4260
    @vinodvashistha4260 2 місяці тому

    Great work, Saurabh!

  • @krishnakantsharma8021
    @krishnakantsharma8021 2 місяці тому

    Sir bohut badiya smjhaate ho par pls gala thoda saaf krke videos banaye kabhi kabhi irritating ho jata hai ye

  • @devprasad609
    @devprasad609 2 місяці тому

    Well streamlined video and content.!

  • @AnkitaKulshreshtha
    @AnkitaKulshreshtha 2 місяці тому

    Very informative 🎉🎉 Thank you

  • @mojijojo-r1d
    @mojijojo-r1d 2 місяці тому

    Thankyou so much....with your help I got role in jpmc... however I am scared as I don't belongs to coding background ...how I will survive as data engineer

  • @mashalkhan7699
    @mashalkhan7699 2 місяці тому

    Sir you are great and No. 1

  • @hitesh7719
    @hitesh7719 2 місяці тому

    Excellent video . Can not thank you enough for giving me the confidence of mastering redshift foundation from both your redshift theory and this practical tutorial.