Hello Sir, I was asked the following questions for AWS Developer role. Please make a video on this. Thanks. Q1. We have *sensitive data* coming in from a source and API. Help me design a pipeline to bring in data, clean and transform it and park it. Q2. So where does pyspark come into play in this? Q3. Which all libraries will you need to import to run the above glue job? Q4. What are shared variables in pyspark Q5. How to optimize glue jobs Q6. How to protect sensitive data in your data Q7. How do you identify sensitive information in your data Q8. How do you provision a S3 bucket? Q9. How do I check if a file has been changed or deleted? Q10. How do I protect my file having sensitive data stored in S3 Q11. How does KMS work? Q12. Do you know S3 glacier? Q13. Have you worked on S3 glacier?
was curious why spark handles smaller files differently, & also had a fixed view that partition size is 128 MB all times, that view of mine is debunked now, beautifully explained , thank you very much sir :)
Thanks for the simplified explanation, please make next video on deciding configuration for our jobs , ex: which one is better config i)10 executors with 4 cores and 4gb ram each or ii) 5 executors with 8 cores and 8 gb ram, there is no proper explanation about this concept anywhere
thank you so much for the great explanation in detail, can you discuss more on like many times interviewer might ask you have worked on how much big data size for that what could be cluster configuration, how you decide it, what can be optimized solution, what kind of data and its size, and more on next to next video expectation on spark optimization techniques
Hi Sir, Thankyou for the explanation.. I have a situation, I am executing a databricks pipeline through Airflow. In one of the task, I am writing the data from dataframe to a path ( in parquet file). The writing operation suppose to create the path on daily basis and write the data into the path. Path is being created but after writing, when I am checking the count, it is showing zero. It's not giving any error as well so really difficult to identify the issue. But, when I am reprocessing the same task then it's writing the data.
Hi sir, I wanted to get confirm. are you saying based on cpu cores, number of partition would happen. Like you said for 1GB data, there would be 8 partition due to paralellism then it will be 4 partition, correct. Pls correct me if i am wrong.
A very interesting start of an exciting series :) ... appreciate all your effort .... Just wanted to confirm one thing ... in case of RDD-s the partition size is always 128 mb right (what you explained applies for dataframe/higher level API-s)?
In the example I have, I am getting df.rdd.getNumPartitions() equal to 200 which seems to be the default. I have 160 cores available in the cluster. How should we understand this in the light of what you say in the video, I feel like this part is missing. Also, when should we use spark.conf.set("spark.sql.shuffle.partitions", "auto")
in the context of video he just explained about intial stage of partitions, in your case data is get shuffled that why it has created 200 by default partitions.
As per my understanding, 160 tasks will be perfomed parellely and the remaining 40 tasks will wait in queue. And those 40 tasks will be performed when any 40 out of 160 cores are available again.
Hello Sir, I was asked the following questions for AWS Developer role. Please make a video on this. Thanks.
Q1. We have *sensitive data* coming in from a source and API. Help me design a pipeline to bring in data, clean and transform it and park it.
Q2. So where does pyspark come into play in this?
Q3. Which all libraries will you need to import to run the above glue job?
Q4. What are shared variables in pyspark
Q5. How to optimize glue jobs
Q6. How to protect sensitive data in your data
Q7. How do you identify sensitive information in your data
Q8. How do you provision a S3 bucket?
Q9. How do I check if a file has been changed or deleted?
Q10. How do I protect my file having sensitive data stored in S3
Q11. How does KMS work?
Q12. Do you know S3 glacier?
Q13. Have you worked on S3 glacier?
PySpark Scenario Based Interview Question And Answers:
ua-cam.com/play/PLqGLh1jt697zXpQy8WyyDr194qoCLNg_0.html&si=Ddhve6jjcy0ZvaLV
was curious why spark handles smaller files differently, & also had a fixed view that partition size is 128 MB all times, that view of mine is debunked now, beautifully explained , thank you very much sir :)
I loved your presentation and understand more with 1 video about configuration ❤
Thanks for the simplified explanation, please make next video on deciding configuration for our jobs , ex: which one is better config i)10 executors with 4 cores and 4gb ram each or ii) 5 executors with 8 cores and 8 gb ram, there is no proper explanation about this concept anywhere
Sir,
Great to see such scenarios explained clearly. We would love to watch some interview questions on databricks. Kindly post them.
Insightful and informative concept, thank you Sir for clearing it out with ease ✅
thank you Soumik
Thank you Sir, also please continue advanced sql playlist, I have completed both basic and advanced playlist it's very helpful.
Thank you Sir for explaining the answers with a scenario. It's really helpful.
Was literally waiting for your video series... Congo for a great start sumit sir.
thank you so much for the great explanation in detail, can you discuss more on like many times interviewer might ask you have worked on how much big data size for that what could be cluster configuration, how you decide it, what can be optimized solution, what kind of data and its size, and more on next to next video expectation on spark optimization techniques
Good explanation sir. Make a video on possible scenario based questions like this
3rd scenario is not clear. Are we updating ant congratulation setting to avoid small files problem
Great explanation as always. Thank you Sumit sir, waiting for much such videos in future 😊
thank you Arpit
Please continue the series
Excellent explanation, highly informative!
very insightful please cover more spark internals scenerio based questions
Well explained as always. I was asked the same question in one of the interviews.
thats great to know..
Hi Sir, Thankyou for the explanation..
I have a situation, I am executing a databricks pipeline through Airflow. In one of the task, I am writing the data from dataframe to a path ( in parquet file). The writing operation suppose to create the path on daily basis and write the data into the path. Path is being created but after writing, when I am checking the count, it is showing zero. It's not giving any error as well so really difficult to identify the issue.
But, when I am reprocessing the same task then it's writing the data.
Great explanation sir as always
Really very well explained!
Thank you Sir, the start of the series is great. Do we have to explain each answer with scenarios?
Hi sir, I wanted to get confirm. are you saying based on cpu cores, number of partition would happen. Like you said for 1GB data, there would be 8 partition due to paralellism then it will be 4 partition, correct. Pls correct me if i am wrong.
Very well explained
Thank you sir!
Where is the next part ?
A very interesting start of an exciting series :) ... appreciate all your effort .... Just wanted to confirm one thing ... in case of RDD-s the partition size is always 128 mb right (what you explained applies for dataframe/higher level API-s)?
thats correct, in case of rdd.. it depends on the block size of underlying filesystem. in case of hdfs it will be 128 mb.
@@sumitmittal07And in case of object store/s3, will it be default Parallelism or the number of splits in source file from s3
In the example I have, I am getting df.rdd.getNumPartitions() equal to 200 which seems to be the default. I have 160 cores available in the cluster. How should we understand this in the light of what you say in the video, I feel like this part is missing. Also, when should we use spark.conf.set("spark.sql.shuffle.partitions", "auto")
in the context of video he just explained about intial stage of partitions, in your case data is get shuffled that why it has created 200 by default partitions.
As per my understanding, 160 tasks will be perfomed parellely and the remaining 40 tasks will wait in queue. And those 40 tasks will be performed when any 40 out of 160 cores are available again.
Sir i have watched your SQL complete playlist can i face the sql interview now??
Yes definitely