Відео

Thank you 👍 Don't forget to share with your network over LinkedIn ♻️

One region one metastore. Try using same metastore created at eastus or try creating workspace in different region with another metastore in that region

Used by both personas DE and DS

Make sure you are using adls gen 2 with hierarchical namespace enabled

This is Spark Training, nothing to do with Databricks. Spark is a open source framework which works without Databricks as well. If you are working with Databricks notebooks then you dont need to create Spark Session. And in this training we are not covering Databricks. We cover Databricks in a different playlist.

@ got it buddy, great work

Hello, The count 40 that you see for shuffle is at each executor level. Now, each task process its own partition and write 10 records for each partition because of 10 dept. So at each executor 4X10 = 40 records The example for 0th partition was just to show that each partition contains data for more that one dept. To understand it better please try the same in your notebook.

Hello Rakesh, Please dont rush to complete the video, the importance is to learn while you watch it. I would recommend to watch 1 hour daily for next 6 days and practice along with it. And its generic PySpark, mllib is not covered it in. But once you understand this learning mllib will not take time. Don't forget to share this with your network ♻️

@easewithdata thank you sir u know around 2.5 hrs in 0.75x I just completed .30 min of video bcz I want to gain knowledge rather than completing video am I preparing for ds is it enough to skill ? And sir if u have time to create tutorial on AWS or azure cloud services keeping in context of data science and analytics ?

Thank you so much for your kind words. I know my reach and subscription rate is not good, I want people to learn all basics and understand the concepts. Please make sure you share this will all your friends and share this with your network over LinkedIn. Your help means a lot 🥰

@@easewithdata sure sir am great full and told my friends family to subscribe please upload more videos

Details already mentioned in the comments of the video.

Kafka allows you to work over real time, you can use api, python code, java sdk etc to publish data over Kafka

Declarative framework like DLT allows you to create table without schema and manage it automatically as per your data.

SCD is a type of dimension to maintain data in DWH. COPY INTO is to bring data into Data Lake from utilizing file sources.

Thank you, please make sure to share with your network 😃

Kudos 👏 I hope you loved your journey of knowledge 😊 Dont forget to share this with your friends over LinkedIn and tag us ♻️

@@easewithdata yes sure

For Associate Exam preparation - Buy udemy exam preparation by Derar alhussein. This should be enough to clear association certification. Dont forget or recommend the content to your network over LinkedIn ♻️

@ thank you for the sharing the information.🙏 Sure will recommend and share

Thank you 👍 Please make sure to share with your network over LinkedIn ♻️

Thank you 👍 Don't forget to share this with your network over LinkedIn ♻️

Databricks is built on spark and basic fundamental is to have multiple nodes(machine) to process data so we have to use cluster there's is no option and cluster are nothing but bunch of vms. In case if adf and synapse u are taking that are also internally invoked the vms to process our jobs it's just that it is hided from user and azure manage that and we just see it as runtime environment there

Everything required compute/process/cpu in the background for execution, sometimes this is completely abstracted from the end users. Even ADF and Synpase has that in background. So does all RDMS systems, this is why you install them on machines. ADB allows you to configure it as per your requirement, thus you see VM's being spinned up and used.

Yes, enough to get you started and make you comfortable with Spark.

I was wondering if you could make a video on how to perform upserts in PySpark, especially when working with subqueries in MERGE or UPDATE statements. Handling correlated subqueries and updates with complex joins is often tricky. Would love to see your insights on optimizing such operations!

Thank you 👍 You can checkout the Databricks Zero to Hero series to learn more about Merge in Delta tables. I will try to cover optimizations in future videos. ua-cam.com/play/PL2IsFZBGM_IGiAvVZWAEKX8gg1ItnxEEb.html Don't forget to share with your network over LinkedIn ♻️

@@easewithdata can I practise these databricks playlist in Fabric notebooks?, please let me know

@@Shreekanthsharma-t6x Yes, if it allows you to run PySpark code.

@@easewithdata thanks a lot

Thank you 👍 Don't forget to share with your network over LinkedIn ♻️

Thank you ❤️ Dont forget to share this with your network over LinkedIn ♻️

Thanks subham. I did not see this update. I updated the dockerfile using ChatGpt. and it worked. # Base Python 3.10 image FROM python:3.10-bullseye # Expose ports EXPOSE 8888 4040 # Change shell to /bin/bash SHELL ["/bin/bash", "-c"] # Upgrade pip RUN pip install --upgrade pip # Install OpenJDK RUN apt-get update && \ apt-get install -y --no-install-recommends openjdk-11-jdk && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* # Fix certificate issues RUN apt-get update && \ apt-get install -y --no-install-recommends ca-certificates-java && \ apt-get clean && \ update-ca-certificates -f && \ rm -rf /var/lib/apt/lists/* # Install nano and vim RUN apt-get update && \ apt-get install -y --no-install-recommends nano vim && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* # Setup JAVA_HOME -- useful for Docker commandline ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64/ ENV PATH $JAVA_HOME/bin:$PATH # Download and Setup Spark binaries WORKDIR /tmp RUN wget archive.apache.org/dist/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz && \ tar -xvf spark-3.3.0-bin-hadoop3.tgz && \ mv spark-3.3.0-bin-hadoop3 /spark && \ rm spark-3.3.0-bin-hadoop3.tgz # Set up environment variables ENV SPARK_HOME /spark ENV PYSPARK_PYTHON /usr/local/bin/python ENV PYTHONPATH $SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.9.5-src.zip ENV PATH $PATH:$SPARK_HOME/bin # Fix configuration files RUN mv $SPARK_HOME/conf/log4j2.properties.template $SPARK_HOME/conf/log4j2.properties && \ mv $SPARK_HOME/conf/spark-defaults.conf.template $SPARK_HOME/conf/spark-defaults.conf && \ mv $SPARK_HOME/conf/spark-env.sh.template $SPARK_HOME/conf/spark-env.sh # Install Jupyter Lab, PySpark, Kafka, boto & Delta Lake RUN pip install jupyterlab==3.6.1 pyspark==3.3.0 kafka-python==2.0.2 delta-spark==2.2.0 boto3 # Change to working directory and clone git repo WORKDIR /home/jupyter RUN git clone github.com/subhamkharwal/ease-with-apache-spark.git # Fix Jupyter logging issue RUN ipython profile create && \ echo "c.IPKernelApp.capture_fd_output = False" >> "/root/.ipython/profile_default/ipython_kernel_config.py" # Start the container with root privileges CMD ["jupyter", "lab", "--ip=0.0.0.0", "--allow-root"]

Thank you 💓 Don't forget to share with your Network on LinkedIn ♻️

Thank you for letting me know, I fixed it. Please download the latest Dockerfile from github.com/subhamkharwal/docker-images/tree/master/pyspark-jupyter-lab and try again. Please let me know if that works

Please download the latest Dockerfile from github.com/subhamkharwal/docker-images/tree/master/pyspark-jupyter-lab and try again. Please let me know if that works

Thank you so much 💓 Don't forget to share this with your network over LinkedIn ♻️

Thanks 💓 Don't forget to share with your Network on LinkedIn ♻️

Thank you 👍 Don't forget to share with your network over LinkedIn ♻️

You can use PySpark notebook by running this command in docker docker pull jupyter/pyspark-notebook or use Databricks community edition

Absolutely, this is how spark works. But it you want to explain the complete process this is the video - ua-cam.com/video/CYyUuInwgtA/v-deo.htmlsi=eRNtot_osWZ-DvlY

OK 🙋

I will cover this later in this series.

This series currently on hold.

Absolutely join works with Hive. If you select more than one column then the hashing will happen with combination of both columns.

Glad you found it helpful 😊 Please make sure to share with your network over LinkedIn

Glad you enjoyed the video. Make sure to share it with your network over LinkedIn! ♻️

Here you can find resources on DLT META - databrickslabs.github.io/dlt-meta/

Thanks bhai ❤