1:10 Setting up Miniconda. 3:40 Installing ipython, dask and dask_jobqueue. 4:57 Making a cluster with dask_jobqueue: requesting resources, submitting jobs and connecting to our dask cluster. 8:08 Example: simpple demonstration of parallelization. 9:51 Under the hood: the job script submitted by dask. 10:43 Cluster configuration. 15:35 Distributed configuration. 17:50 SSH port forwarding + accessing the dask dashboard from your local machine. 24:00 Set up your interactive work environment with jupyter lab. 30:10 Further information. Thanks for the video!
Hi Matthew, awesome, crystal clear and really useful video! One thing you may consider next time is to not place the bottom of the terminal all the way to the bottom of the video frame. The youtube progress and control bar is kind of hiding your last typed command whenever I press pause, when i'm trying to implement your setup at the same time as I watch the video. Thanks again for the great tutorial!
Thank you for this explanation, very clear I am having hard time running parallel computation with ipyparallel inside jupyter. I lost contact with my workers/engines after reconnecting to the notebook server.
Hi and thanks for the video. For the number of processes, after following your steps, sometimes I only see 0/10 or 4/10 for me when I print out client. Do you know what might be causing this? Also when the job are done processing in the queue, client processes print out back to 0 for me.
1:10 Setting up Miniconda.
3:40 Installing ipython, dask and dask_jobqueue.
4:57 Making a cluster with dask_jobqueue: requesting resources, submitting jobs and connecting to our dask cluster.
8:08 Example: simpple demonstration of parallelization.
9:51 Under the hood: the job script submitted by dask.
10:43 Cluster configuration.
15:35 Distributed configuration.
17:50 SSH port forwarding + accessing the dask dashboard from your local machine.
24:00 Set up your interactive work environment with jupyter lab.
30:10 Further information.
Thanks for the video!
Hi Matthew, awesome, crystal clear and really useful video! One thing you may consider next time is to not place the bottom of the terminal all the way to the bottom of the video frame. The youtube progress and control bar is kind of hiding your last typed command whenever I press pause, when i'm trying to implement your setup at the same time as I watch the video. Thanks again for the great tutorial!
Thanks Matthew for this, really helped me get setup on my university cluster!
Thank you for this video. Along with explaining Dask's usage, you have introduced a very useful workflow.
Excited to try this. Thanks!
Thank you, very enlightening.
This is great solving access to compute.
How do you handle data and code dependencies?
In this case we're just using the network file system to handle software dependencies and data. This is common for how people use HPC systems.
Thank you for this explanation, very clear I am having hard time running parallel computation with ipyparallel inside jupyter. I lost contact with my workers/engines after reconnecting to the notebook server.
Thank you very much for the tutorial!
Can use my jupyter notebook instead of jupyter lab?
Hi and thanks for the video. For the number of processes, after following your steps, sometimes I only see 0/10 or 4/10 for me when I print out client. Do you know what might be causing this? Also when the job are done processing in the queue, client processes print out back to 0 for me.
Maybe your worker jobs haven't yet started?
1:00 dashdashboardboard