This is how a University Professor should teach. Great content, clear from the start to the end. I had no prior knowledge of NetCDF and in 2 days thanks to this channel I can now perform all operations required to the extent my job interfaces with climate data. Just excellent.
Thank you so much for your kind comments. ICTP is not a university but I do a lot of teaching there anyway, so that gives me the chance to practice 😉 i had some technical issues that slowed my output recently but i hope to get more material out late summer... Best wishes and I hope you subscribed to get notifications of my sporadic output
Dear Professor. Thankyou very much for such informative video and learning materials. I was looking for resampling and got this video which will be very very helpful now. I am a PhD research student from IIT Delhi. Working with Prof Sagnik and Prof Dash. I met you during an workshop by ICTP at IIT Delhi in 2016 December. I am the boy who got ligament knee surgery during that time (If you remember 🙂🙂🙏🏻🙏🏻). Thanks a lot.
Yes of course i remember you! Glad you find the video helpful. Was thinking to try to rerun a school in India in 24; let's see if I can get the funding and make it work . Please pass on the word about the channel, more material on the way
Thank you for your kind comments. Please don't forget to subscribe and let anyone know you think might find this material helpful. I'm slowly but surely building up a suite of videos whenever I get a few spare moments and hope to add some application examples soon too.
Thanks, I appreciate the feedback - I'll put a video on ncgen on my "to-do" list - Please pass the word about my channel, I need feedback to know what the priorities are for people regarding data analysis :-)
Thank you for this tutorial! It was very helpful! Question: Do I need to regrid the original NetCDF file (0.25 deg resolution) to a finer resolution for extracting temperature time series for the particular point in the Antarctic region? If so, what the resolution should be?
Hi sir , thank you for such an amazing video . I got what I was looking for finally after spending a day searching 😃 . Sir I noticed you talked about remapcon when going from fine scale to coarse scale . Currently I'm working on gcm Precipitation data , monthly scale . The spatial resolution is coarse . I wanted to match it's resolution to my reference Grided data set that's on fine scale . So can I use remapcon? Or it's recommend to use remapbil
Hi . I am currently taking your class in ICTP.edx online and I am applying all the techniques in my research. I just have a question, in the ncdump -h 1. dimensions: time = UNLIMITED; (36 currently) ---> my datasets have different numbers on the parenthesis. Does this cause a problem in the grid dimensions lon and lat dimensions are uniform. 2. When using mergetime, I got an error cdo mergetime(abort): Grid size of the input parameter precip do not match. From the video, regridding was done with the issue of resolution. Would love to hear your advice when it comes to the variables. Thank you so much for doing these videos. They are really amazing and informative.
Hi shielo this is a bit tricky to answer in UA-cam Chat. Can you copy the discussion over to the discussion pages on the openedx perhaps? What are the inputs of the mergetime command by the way? Also which question? Best wishes
Great videos sir! I have pixel values for different longitude and latitude of specific regions. These grided datasets are monthly Tmax, Tmin, and monthly rainfall values. My question is can I add all pixel values to obtain the mean areal average value of the region?
my problem is not an interpolation but a remapping. my data instead to have a regular grid of longitude starting from 0 to 360, it starts from 320 to 319. I mean 320 to 360 and from 1 to 319. how to fix that.
Deflation. Do you mean file compression? I've got plenty of new material in the pipeline, just struggling to find the time to finish the editing at the moment due to a heavy teaching load and paper revisions... But will be arriving soon
great video, A question: I've two data sets with different spatial resolutions and I want to recalculate the values of a grid decreasing the resolution . i'm try with the bi-linear method, should i re-grid the higher resolution (0.44) to lower resolution (0.03)? please give command e.g cdo remapbil,r720x360 input.nc out.nc
Thanks for your kind feedback. As i say in the video, the choice of regridding method depends on the direction. If there are features of the high resolution you want to retain you can go from low to high. In this case you can use bilinear or nearest neighbour. I wouldn't use the command you state but instead directly map one to the other e.g cdo remapnn,high.nc low.nc low_remap.nc but if you go the other direction better to use conservative remapping for reasons i explain in the video. E.g. cdo remapcon,low.nc high.nc remapped.nc
PS. Don't forget to subscribe and spread the word about the channel. I know i haven't uploaded anything for a couple of months, work totally crazy at the moment but am working on new material
@Climate Unboxed it's indeed an amazing tutorial. I am using cdo to remap the file downloaded from ERA5 which have multiple variables and I have sentinel 5P file as target grid when I use "cdo.remapbil("l3_mean_europe_2018_2020_005dg.netcdf", input='met_para_Europ.nc', output="output.nc")" it gives me an error "expected str, bytes or os.PathLike object, not NoneType" I could not get it. Can you please help?
Hi there, your command looks ok, although I'm not a frequent user of the python package. Perhaps post on stack overflow with a link to the file and I'll try to take a look in detail?
Hi! tks for this video. A question: I've two datasets with different spatial resolutions and I want to recalculate the values of a grid decreasing the resolution (for pixel-to-pixel comparison). Trivially with the conservative method, I should calculate average the values of the higher resolution pixels (0.05) within the polygon of the lower resolution (0.1)? Or is this not correct? Thank you again.
Hi Camilo. There is not a correct method, both are fine. You can interpolate the coarse to the fine grid or the fine to the coarse. The important thing is the take care of the method used if you coarsen the fine grid as per the notes in the video. I hope you found the video helpful! Please be sure to subscribe so you receive notification when I upload new material. 😉
Yes indeed, but with noisy fields like precipitation you probably want to use remapcon. But it depends on the resolution change fine to coarse or the other way around. I hope the material in the video was useful for this.
Fantastic Video!! Waiting to watch more of your video. Can you teach me how to extract Mankendall, sen slope & their P values of time series nc files and write the output slopes and p values in .nc file.
it is grate lecture .when i apply regridding to a particular country precipition data. the obtained data crossing the boundary line of the country. could you please explain why this happening.
CDO doesn't know about boundaries, only the grid you are projecting to... Does the output field match the target grid specified? Next week i will release a video on land sea masks to handle coastal boundaries correctly. However if you want to mask data using a country shape file you need to do this in R or python. I'll eventually show how to do this too but it is lower down the Todo list I'm afraid...
When I try to remap my data using 'cdo remapbil,example_file input_file output_file, I get the following error: Unsupported generic coordinates (Variable: X)! Strange, because my netcdf has 3 dimensions (time, lat, lon). Have you experienced such error as well?
@John van de Ven @@climateunboxed I had the same question - thanks for clarifying. I'd also like to add when remaping, sometimes the variable name "longitude/latitude" changes to "lat/lon". I'm guessing this can also be avoided if the data is CF compliant Thank you very much for these tutorials - they are very helpful.
Great video- thanks. Climate projections data are often in rotated coordinate especially data from CORDEX and it is so confusing to mask it out for the region of interest, Can you please make a video on how to convert rotated coordinate to regular lat-long and then clip it out for a sub-region using CDO?
Sounds like you missed the comma or the gridspec file not correct. Difficult to answer here, without seeing the command and file. Try to post both to stack overflow with the tag cdo-climate and I can try to help there
Great video, thank you. I found this video while I was trying to find a solution to this error: 'cannot apply_along_axis when any iteration...' when regridding. This is using xarray. if you were able to provide advice on this, it would be greatly appreciated.
hi James, tricky to help with only the error message - I suggest you post the question with further details on stackoverflow, tag it with xarray and python. Sorry I can't be of more assistance here...
Are you piping (i.e. concatenating commands)? This can cause a seg fault if you don't force sequential operation with -L which does a Lock I/O (sequential access). [I'll post a video on piping soon] - Otherwise not sure, could be the compilation of cdo on the system. Try to update within apt...
hi - If you are on a mac, I advise to use homebrew and then it is "brew install cdo" - if you are on linux like ubuntu or mint, it is "sudo apt-get install cdo" - if you are on windows 10, then the best solution is to install the linux subsystem (this is available directly within windows 10 and is very easy to activate, i.e. do NOT use a wineserver or cygwin), and then you can open a ubuntu terminal withins windows and follow the linux instructions. :-)
@@climateunboxed Great video. Although I tried installing cdo using all sorts of ways. Also using homebrew. I still get the following error when I try to run the following code: cdo remapbil,gridfile.txt input.nc output.nc (where input.nc and output.nc are named differently) -> Unsupported file type (library support not compiled in) To create a CDO application with NetCDF4 support use: ./configure --with-netcdf= ... Any tips? Maybe make a quick video on how to download and install cdo properly. Not a lot of information out there.
@@achterbumper Sorry to be slow to reply, moved house and no internet yet and zero phone signal !!! I feel cut off from the world!!! Anyway, try uninstalling cdo with brew, THEN try first to do "brew install netcdf", and then reinstall cdo with brew... let me know if that helps
@@climateunboxed I do am on a mac. After trying several methods which didn't work, including homebrew initially. I tried again by uninstalling all packages (hdf,netcdf,cdo) and reinstalling them with homebrew. That seemed to do the trick! Thanks for the tip!
Dear Adrian, on regridding my tmax file from coarser to finer resolution it is also using the fill value/ missing value (i.e 99) in the calculation for the nearby grids. How to avoid this?
Do you mean some cells are marked as missing with 99 ? This is correct behavior. But if you mean the 99 is being used as a field value, ( e.g. you get some strange entries such as 56 due to 99 being "merged" with nearby true values) it shouldn't do this if 99 is defined as missing in the field metadata. Have you checked this is the case using ncdump -h ? If it is not set I can tell you how to add out using nco
@@climateunboxed Dear Adrian, If possible, can I mail my tmax file to you? It says _FillValue = NaN, // double, but when I am looking at the data, wherever value is missing 99.9 is written. And this 99.9 is also participating in the regridding process, and this is giving trouble to me.
@@arjunkaushik5075 Try this command on the original file and see if it helps: ncatted -a _FillValue,,m,f,99.9 tmax.nc or if you want to use cdo you can alternatively try cdo setmisstoc,99.9 t2m.nc t2m_new.nc I'm afraid I can't do bespoke file investigations, as I'm snowed under with the day job at the moment (in fact, I'm way behind schedule on these videos too, which I need to finish for a EDX course I am launching)
This is how a University Professor should teach. Great content, clear from the start to the end. I had no prior knowledge of NetCDF and in 2 days thanks to this channel I can now perform all operations required to the extent my job interfaces with climate data. Just excellent.
Thank you so much for your kind comments. ICTP is not a university but I do a lot of teaching there anyway, so that gives me the chance to practice 😉 i had some technical issues that slowed my output recently but i hope to get more material out late summer... Best wishes and I hope you subscribed to get notifications of my sporadic output
videos are very instructive, thanks a lot!
BSc Meteorology student in Istanbul
You are welcome! Thanks for the kind words
Dear Professor. Thankyou very much for such informative video and learning materials. I was looking for resampling and got this video which will be very very helpful now.
I am a PhD research student from IIT Delhi. Working with Prof Sagnik and Prof Dash. I met you during an workshop by ICTP at IIT Delhi in 2016 December. I am the boy who got ligament knee surgery during that time (If you remember 🙂🙂🙏🏻🙏🏻).
Thanks a lot.
Yes of course i remember you! Glad you find the video helpful. Was thinking to try to rerun a school in India in 24; let's see if I can get the funding and make it work . Please pass on the word about the channel, more material on the way
@@climateunboxed That's great . Surely I will let others know about this helpful tutorials and channel. Thank you, Dear Professor.
Simply presented and well explained! Thanks.
Thanks so much for this sharing !!!
You are very welcome
Hats off. God bless you.....
I loved this, Thank you for clarifying many useful concepts.
You are very welcome!
Thank you Adrian, great tutorial !
Thank you for your kind comments. Please don't forget to subscribe and let anyone know you think might find this material helpful. I'm slowly but surely building up a suite of videos whenever I get a few spare moments and hope to add some application examples soon too.
Dear Adrian, I need to interpolate my cordex data do era5land's grids. Would this video help me to do it? How may I find a solution to my need?
Great content ! Thanks !! Would really appreciate if you could upload videos about conversion of text files to netcdf or vice versa. Thanks
Thanks, I appreciate the feedback - I'll put a video on ncgen on my "to-do" list - Please pass the word about my channel, I need feedback to know what the priorities are for people regarding data analysis :-)
This is a great tutorial ! Great job Adrian
Thank you for this tutorial! It was very helpful! Question: Do I need to regrid the original NetCDF file (0.25 deg resolution) to a finer resolution for extracting temperature time series for the particular point in the Antarctic region? If so, what the resolution should be?
Great stuff sir. Thank you.
You are very welcome, please advertise the channel if you think it is useful for your colleagues
Very useful content thanks you !!
You are welcome!
I am a newbie in the field of meteorology but you are helping me alot. I have finished watching all your videos though haha!
That's great to know. Thanks 🙂
thank you so much sir, such a wonderful presentation
thanks again sir
For getting reanalysis values for individual points, would it be better to use linear or cubic interpolation?
Great sir
Hi sir , thank you for such an amazing video . I got what I was looking for finally after spending a day searching 😃 .
Sir I noticed you talked about remapcon when going from fine scale to coarse scale .
Currently I'm working on gcm Precipitation data , monthly scale . The spatial resolution is coarse . I wanted to match it's resolution to my reference Grided data set that's on fine scale .
So can I use remapcon? Or it's recommend to use remapbil
Hi . I am currently taking your class in ICTP.edx online and I am applying all the techniques in my research. I just have a question, in the ncdump -h
1. dimensions:
time = UNLIMITED; (36 currently) ---> my datasets have different numbers on the parenthesis. Does this cause a problem in the grid dimensions
lon and lat dimensions are uniform.
2. When using mergetime, I got an error
cdo mergetime(abort): Grid size of the input parameter precip do not match.
From the video, regridding was done with the issue of resolution. Would love to hear your advice when it comes to the variables.
Thank you so much for doing these videos. They are really amazing and informative.
Hi shielo this is a bit tricky to answer in UA-cam Chat. Can you copy the discussion over to the discussion pages on the openedx perhaps? What are the inputs of the mergetime command by the way? Also which question? Best wishes
Dear sir, Do you have any notes or any publicatio to understand the same
Thanks you for awesome content
Great videos sir! I have pixel values for different longitude and latitude of specific regions. These grided datasets are monthly Tmax, Tmin, and monthly rainfall values. My question is can I add all pixel values to obtain the mean areal average value of the region?
yes indeed - please see my video on calculating spatial statistics in my playlist - it tells you exactly how to perform this task
What do you opinion if i'm using bilinear interpolatiom for regrid surface air temperature ? Does it "proper"
Temperature is a fairly "smooth"field so you can get away with bilinear interpolation in most circumstances
my problem is not an interpolation but a remapping. my data instead to have a regular grid of longitude starting from 0 to 360, it starts from 320 to 319. I mean 320 to 360 and from 1 to 319. how to fix that.
Great stuff! If you find the time, a video for chunks and chunk sizes would be really appreciated. Deflation wouldn't be bad either
Deflation. Do you mean file compression? I've got plenty of new material in the pipeline, just struggling to find the time to finish the editing at the moment due to a heavy teaching load and paper revisions... But will be arriving soon
great video, A question: I've two data sets with different spatial resolutions and I want to recalculate the values of a grid decreasing the resolution . i'm try with the bi-linear method, should i re-grid the higher resolution (0.44) to lower resolution (0.03)?
please give command
e.g cdo remapbil,r720x360 input.nc out.nc
Thanks for your kind feedback. As i say in the video, the choice of regridding method depends on the direction. If there are features of the high resolution you want to retain you can go from low to high. In this case you can use bilinear or nearest neighbour. I wouldn't use the command you state but instead directly map one to the other e.g cdo remapnn,high.nc low.nc low_remap.nc but if you go the other direction better to use conservative remapping for reasons i explain in the video. E.g. cdo remapcon,low.nc high.nc remapped.nc
PS. Don't forget to subscribe and spread the word about the channel. I know i haven't uploaded anything for a couple of months, work totally crazy at the moment but am working on new material
@Climate Unboxed it's indeed an amazing tutorial. I am using cdo to remap the file downloaded from ERA5 which have multiple variables and I have sentinel 5P file as target grid when I use "cdo.remapbil("l3_mean_europe_2018_2020_005dg.netcdf", input='met_para_Europ.nc', output="output.nc")" it gives me an error "expected str, bytes or os.PathLike object, not NoneType" I could not get it. Can you please help?
Hi there, your command looks ok, although I'm not a frequent user of the python package. Perhaps post on stack overflow with a link to the file and I'll try to take a look in detail?
Hi! tks for this video. A question: I've two datasets with different spatial resolutions and I want to recalculate the values of a grid decreasing the resolution (for pixel-to-pixel comparison). Trivially with the conservative method, I should calculate average the values of the higher resolution pixels (0.05) within the polygon of the lower resolution (0.1)? Or is this not correct? Thank you again.
Hi Camilo. There is not a correct method, both are fine. You can interpolate the coarse to the fine grid or the fine to the coarse. The important thing is the take care of the method used if you coarsen the fine grid as per the notes in the video. I hope you found the video helpful! Please be sure to subscribe so you receive notification when I upload new material. 😉
can we interpolate rainfall data using the methods mentioned in the video
Yes indeed, but with noisy fields like precipitation you probably want to use remapcon. But it depends on the resolution change fine to coarse or the other way around. I hope the material in the video was useful for this.
Fantastic Video!! Waiting to watch more of your video. Can you teach me how to extract Mankendall, sen slope & their P values of time series nc files and write the output slopes and p values in .nc file.
I'll try to get to more complex statistics soon - my next videos will be considering simple functions first though... hopefully soon :-)
it is grate lecture .when i apply regridding to a particular country precipition data. the obtained data crossing the boundary line of the country. could you please explain why this happening.
CDO doesn't know about boundaries, only the grid you are projecting to... Does the output field match the target grid specified? Next week i will release a video on land sea masks to handle coastal boundaries correctly. However if you want to mask data using a country shape file you need to do this in R or python. I'll eventually show how to do this too but it is lower down the Todo list I'm afraid...
When I try to remap my data using 'cdo remapbil,example_file input_file output_file, I get the following error: Unsupported generic coordinates (Variable: X)! Strange, because my netcdf has 3 dimensions (time, lat, lon). Have you experienced such error as well?
That's very strange, are the units set to be CF compliant on the lon and lat? e.g. degrees_east etc?
@John van de Ven @@climateunboxed I had the same question - thanks for clarifying. I'd also like to add when remaping, sometimes the variable name "longitude/latitude" changes to "lat/lon". I'm guessing this can also be avoided if the data is CF compliant Thank you very much for these tutorials - they are very helpful.
Great video- thanks. Climate projections data are often in rotated coordinate especially data from CORDEX and it is so confusing to mask it out for the region of interest, Can you please make a video on how to convert rotated coordinate to regular lat-long and then clip it out for a sub-region using CDO?
I'll add it to my list. I've got a few others in the pipeline to finish first though, so it will take a little while. Sorry!
@@climateunboxed Any update on this? There is so many confusing answer on converting rotated coordinate into regular coordinate system. Thanks
cdo remapcon (Abort): Open failed on gridspec.txt!
How to resolve this?
Sounds like you missed the comma or the gridspec file not correct. Difficult to answer here, without seeing the command and file. Try to post both to stack overflow with the tag cdo-climate and I can try to help there
Great video, thank you.
I found this video while I was trying to find a solution to this error:
'cannot apply_along_axis when any iteration...' when regridding. This is using xarray.
if you were able to provide advice on this, it would be greatly appreciated.
hi James, tricky to help with only the error message - I suggest you post the question with further details on stackoverflow, tag it with xarray and python. Sorry I can't be of more assistance here...
great lecture, How can it solve 'segmentation fault core dumped' when i run cdo remapbil dridspace input.nc output.nc
Are you piping (i.e. concatenating commands)? This can cause a seg fault if you don't force sequential operation with -L which does a Lock I/O (sequential access). [I'll post a video on piping soon] - Otherwise not sure, could be the compilation of cdo on the system. Try to update within apt...
Thank you for the video. I am struggling with the download of CDO. Is there an easy way to be able to use this command?
hi - If you are on a mac, I advise to use homebrew and then it is "brew install cdo" - if you are on linux like ubuntu or mint, it is "sudo apt-get install cdo" - if you are on windows 10, then the best solution is to install the linux subsystem (this is available directly within windows 10 and is very easy to activate, i.e. do NOT use a wineserver or cygwin), and then you can open a ubuntu terminal withins windows and follow the linux instructions. :-)
@@climateunboxed Great video. Although I tried installing cdo using all sorts of ways. Also using homebrew. I still get the following error when I try to run the following code: cdo remapbil,gridfile.txt input.nc output.nc (where input.nc and output.nc are named differently) -> Unsupported file type (library support not compiled in) To create a CDO application with NetCDF4 support use: ./configure --with-netcdf= ... Any tips? Maybe make a quick video on how to download and install cdo properly. Not a lot of information out there.
@@achterbumper Sorry to be slow to reply, moved house and no internet yet and zero phone signal !!! I feel cut off from the world!!! Anyway, try uninstalling cdo with brew, THEN try first to do "brew install netcdf", and then reinstall cdo with brew... let me know if that helps
which platform are you on? I'll try to put a video together for this, but I need to finish some others first that I need for an online course.
@@climateunboxed I do am on a mac. After trying several methods which didn't work, including homebrew initially. I tried again by uninstalling all packages (hdf,netcdf,cdo) and reinstalling them with homebrew. That seemed to do the trick! Thanks for the tip!
Dear Adrian, on regridding my tmax file from coarser to finer resolution it is also using the fill value/ missing value (i.e 99) in the calculation for the nearby grids. How to avoid this?
Do you mean some cells are marked as missing with 99 ? This is correct behavior. But if you mean the 99 is being used as a field value, ( e.g. you get some strange entries such as 56 due to 99 being "merged" with nearby true values) it shouldn't do this if 99 is defined as missing in the field metadata. Have you checked this is the case using ncdump -h ? If it is not set I can tell you how to add out using nco
@@climateunboxed Please tell me how to set this.
What did the tmax metadata look like though? Can you cut and paste it here first?
@@climateunboxed Dear Adrian, If possible, can I mail my tmax file to you? It says _FillValue = NaN, // double, but when I am looking at the data, wherever value is missing 99.9 is written. And this 99.9 is also participating in the regridding process, and this is giving trouble to me.
@@arjunkaushik5075 Try this command on the original file and see if it helps: ncatted -a _FillValue,,m,f,99.9 tmax.nc
or if you want to use cdo you can alternatively try cdo setmisstoc,99.9 t2m.nc t2m_new.nc
I'm afraid I can't do bespoke file investigations, as I'm snowed under with the day job at the moment (in fact, I'm way behind schedule on these videos too, which I need to finish for a EDX course I am launching)