Thank you for the walk through but I am confused as to how only a point is enough to extract a section out, shouldn't we be specifying 2 points instead that can meaningfully define a rectangle of which we want to extract the data?
Thank you so much for all video you did it's very helpful. This two method used to extra the time series dataset at specific location they make some interpolation for the data, I would like to ask if there is another methods we can use to select the time series data at specific station from netcdf without interpolation?
Thanks for your kind feedback. If you use remapnn,lon=x/lat=y this does not make any interpolation and simply extracts the nearest grid cell series. I apologize if that wasn't clear in the video. You can use the lon lat arguments with other remap interpolation methods but of course they interpolate to get the series at your specified point.
Dear Adrian Tompkins, Thank you very much for these set of videos, they are the most accessible and valuable material about reanalysis data I've found! I will leave some questions about my doubts on videos comments. Thank you very much! About this video, I would like to ask about convert the output of the location info, for a CSV file. Also another doubt I have is, if we have more than one variable in the same NetCFD/GRIB file. Here you have the 2m temperature variable, but imagine we have also 2m dewpoint temperature, 2m temperature, 10m u-component of wind, 10m v-component of wind, total precipitation, etc. How to convert them to a CSV file (one column for each variable). Lastly, I don't know if it is possible, but how to convert the Units, for instance, from Kelvin to degrees celsius. Thank you very much!
Dear Pedro. ncdump gives a text output but to convert to CSV you need to use R or python etc. However there is not really a reason to ever need to have CSV. It is much easier to work directly with the netcdf format. Lastly to convert to C use cdo expr,'Tc=Temp-273.15' in.nc out.nc there is a video coming on expr soon....
Thank you, thank you, sir. Your clips are very useful. I want you to ask a question. I hope you will answer me as soon as possible. Is it possible to extract the country of Morocco from this map? and how?
you can select a lon-lon rectangle region from a dataset very easily, please see this video for details ua-cam.com/video/ewii1p1kChw/v-deo.html - if you want to mask out the sea points that is also possible and will be the subject of a future video soon
Hello mr. Tompkins, What is the order of the videos? I think the playlist is a little disordinate (vg the video #3 "Extracting subregions..." uses nc dump, that is explained in video #5 "Using ncdump...") Love this channel and already passed the word :D
oh my, you are correct... I'm still learning the youtube ropes - I'll sort that out soon, new video arriving this week I hope. I soon I will launching a new openEDX course that threads all the videos together with exercises to do.... Thanks so much for spreading the word :)
@@juliocgfreitas as soon as it is ready i will gladly post it. Things have been a bit hectic over the summer with grant applications. I'm really behind on the video front. Sorry, I really hope though to be able to post more material very soon
Thank you for the tutorial, this is so helpful!. One question, which one is more recommended to use? NN or Bilinear? is there any pros and cons for each method?
as I say in the video, it really depends on the fields, the relative changes in resolution and also what you want/need to do. There is not a "best" choice I'm afraid, it is very task specific.
Neither! Keep it in netcdf format, that way you have all the metadata and you can still read it easily. There is never a good reason to convert to a non self describing format... Unless you want to read it in excel i suppose, in which case you probably want CSV 😱
I love your videos Sir ,really helpful! I faced netcdf data whose time coordinate is showing 0000-00-00 00:00:00 when I employed cdo seinfo ifile ofile. I wanna it to have the same time coordinate to another netcdf file having the same variable . I am grateful if you help. Thanks.
Sir, but If I have a continent grid shapefile with defined cell number and their location (e.g., cell number 560990 lat 43, lon= 37) and I want to extract my NetCDF climate variable for each grid. What should I do? Thank you in advance.
HI, I want to do this for a list of lat, lon locations - namely sst parallel to the Portuguese coast. That is, along a curve. Is there a simple way to do that?
Oh nice question. You would need to use a distance-to-coast dataset for this. I would know how to mask all the parallel points. Would that be enough? What do you want to do with the data after? Upload this to stack overflow and tag it cdo-climate, then I can give the answer the full treatment...
@@climateunboxed Hello Adrian. I only just discovered "cdo" and the past couple of days watched a few of your youtube tutorials. Let me give you more background. We are working with satellite observations of SST (pathfinder 5 series) and specifically a dataset of weekly SST created by Selig et al for their project on coral reefs (CoRTAD). Our interest is in generating an upwelling index based on the difference between coastal SST and open ocean SST. We have a set of (lon,lat) coordinates parallel to the coast (the midshelf). (We use the CoRTAD gap filled data.) We can extract the midshelf and open ocean SST "lines" using matlab, but ideally would like to do it directly from the original dataset. Then we want to save the extracted multi-point time series in a netcdf file for general use by researchers interested in coastal upwelling. We then want to update this historical sst upwelling index on a weekly or monthly basis into the future. The latter effort requires that we work with a gap filled SST dataset generated by ourselves or in collaboration with others. In addition, I would like to bring to your attention results of a recently published paper which you might find interesting, or know others who would be: "Correlating Extremes in Wind Divergence with Extremes in Rain over the Tropical Atlantic" doi: 10.3390/rs14051147
Thanks for useful video you have shown how to extract a location for one pair of lat and longitude. how we can do this for a series of Latitude and longitude please
Good question, with CDO you would need to put this in a bash loop to create a set of individual point files, I'll post how to do this asap in the video description. You can also do in xarray of python. Note that to put the series of points in a single file implies the use of *unstructured* grids, as the usual structure for netcdf is lon/lat gridded. This is a bit advanced and will need a separate video. Probably easier to stick to a set of single point separate files for now.
Hi Adrian, I've extracted data using the same command you're talking from a .nc file using cdo. Again, I extracted the data for the same location from the same .nc file using ArcGIS. Then I compare the both outputs and they are different by far margin. It is observed that values extracted by cdo is much more higher than the values extracted by ArcGIS? Can you please tell me why I'm getting such differences?
@@hajarelhlaissi914 No, I didn't. I was looking for a specific point. However, it is possible to extract data for a specific country. There are two ways you can do the job. Firstly, clipping from the .nc file by the country's shape file, and secondly, extracting the grid coordinate from .nc file and then using those specific coordinate you can extract the data from the same .nc file.
I have found these tutorials only now, unfortunately. They are very instructive but at one point I cannot follow anymore as the function cdo seems not to exist anymore. Will you update that?
Would you please help me with how to install CDO since I am suffering from the error of the package (Proj, nethdf, etc ) I have tried reinstall them but the same, I am using Window10,Toshiba
First of all - have you enabled the linux subsystem under windows 10 ? There are many guides to show you how to do this (e.g. www.ssl.com/how-to/enable-linux-subsystem-install-ubuntu-windows-10/) - Once that is enabled it is then very straight forward to install netcdf and cdo packages using the standard "sudo apt install cdo" etc commands
I want to extract temperature values from netcdf file for Southern Africa (1979-2017) ..to draw Taylor diagram (Era-5 and Model) do I chose a single point(lat/lan in Southern africa) or do i chose the whole area( lon-lon / lat lat) and average then extract?
Well that really depends on what you want to validate. Do you have a station or a field site you are focusing on? In general more averaging reduces noise to reveal the signal, for example, at ecmwf we usually needed to test model changes with at least 6 months of forecasts to see if large area (hemisphere or continent scale) skill scores were statistically significantly better or worse.
nn=nearest neighbour and bil=bilinear interpolation - thanks for pointing that out and sorry it was not clear... I will put a pop out link in at that point to my other video on interpolation and regridding which explains all those techniques.
Thank you Sir 🙏 from India
you are very welcome :-)
@@climateunboxed sir can i have your email id ?
Thank you for the walk through but I am confused as to how only a point is enough to extract a section out, shouldn't we be specifying 2 points instead that can meaningfully define a rectangle of which we want to extract the data?
Thank you so much for all video you did it's very helpful. This two method used to extra the time series dataset at specific location they make some interpolation for the data, I would like to ask if there is another methods we can use to select the time series data at specific station from netcdf without interpolation?
Thanks for your kind feedback. If you use remapnn,lon=x/lat=y this does not make any interpolation and simply extracts the nearest grid cell series. I apologize if that wasn't clear in the video. You can use the lon lat arguments with other remap interpolation methods but of course they interpolate to get the series at your specified point.
@@climateunboxed Thank you very much. Now I can use the remapnn,lon=x/lat=y , instead of remapbil,lon=x/lat=y.
Dear Adrian Tompkins,
Thank you very much for these set of videos, they are the most accessible and valuable material about reanalysis data I've found!
I will leave some questions about my doubts on videos comments. Thank you very much!
About this video, I would like to ask about convert the output of the location info, for a CSV file.
Also another doubt I have is, if we have more than one variable in the same NetCFD/GRIB file. Here you have the 2m temperature variable, but imagine we have also 2m dewpoint temperature, 2m temperature, 10m u-component of wind, 10m v-component of wind, total precipitation, etc. How to convert them to a CSV file (one column for each variable).
Lastly, I don't know if it is possible, but how to convert the Units, for instance, from Kelvin to degrees celsius.
Thank you very much!
Dear Pedro. ncdump gives a text output but to convert to CSV you need to use R or python etc. However there is not really a reason to ever need to have CSV. It is much easier to work directly with the netcdf format. Lastly to convert to C use cdo expr,'Tc=Temp-273.15' in.nc out.nc there is a video coming on expr soon....
Thank you, thank you, sir. Your clips are very useful. I want you to ask a question. I hope you will answer me as soon as possible. Is it possible to extract the country of Morocco from this map? and how?
you can select a lon-lon rectangle region from a dataset very easily, please see this video for details ua-cam.com/video/ewii1p1kChw/v-deo.html - if you want to mask out the sea points that is also possible and will be the subject of a future video soon
Hello mr. Tompkins,
What is the order of the videos? I think the playlist is a little disordinate (vg the video #3 "Extracting subregions..." uses nc dump, that is explained in video #5 "Using ncdump...")
Love this channel and already passed the word :D
oh my, you are correct... I'm still learning the youtube ropes - I'll sort that out soon, new video arriving this week I hope. I soon I will launching a new openEDX course that threads all the videos together with exercises to do.... Thanks so much for spreading the word :)
@@climateunboxed May I have a link to your openEDX courses, please?
@@juliocgfreitas as soon as it is ready i will gladly post it. Things have been a bit hectic over the summer with grant applications. I'm really behind on the video front. Sorry, I really hope though to be able to post more material very soon
Thank you for the tutorial, this is so helpful!. One question, which one is more recommended to use? NN or Bilinear? is there any pros and cons for each method?
as I say in the video, it really depends on the fields, the relative changes in resolution and also what you want/need to do. There is not a "best" choice I'm afraid, it is very task specific.
Nice !
Thanks :)
Love all your videos. Once ready - Share the openEDX course details. One doubt - how to store this single 'point' data ina .txt or .csv?
Neither! Keep it in netcdf format, that way you have all the metadata and you can still read it easily. There is never a good reason to convert to a non self describing format... Unless you want to read it in excel i suppose, in which case you probably want CSV 😱
Thanks for the nice video , how to save the time series as a excel or csv file please
I love your videos Sir ,really helpful! I faced netcdf data whose time coordinate is showing 0000-00-00 00:00:00 when I employed cdo seinfo ifile ofile. I wanna it to have the same time coordinate to another netcdf file having the same variable . I am grateful if you help. Thanks.
Sir, but If I have a continent grid shapefile with defined cell number and their location (e.g., cell number 560990 lat 43, lon= 37) and I want to extract my NetCDF climate variable for each grid. What should I do? Thank you in advance.
CDO can't handle shapefiles unfortunately. I use Python to work with those...
HI, I want to do this for a list of lat, lon locations - namely sst parallel to the Portuguese coast. That is, along a curve. Is there a simple way to do that?
Oh nice question. You would need to use a distance-to-coast dataset for this. I would know how to mask all the parallel points. Would that be enough? What do you want to do with the data after? Upload this to stack overflow and tag it cdo-climate, then I can give the answer the full treatment...
@@climateunboxed Hello Adrian. I only just discovered "cdo" and the past couple of days watched a few of your youtube tutorials. Let me give you more background. We are working with satellite observations of SST (pathfinder 5 series) and specifically a dataset of weekly SST created by Selig et al for their project on coral reefs (CoRTAD). Our interest is in generating an upwelling index based on the difference between coastal SST and open ocean SST. We have a set of (lon,lat) coordinates parallel to the coast (the midshelf). (We use the CoRTAD gap filled data.) We can extract the midshelf and open ocean SST "lines" using matlab, but ideally would like to do it directly from the original dataset. Then we want to save the extracted multi-point time series in a netcdf file for general use by researchers interested in coastal upwelling. We then want to update this historical sst upwelling index on a weekly or monthly basis into the future. The latter effort requires that we work with a gap filled SST dataset generated by ourselves or in collaboration with others.
In addition, I would like to bring to your attention results of a recently published paper which you might find interesting, or know others who would be: "Correlating Extremes in Wind Divergence with Extremes in Rain over the Tropical Atlantic" doi: 10.3390/rs14051147
Thanks for useful video you have shown how to extract a location for one pair of lat and longitude. how we can do this for a series of Latitude and longitude please
Good question, with CDO you would need to put this in a bash loop to create a set of individual point files, I'll post how to do this asap in the video description. You can also do in xarray of python. Note that to put the series of points in a single file implies the use of *unstructured* grids, as the usual structure for netcdf is lon/lat gridded. This is a bit advanced and will need a separate video. Probably easier to stick to a set of single point separate files for now.
@@climateunboxed Hi Professor have you posted the video or can you please guide
Hi Adrian,
I've extracted data using the same command you're talking from a .nc file using cdo. Again, I extracted the data for the same location from the same .nc file using ArcGIS.
Then I compare the both outputs and they are different by far margin. It is observed that values extracted by cdo is much more higher than the values extracted by ArcGIS? Can you please tell me why I'm getting such differences?
Sir, did you extract a specific country from this map??
@@hajarelhlaissi914 No, I didn't. I was looking for a specific point.
However, it is possible to extract data for a specific country. There are two ways you can do the job. Firstly, clipping from the .nc file by the country's shape file, and secondly, extracting the grid coordinate from .nc file and then using those specific coordinate you can extract the data from the same .nc file.
@@turning_point96 Yes, thanks, but I have a problem with extracting some areas
@@hajarelhlaissi914 What's that?
@@turning_point96 I extracted the country of Morocco, but I did not find all some areas
I have found these tutorials only now, unfortunately. They are very instructive but at one point I cannot follow anymore as the function cdo seems not to exist anymore. Will you update that?
CDO is still very much alive and kicking! 😉 Which platform are you on?
Would you please help me with how to install CDO since I am suffering from the error of the package (Proj, nethdf, etc ) I have tried reinstall them but the same, I am using Window10,Toshiba
First of all - have you enabled the linux subsystem under windows 10 ? There are many guides to show you how to do this (e.g. www.ssl.com/how-to/enable-linux-subsystem-install-ubuntu-windows-10/) - Once that is enabled it is then very straight forward to install netcdf and cdo packages using the standard "sudo apt install cdo" etc commands
I want to extract temperature values from netcdf file for Southern Africa (1979-2017) ..to draw Taylor diagram (Era-5 and Model) do I chose a single point(lat/lan in Southern africa) or do i chose the whole area( lon-lon / lat lat) and average then extract?
Well that really depends on what you want to validate. Do you have a station or a field site you are focusing on? In general more averaging reduces noise to reveal the signal, for example, at ecmwf we usually needed to test model changes with at least 6 months of forecasts to see if large area (hemisphere or continent scale) skill scores were statistically significantly better or worse.
@@climateunboxed Thanks
I am focusing on a field site. I guess i'l have to select a box then calculate a field mean. I get it now...Thank you so much
What does method (nn and bil) mean here
nn=nearest neighbour and bil=bilinear interpolation - thanks for pointing that out and sorry it was not clear... I will put a pop out link in at that point to my other video on interpolation and regridding which explains all those techniques.