Відео

You can look at the definition of the standard name which is: cfconventions.org/Data/cf-standard-names/current/build/cf-standard-name-table.html surface_temperature_anomaly The surface called "surface" means the lower boundary of the atmosphere. "anomaly" means difference from climatology. The surface temperature is the (skin) temperature at the interface, not the bulk temperature of the medium above or below. It is strongly recommended that a variable with this standard name should have the attribute units_metadata="temperature: difference", meaning that it refers to temperature differences and implying that the origin of the temperature scale is irrelevant, because it is essential to know whether a temperature is on-scale or a difference in order to convert the units correctly (cf. cfconventions.org/cf-conventions/cf-conventions.html#temperature-units). Climatology refers to a 30 year period 1961-1990 as described here in the CF conventions doc cfconventions.org/Data/cf-conventions/cf-conventions-1.11/cf-conventions.html#climatological-statistics

You're welcome!

Glad it was helpful!

Sorry, I only provide help with FAIR compliant data formats. XML files could potentially be FAIR if the contents and structure adhere to commonly used conventions. Otherwise they are just a container.

@@LukeDataManager It would be helpful if you make video on filtering of data and how to remove the errors as I want the point data basically the coordinates.If you can provide me your mail it would be really helpful.

@naureenfatima9715 sorry I don't have time for individual requests.

@@LukeDataManager Okay . No problem

Thanks, something like this ax.set_extent([-30, 60, 30, 75], crs=ccrs.PlateCarree()) # Longitude and latitude

Great to hear things like this, good luck with it! Nice job!

Hope so!

Thanks. You might find my video on creating a netcdf file useful. You can find it in the same playlist and in the description to this video.

Always great to hear that, thanks and good luck in your new job!

You're welcome!

Pretty much all python code can work in jupyter notebook. Regarding your data it really depends what you are trying to achieve.

Thanks for the comment. Your question regarding data analysis is a bit outside of my area, but good luck!

Thanks!

In this code anom <- var.get.nc(data, 'anom', start=c(NA, NA, 1, time_index), count=c(NA,NA,1,1)) Count is the number of values. So you need to find the index of the 'end date' and then subtract the index of the 'start date' to give you the number of elements. There is a new library in development in R that will allow you to do this by only entering the dates themselves. github.com/pvanlaake/ncdfCF So watch this space!

'Pip install xarray' in your terminal

Very unusual. Where can I find the data?

​@@LukeDataManager You can download from ocean color web site, is a NASA page, this is the link where you can get a .nc file : oceancolor.gsfc.nasa.gov/cgi/browse.pl?sen=amod or if you want a can send you a email with the file attached

You can fix the `AttributeError` by following these steps: 1. **Check for Circular Imports**: Ensure there are no circular dependencies in your imports. 2. **Correct Import Statement**: ```python import xarray as xr dataset = xr.open_dataset('your_dataset_file.nc') ``` 3. **Avoid Module Name Conflicts**: Ensure your script is not named `xarray.py` or similar. 4. **Verify Environment**: Make sure you are using the environment where xarray is installed.

This is in preparation. The code is here lhmarsden.github.io/NetCDF_in_R_from_beginner_to_pro/04_creating_a_cfnetcdf_file.html I will be making the video over the summer.

Can you provide a link to the data?

@@LukeDataManager it is using Copernicus Marine Service data product, velocity, SLA, Wind etc. I am interested in making plot using R. May I get you personal website or email to further contact? please

@@abdulkarim9055 I have limited time or funding for one-on-one tuition - these videos are only a small part of my job. I won't add my email address here or I will receive a lot of requests for help - I already receive quite a lot these requests. I hope you understand. I think it is a better use of my time to provide tutorials that many people can use. But if you leave a link to the data here in the comments I will take it into consideration.

You're welcome!

Ja

Islansk - but have Norwegian relatives

@@haraldurkarlsson1147 I am not norwegian but live and work in Norway

I see. But still Norway is a great place to live and work in.

700 in this case is the 700th element in the list. Alone this tells you nothing about what the time is. The value of the 700th element along with the units can be used to calculate the time.

NA means all values. 1 means the first value (in start) or only one value (in count)

@@LukeDataManager Somewhat confusing since NA has a different meaning in R.

@haraldurkarlsson1147 NA stands for not available. I think there is some consistency with how it is used here. NA as in don't include a start or count for that dimension.

Do you have an example of a netcdf file created with either RNetCDF or ncdf4 that you can't open and process with the other library? In my experience, the files created by either program are interoperable and usable by the other.

I have downloaded climate data twice - some time ago and yesterday. With the older file I used ncdf4 and with the more recent one RNetCDF. For the first I ran: gistemp <- nc_open(here("data", "climate_data","gistemp1200_GHCNv4_ERSSTv5.nc")). In my Global Environment it is a ncdf4 type object. For the second one I used: glob_temp <- open.nc(here("data", "NOAAGlobalTemp_v6.nc")). This file is listed as a NetCDF object type in my Global Environment. When I attempt to use ncdf4 functions on the NetCDF file I get an error message and visa versa. The error message for ncdf4 trying to act on the NetCDF file is: Error in ncatt_get(glob_temp, "anom", "standard_name") : Error, first passed argument must be an object of class ncdf4

Because many sections of the scientific community use R already. And why not use R?

CF-NetCDF files can be fully FAIR compliant and therefore a lot of important scientific datasets are published in CF-NetCDF. Many other file formats like HDF are good containers but the data and metadata within must follow well defined and commonly used conventions, and structured in a standardised format to be fully FAIR compliant. Some conventions are not mature enough, yet at least.

Thanks

You can use the `interp` method in xarray to interpolate the data from 1 degree to 0.01 degrees. Here's how you can do it: ```python import xarray as xr # Load your xarray dataset # For example, assuming your dataset is named 'data' Let's imagine you have an xarray object called 'data' data_interp = data.interp(lat=range(data.lat.min(), data.lat.max(), 0.01), lon=range(data.lon.min(), data.lon.max(), 0.01)) However, one problem is that you also need to interpolate the data between 359 degrees and 1 degrees. Luckily for you, I have a function you can use for this. In the below function the data are sampled at 2.5 degree intervals and I am interpolating to 0.5 degrees. Method should be one of "linear", "nearest", "zero", "slinear", "quadratic", "cubic", "polynomial" def interpolate_data(ds, method): ds_90_to_270 = ds.sel(lon=slice(87.5, 272.5)) ds_90_to_270_interp = ds_90_to_270.interp(lat=np.arange(-90, 90, 0.5), lon=np.arange(87.5, 272.5, 0.5), method=method) ds_90_to_270_interp = ds_90_to_270_interp.sel(lon=slice(90, 270)) ds_90_to_neg90 = ds_90_to_270_interp.assign_coords(lon=(ds_90_to_270_interp.lon + 180) % 360 - 180) # Combine the interpolated parts ds_0_to_90 = ds.sel(lon=slice(0, 92.5)) ds_270_to_360 = ds.sel(lon=slice(267.5, 360)) ds_combined = xr.concat([ds_0_to_90, ds_270_to_360], dim='lon') ds_neg90_to_90 = ds_combined.assign_coords(lon=(ds_combined.lon + 180) % 360 - 180) ds_neg90_to_90_interp = ds_neg90_to_90.interp(lat=np.arange(-90, 90, 0.5), lon=np.arange(-92.5, 92.5, 0.5), method=method) ds_neg90_to_90_interp = ds_neg90_to_90_interp.sel(lon=slice(-90, 90)) interpolated_ds = xr.concat([ds_neg90_to_90_interp, ds_90_to_neg90], dim='lon') interpolated_ds = interpolated_ds.sortby('lon') return interpolated_ds

For multiple files just include that within your for loop. The execution of the function I mean. Defining the function can go at the top of your code.

Glad you think so!

Thanks!

This really depends on which order you want to open your files. You could manually create a list of filenames if you have a specific order in mind and do a for loop through that list.

ok，thank you very much

There has been some discussion on this over the past few years. github.com/Unidata/netcdf-c/issues/2173

@@LukeDataManager thank you sir for the reference. I have read that but I can't understand anything since I am not majored in computer science or anything related that (I am majored in environmental engineering). From what I've gathered, we can compress NetCDF files using nccopy, ncks (NCO), CDO, and nccompress. I need to find a better NetCDF compression method because I have very limited resource to store 10-year worth of ERA5 data for my regional climate model. I would be thankful if you could give me some insight for that. Thank you so much.

I think zstandard is possible in NetCDF. For example if you are using the netcdf4 li rary in Python you can read about different compressions in the section 'Efficient compression of netCDF variables' on this page unidata.github.io/netcdf4-python/

Thank you! I agree in principle with your comments regarding moving from NetCDF to .bil but I am not familiar with the dataset or providers so I'm reluctant to say too much.

@@LukeDataManager Fair enough! Thanks!

Seems to me that NOAA are having some issue with their THREDDS server right now. Try here: www.ncei.noaa.gov/data/noaa-global-surface-temperature/v6/access/gridded/ And you should be able to download v6 of the data in a NetCDF file

@@LukeDataManager When I reach this part ''' desired_date = '2022-12-01' data_for_desired_date = xrds.sel(time=desired_date) print(data_for_desired_date) ''' It gives TypeError: get_loc() got an unexpected keyword argument 'method' Did I miss to import some libraries?

It worked in Colab. I might have missed something in local device.

Is it a depth profile or geospatial data like the ones in this video? You can use xrds.sel to obtain a subset of an xarray object for your desired latitude and longitude.

docs.xarray.dev/en/latest/generated/xarray.Dataset.sel.html

Sure, send me the details.

@@LukeDataManager Thanks Luke, Do you have an email where I can reach you on ? Thanks for great videos. Rasmus

You found me. I don't like to share my email on UA-cam comments if possible. But I am on LinkedIn and respond to messages there.

Glad you think so!

Do you mean if you have multiple files on your computer? Then you need to open them one by one in a for loop for example. You might find this video useful: ua-cam.com/video/AbLRV5YUW2g/v-deo.html In it I extract data from multiple NetCDF files that are hosted on a THREDDS server. But you can adapt the code to loop through files on your computer instead of a THREDDS server. Just use the relative or absolute filepath to your netcdf files instead of the OPeNDAP URL.

@@LukeDataManager Thank you. I will try

I am not entirely sure what you mean, and pressure is not provided in either of the datasets. When looking at the variables, you might see for example CHLOROPHYLL_A_TOTAL (DEPTH) float64 .... In this example, CHLOROPHYLL_A_TOTAL is the variable name, DEPTH is the name of the dimension, and float64 is the format of the values in the variable.

@@LukeDataManager ohh..yes I was talking about (DEPTH), and I got it now that the data within the file is organized and indexed along this dimension. Thanks!

Glad you solved it!

I'm glad you think so!

What type of data is it? And where are you based?

There are hundreds of data centres. Choosing one is hard. There are a few things to consider when making your choice. 1. Are you aware of any data access portals that might be relevant for your data? It is not practical for data users to look through all the data centres for relevant data. Data access portals aim to make all relevant data available in one place by providing a catalogue of access points to datasets in contributing data centres. A data access portal might focus on a certain region or a certain type of data. Contributing data centres most likely follow commonly used standards for metadata and access methods to make this possible. They are interoperable. Other data centres (like zenodo) do not. If one data access portal can point to one data centre, other data access portals can also point to that data centre using the same method. 2. Can you speculatively find data in a data centre you are interested in? Can you search by coordinates, time the data were collected, some keywords? If not, avoid that data centre. 3. Some data centres specialise in certain types of data. E.g. GBIF is a good place for biodiversity data and associated measurements, genbank for genome data. 4. Can the data centre ensure your data will be preserved over decades and centuries? Some private companies or projects might lose funding. What then? What will happen to your data? Does the data centre have a plan? Often national data centres that are funded by national research funding are considered quite secure forms of funding. This is a difficult question to answer but if you let me know more about your data I can suggest a data centre. Pangaea is good and quite broad, and is harvested by many access portals. But there are likely other options for you too.

My pleasure!

Thanks!

Thanks, that is an interesting suggestion. Are there any particular models within the CMIP6 data that you think I should focus on? Looks like there is a lot of different data in their THREDDS catalogue: ds.nccs.nasa.gov/thredds/catalog/AMES/NEX/GDDP-CMIP6/catalog.html

@@LukeDataManager Thank you for your reply. I checked the link you shared. Could you make a video by using precipitation from "MRI-ESM2-0" and visualizing its monthly and yearly precipitation? Please use a folder for example MRI-ESM2-0/ssp585/r1i1p1f1/pr/ in the THREDDS catalogue.

@@user-asam2 I will consider making this. I have been thinking about making videos on specific, in-demand datasets for a while and this is a good example, so thanks! I might focus on several things in CMIP6 rather than being too specific but hopefully it will be useful. And I will consider using the precipitation data you suggested as an example. Though I have quite a long list of tasks and this has to be done outside of work hours so be patient :)

@@LukeDataManager Thanks! Manupulating precipitation in CMIP6 is very helpful. Sure, I understand you have many tasks to do. No pressure. Thank you so much for such informative videos!

I'm not entirely sure what you mean but maybe you will find this helpful. import xarray as xr Load your dataset (replace 'filename.nc' with your actual file) ds = xr.open_dataset('filename.nc') # Suppose your latitude range is from 20 to 30 degrees, longitude range is from -10 to 10 degrees, and time range is from '2024-01-01' to '2024-01-31' Selecting latitude range from 20 to 30 degrees lat_range = ds.sel(lat=slice(20, 30)) Selecting longitude range from -10 to 10 degrees lon_range = lat_range.sel(lon=slice(-10, 10)) Selecting time range from '2024-01-01' to '2024-01-31' time_range = lon_range.sel(time=slice('2024-01-01', '2024-01-31'))

@@LukeDataManager i can do it individually, so scratch the multiple data points part. However, the year, month, time, and hour are each individual variables.

Interesting. I think I would be tempted to extract each variable as numpy arrays first and then combine them. import numpy as np years = np.array([2024, 2024, 2024]) months = np.array([2, 2, 2]) days = np.array([28, 29, 29]) hours = np.array([10, 12, 15]) # Create datetime array timestamps = np.array([np.datetime64(f'{y}-{m:02}-{d:02}T{h:02}') for y, m, d, h in zip(years, months, days, hours)]) Then you can add that numpy array to your dataframe if you need to. df['timestamps'] = timestamps Assuming you have one row per time. If not, create another dataframe of your year, month.... first and then merge the dataframe together based on a shared column or list of columns

And if you are ever creating a netcdf file don't encode time like that! There is a section the next video in the series about how to encode time in CF-NetCDF.

Thanks, I am working on a series of videos entitled something like 'NetCDF files in Python: from beginner to pro'. Should be 7-8 videos, to be released next year

You are welcome!