Thank you so much ! I've been looking for these types of information for weeks. This is so far the mos informative stuff I watched. I just need to get how to apply it with R now hahaha. (Forest scientist trying to work with spectral data here).
Always makes me happy when it helps someone! In the following link I started writing R scripts for SNV, MSC, and normalization. May not be optimal but they will give you a good start. www.uprm.edu/ccs-cicsa/files-info-for-research/r-language-resources/ Have a great day! - Edwin
I also mention Orange. It is a Python-based free software and it has a Spectroscopy add-on, which is excellent for the spectra visualization & preprocessing. The respective tutorials are encountered in UA-cam too.
Professor, which one from the programs you've mentioned on the video that can "mass" preprocess spectra? like can do multiple spectra in one touch Thank you
Hello! Thank you! The best video in the net on data pre-processing I found so far! In Unscrambler, when you run the PCA based on the original data, you have the choice to select mean centering, which I think is meant to get rid off scattering. Would you in all,cases preprocess data with SNV and maximum centering?
Hello! Thank you for your kind words. The choice depends on what I (Edwin) want to study with my spectra. For presentation I would use baseline correction and SNV to maintain as much as the original shape of the spectra as possible. For creating models, I usually use 12 different combinations of DP methods. Use SNV, MSC, SG1, and SG2 separately and then combine them in different orders (SNV+SG1, SG1+SNV, MSC+SG1, etc.). Once I have a matrix with each different DP method, I create a model for each different matrix. This way I can determine which combination gave the most optimal results. However if the artifact can be seen clear as day on the data you can simply use the DP method that best reduces the variation. Hope this helps! We are welcomed to any suggestions and/or corrections. Hope you have a great day!
Hello, an excellent video. The best explanation of spectral preprocessing techniques I've seen in years. Taking advantage of the occasion, I have a question that I would appreciate if you would help me by clarifying it. All these current techniques are on the spectra individually (rows), but in the literature, I find other techniques, such as mean centering, auto-scale, and variance scale, among others, that act on the variables (columns). In some manuals, I found the latter necessary because several multivariate algorithms compute results driven by variance patterns in the independent variables. Specifically, my question is: When to use tools such as mean centering, auto-scale, and variance scale, or are they already integrated into techniques such as PLS?
Thank you for your kind words. Hope you are doing well. Mean centering is used to remove the common information on your data. Chemometrics assumes that variation implies information, hence why mean centering is so useful. When using PCA or PLS, mean centering the data allows the average variation between samples to be placed on the origin (0,0) of the scores plot. Auto scaling is usually used to leave the mean at zero and the standard deviation at one. From what I've understood, you scale variance when the variables that are being analyzed have very different magnitudes. This make some variables overshadow others, hence dividing the std allows them to play on an even field. Hope this gave some insight. Regards! - Edwin
Thank you so much for this informative and well-explained session. I have been trying to write some code in MATLAB to perform MSC, could you please help with a script to execute that? Thanks
Hello , thank you for this nice video , for the scattering , you mentioned that one of reason for scattering is the molecule not at the same distance , so how can i make the molecule at the same distance if i prepared as example solution contain dye dissolved in water , i prepared different concentrations then i measured spectra for them and i got scattered for the data, thank you.
Thank you for your attention!
Comments?
Suggestions?
Recommendations?
All options are welcomed!
Very good & clear explanation. Thanks.
Man. I just found out that I need to preprocess my data for MSC, this channel has really helped me. Thank you for the good job!!
thank you for your very clear explanation. It helps me a lot
Thank you so much ! I've been looking for these types of information for weeks. This is so far the mos informative stuff I watched. I just need to get how to apply it with R now hahaha. (Forest scientist trying to work with spectral data here).
Always makes me happy when it helps someone! In the following link I started writing R scripts for SNV, MSC, and normalization. May not be optimal but they will give you a good start. www.uprm.edu/ccs-cicsa/files-info-for-research/r-language-resources/
Have a great day!
- Edwin
you dont need R that much.There's a Git link in which the codes for all these are already given
Wow that is awesome!!! The only point is the voice quality but your content and your presentation was awesome. Thanks a lot
I also mention Orange. It is a Python-based free software and it has a Spectroscopy add-on, which is excellent for the spectra visualization & preprocessing. The respective tutorials are encountered in UA-cam too.
Professor, which one from the programs you've mentioned on the video that can "mass" preprocess spectra? like can do multiple spectra in one touch
Thank you
Hello! Thank you! The best video in the net on data pre-processing I found so far! In Unscrambler, when you run the PCA based on the original data, you have the choice to select mean centering, which I think is meant to get rid off scattering. Would you in all,cases preprocess data with SNV and maximum centering?
Hello! Thank you for your kind words. The choice depends on what I (Edwin) want to study with my spectra. For presentation I would use baseline correction and SNV to maintain as much as the original shape of the spectra as possible. For creating models, I usually use 12 different combinations of DP methods. Use SNV, MSC, SG1, and SG2 separately and then combine them in different orders (SNV+SG1, SG1+SNV, MSC+SG1, etc.). Once I have a matrix with each different DP method, I create a model for each different matrix. This way I can determine which combination gave the most optimal results. However if the artifact can be seen clear as day on the data you can simply use the DP method that best reduces the variation. Hope this helps! We are welcomed to any suggestions and/or corrections. Hope you have a great day!
Hello, an excellent video. The best explanation of spectral preprocessing techniques I've seen in years. Taking advantage of the occasion, I have a question that I would appreciate if you would help me by clarifying it. All these current techniques are on the spectra individually (rows), but in the literature, I find other techniques, such as mean centering, auto-scale, and variance scale, among others, that act on the variables (columns). In some manuals, I found the latter necessary because several multivariate algorithms compute results driven by variance patterns in the independent variables. Specifically, my question is: When to use tools such as mean centering, auto-scale, and variance scale, or are they already integrated into techniques such as PLS?
Thank you for your kind words. Hope you are doing well. Mean centering is used to remove the common information on your data. Chemometrics assumes that variation implies information, hence why mean centering is so useful. When using PCA or PLS, mean centering the data allows the average variation between samples to be placed on the origin (0,0) of the scores plot. Auto scaling is usually used to leave the mean at zero and the standard deviation at one. From what I've understood, you scale variance when the variables that are being analyzed have very different magnitudes. This make some variables overshadow others, hence dividing the std allows them to play on an even field. Hope this gave some insight. Regards! - Edwin
Thank you so much for this informative and well-explained session. I have been trying to write some code in MATLAB to perform MSC, could you please help with a script to execute that? Thanks
Hello , thank you for this nice video , for the scattering , you mentioned that one of reason for scattering is the molecule not at the same distance , so how can i make the molecule at the same distance if i prepared as example solution contain dye dissolved in water , i prepared different concentrations then i measured spectra for them and i got scattered for the data, thank you.
am ok with data normalization (i.e., scaling), however I have a doubt concerning to the importance of DP methods for modeling!
Good quality content
can anyone point for me places to download spectral data for my research?? i am lokking for bearings spectrum data
Buenas tardes seria bueno que esots videos tambien los compartieras en español