This was really interesting, thanks ! I recently stumbled across the same issue, and I started wondering whether it was valid to just ignore the "subtract the mean return" step. Your presentation makes it much clearer, thanks. I really like the distinction between correlation of price and correlation of returns, that made everything much clearer. Thanks again :)
Interested to hear more on both points. I would think that since cumulative returns are in sync (given they are indexed by date), that the relational information would be best realised from a non elastic distance metric as opposed to DTW which allows warping. Unless you mean trying to capture lead-lag relationships perhaps?
@@riandolphin DTW {O(N**2)} has a much higher degree of flexibility than ED {O(N)} in comparing time series at higher computational cost. Time Series can exhibit varying speeds, non-uniform/mixed sampling rates, local distortions, breaks etc. Because DTW has an optimal alignment mechanism which allows it to differentiate between "true" dissimilarity vs low dissimilarity with temporal misalignment. ED can be made extremely fast but suffers from a very simple one-to-one mapping and thus, cannot handle most temporal dynamics and is extremely sensitive to outliers/ noise. Noisiness can be tempered using wavelet de-noising. Using weighting schemas can reduce ED's sensitivity to a range of outliers. Nonetheless, even with these modifications, ED is highly limited and "biased" (under temporal distortions) in comparison to DTW. DTW is awesome "correlative class" similarity analysis that should be supplemented with domain specific knowledge/ causal understanding because leveraging on DTW results can be problematic if OOD occurs.
This was really interesting, thanks ! I recently stumbled across the same issue, and I started wondering whether it was valid to just ignore the "subtract the mean return" step. Your presentation makes it much clearer, thanks. I really like the distinction between correlation of price and correlation of returns, that made everything much clearer. Thanks again :)
Happy to hear you found it interesting!
DTW on cumulative returns are better. Also i would suppose a similarity in causal structure would be more rigourous for OOD generalizability
Interested to hear more on both points.
I would think that since cumulative returns are in sync (given they are indexed by date), that the relational information would be best realised from a non elastic distance metric as opposed to DTW which allows warping. Unless you mean trying to capture lead-lag relationships perhaps?
@@riandolphin DTW {O(N**2)} has a much higher degree of flexibility than ED {O(N)} in comparing time series at higher computational cost. Time Series can exhibit varying speeds, non-uniform/mixed sampling rates, local distortions, breaks etc. Because DTW has an optimal alignment mechanism which allows it to differentiate between "true" dissimilarity vs low dissimilarity with temporal misalignment. ED can be made extremely fast but suffers from a very simple one-to-one mapping and thus, cannot handle most temporal dynamics and is extremely sensitive to outliers/ noise. Noisiness can be tempered using wavelet de-noising. Using weighting schemas can reduce ED's sensitivity to a range of outliers. Nonetheless, even with these modifications, ED is highly limited and "biased" (under temporal distortions) in comparison to DTW. DTW is awesome "correlative class" similarity analysis that should be supplemented with domain specific knowledge/ causal understanding because leveraging on DTW results can be problematic if OOD occurs.
Shouldn't the Pearson correlation be calculated on a sequential data? Order does matter.
Absolutely, order does matter when calculating correlation, and it is calculated on sequential (time ordered) data in this work.
👍👍👍