Robust Anomaly Detection + Seasonal-Trend Decomposition : Time Series Talk
Вставка
- Опубліковано 26 сер 2020
- Using the popular seasonal-trend decomposition (STL) for robust anomaly detection in time series!
Code used in this video : github.com/ritvikmath/Time-Se...
Data used in this video : github.com/ritvikmath/Time-Se...
Agree with everyone else, absolute gold. Keep it on, cheers!
Thanks!
my favorite video about outliers. Absolutely simple, amazingly useful and so easy to implement! Thank you so much!
I've watched the whole 36 videos of this playlist and they were really really helpful..THANK YOU SO MUCH SIR !!
I've watched tons of math/data science videos before but always find something missing..finally settled on this channel! continue the great work Ritvik!
very few teachers can be addictive and u r one among them... completed the whole playlist in just 2 days... kudos to u...
You are actually the man, I wish you good fortune my friend
Man… you are amazing! Please never stop teaching this way… this is pure didactic…
Could you explore more videos on how to treat the outliers after detection as you showed with the mean approach from the previous video? Or the mean calc would be just enough to remove the outlier?
Great video, great information, appreciate your help
Big fan of your videos! Keep it up~
Thanks!
This is Gold GOLD, I take it back it's Platinum
Woow its so clear now!!!!. THANK YOU!!!
Such a great video, thanks for sharing it! It would be lovely if you can continuing creating this content, and explain LOESS.
Thank you! Will do!
You are a genius! Thanks for the insight.
Amazing explanation !!
As a data scientist, these techniques I found very helpful, because in real data you need to deal these issues, i would request him to add more videos from machine laerning from his experince , because his way is explain to so simple, I binge watched complete playlist.
Glad it was helpful!
Lovely small video but worth a million likes
thanks!
Nice video! In this example it might be best to do a multiplicative decomposition by taking a log first, which should get rid of the heteroscedasticity you see in the seasonal and residual components.
This video is amazing...
i need this video but done in R, great video thank you
Great video! Would love to see a LOESS explanation :)
Amazing, thank you so much
Absolute Gold content..
Thank you for making this video. Also, can you please make a video on how LOESS work? It will help me in understanding what is going on behind the scene.
Thanks,
Bhuvanesh
Thanks! And yes I am planning to make that LOESS video soon.
Thanks for the video, especially the visuals. Though, do you know any similar command in Stata?
Excellent
A quick search shows Ben and Jerry's released a limited batch of Chocolate Cherry Garcia ice cream in late 2016.
Amazing work bro! I got a quick question. If you want to account for future data, and you want to set an anomaly detection threshold, how can you do it? You can't really use lower and upper from the code as lower gives u a negative value. How can we get the actual numbers from the data? Thx!
This is cool thanks. Is there a way to get the cyclicity as well in this package?
Thank you so much for the video! But I have a question on this. What if I want to make a daily anomaly detection check program? Should I calculate the standard deviation for the entire period every time? Or should I set a window size to calculate for the specific amount of time period?
Hello RitvikMath. Those thing seems so easy for you ..I envy you!! I have one big question for you. Is there a way to use Time series to pick the most "suitable" outcome in a set of finite possible outcomes we have. Not predict but find.
Briliant. I am new to data science and timeseries analysis in general, and i find this video very useful. My question is, how can I quantify or assess the 'strength' of my data's seasonality? more specifically, I want to write a code that automatically detects if my data is seasonal or not. an example of a dataset i want to work on is hourly temperature fluctuations (high temp during the day, low temp during the night). How can I automate a test that checks whether this data follows a seasonal trend or not, and if so at what frequency? Thanks!
Crystal clear...thanks for the video...can you please make a video of LOESS theory part? Thanks in advance
Once you've identified these anomalies would you consider doing anything with them as they could affect your forecast? Would you consider removing the data points? Or maybe replacing them with something else?
is small residuals (remainder) a good thing? I sometimes found stl, x11, seats retrurn different scalse of seasonality and remainders. How to evaluate a decomposition model? Thanks
I often get confuse when interpreting the seasonal_decompose plot. How can we certainly know if the seasonality is daily, weekly, monthly or yearly? I would appreciate a lot your answer
Good video, make one for cyclic decomposition
could u pls cover robust pca for anomaly detection in timeseries, heard its really effective at large scale
Thanks for wonderful Video on STL . I just want to ask you, I am working on real world dataset.When I try to decompose time series into trend and seasonal components, I need to specify period value, but you didn't do that. what is that?
Great video. One clarification - When creating the lower and upper limit thresholds, wouldn't you want to use the mean and standard deviation of the residuals UP TO (but not including) each point. Otherwise you are using future information, which wouldn't allow you to determine if an event is an outlier in real time.
yes great point !
@Shikamaru23... Can you explain how to do this
Can't find any info on how to set "sesonal" and "period" parameters
How about time series in which the more general trend changes, goes up for 6 months, goes down for 4 months, then up again for 5 months, etc?
How do we configure to detect the Anomalies in real time (as soon as possible).. Will there be a delay in identifying if say yesterday's data had an anomaly for a daily frequency dataset
Bro.. Any help would be great!
Hi @ritvikmath, do you trade stocks using time series analysis?
Yes! I'm planning to soon make some videos on how I choose stocks.
@@ritvikmath awesome!
@@ritvikmath hi Ravikant , I really learnt good stuff from your previous videos. I really love watching time series .
@@ritvikmath Can't wait!
Does STL decomposition works as an additive model ?
I used it in a production delivery, lol
@@komalsaini5668 I used my own dataset so so I skip this part, but maybe you could calculate the frequency with a groupby().count() instead of infer_freq
bro your videos are super easy to understand pls marry me no homo
Hahaha nice event called "ice cream is gay"
Ice-cream is gay )))) . Not the central point of the video but anyways, also an anomaly.
can I make my data stationary by removing the trend part for instance? I mean if my data is a called df the do: df - df.trend (in a pseudo code)
when i use plt.plot() to print the base plot it says unhashable type: 'numpy.ndarray'
Hey,
ice_cream_interest = ice_cream_interest.asfreq(pd.infer_freq(ice_cream_interest.index))
this set the column `interest` to NaN
Do not know what I'm doing wrong.
Hi, I am facing the same issue. Did you find a solution?
@@somyatripathi6164 Try loading the csv with "ice_cream_interest = pd.read_csv('ice_cream_interest.csv',parse_dates=[0], index_col=0)".
@johnysmith1375 , this will work:
ice_cream_interest = pd.read_csv('./data/ice_cream_interest.csv')
ice_cream_interest["month"] = pd.to_datetime(ice_cream_interest["month"]) # add this line
ice_cream_interest.set_index('month', inplace=True)
ice_cream_interest = ice_cream_interest.asfreq(pd.infer_freq(ice_cream_interest.index))