Great video. Will you or can you provide additional information on other useful classifiers and also how to merge other data sources like news and sentiment into this code?
First of all, this whole economic chaos was powered by optimism that the FED is done with hiking interest rates. Now that interest rate crash is the situation, where do we go from here? How would you advise I safely allocate $250k funds at this point?
The market is volatile at this time, But doesn't the Federal Reserve's monetary policy and low interest rates contribute to the current valuations? hence I will advice you get yourself a financial advisor that can provide you with entry and exit points on the share/ETF you focus on.
Agreed, my portfolio is well-matched for every market season yielding 85% from early last year to date. I and my CFP are working on a 7 figure ballpark goal, tho this could take another year. IMO, financial advisors are the most sought-after professionals after doctors.
Mine's Rebecca Noblett Roberts. She turned out to be better and smarter than all the advisors I ever worked with till date, I’ve never met anyone with as much conviction.
Clear and to the point. I hate super long videos full of things that don't provide much value. This one was great. I like that he walked through general data science/machine learning steps. In particular the data cleansing which many skip over, but it is actually an important step. Also, a pet peeve of mine is audio quality. This video you can hear the presenter clearly and he doesn't sound like his is working from a tin can.
The features used for the random forest cannot be the high, close, low , open values directly without any transformation because what the model is essentially doing is creating a overfit of non linear decisions to certain prices ranges. It is basically memorizing that when the close was above X value and open below Y value predict 1 or 0. You need to normalize the predictors in some way so that the model can use them independently of how high the value the stock is and truly create generalizable rules. Ratios are good since they use percentage instead of using absolute values and allow the model to use information of multiple candles as well.
Excellent. This tutorial corrects an error that pretty much every other video from others that I have seen has made. Don't seek MSE precision in your target as your goal. That's not what practitioners are looking for. Do what this educator has done instead. This model gets it right as used in the real world. Solid base to work with. Well done!
Watched up to 2:26 and I already know this is going to be excellent. Clear and concise explanation from the start and you know this is going to be more than your ordinary YT tutorial
It's not excellent, you can't beat the market as regular person. You basically compete with Harvard graduates with math, computer science, etc. Degrees. Again, one UA-cam video won't make you beat the market
@alang.2054 Where'd you get that she said she would beat the market from her comment? I read an observation just stating that, this video is higher quality than most YT videos that claim to teach you something specific yet just give you fluff..
I’m new to coding but have always been an avid market watcher and looking for opportunities. Best video I’ve seen since I started scouring the depths of UA-cam for this content last week. Thank you sir!
What are you talking about? Do you really think this guy would show you real ways to make money? On market you compete with professionals in multi billion hedge funds with degrees, you can't beat them with UA-cam video
Before you start investing, it's crucial to understand the basics of investing, different asset classes (stocks, bonds, real estate, etc.), and the associated risks. are you investing for retirement, buying a home, or building an emergency fund? Your goals will help shape your investment strategy.
Don't put all your eggs in one basket. Diversification means spreading your investments across different asset classes (stocks, bonds, real estate) and within those classes (different companies or industries). This helps reduce risk.
I Invest in low-cost index funds or ETFs that track broad market indices, such as the S&P 500. These funds offer diversification across a wide range of stocks and can be a more passive, low-maintenance investment option. They are suitable for investors who prefer a hands-off approach and want exposure to the overall market.
I'm very cautious about giving specific recommendations as everyone's situation varies. Consider independent financial advisors like "Vivian Jean Wilhelm" I've worked with her for some years and highly recommend her. Check if she meets your criteria.
I've heard the stock market often does well during election years. I recently inherited a lump sum and want to invest it wisely to take advantage of this potential upswing. Do you have any tips or strategies?
A good portfolio should have three basic things: ETFs for diversification, dividend stocks for cash flow, and leading tech stocks. When starting especially with a lump sum, it's a good idea to talk to a fiduciary advisor for expert advice.
I believe every Investor should start with ETFs for a solid foundation, then diversify across asset classes and maintain disciplined, regular investing to minimize risks and maximize growth.
You don't need to find the next NVDA to succeed in investing. Just choose top-notch ETFs and partner with a financial advisor like I did. I turned $100k into $40,000 in annual dividends-a significant milestone for me today.
I'm cautious about giving specific recommendations as everyone's situation varies. Consider independent financial advisors like "Melissa Elise Robinson" I've worked with her for 6 years and highly recommend her. Check if she meets your criteria.
Impressive video, and very well narrated !!! I have certainly learnt more about PY and ML... I am curious as to taking the model and learning, knowing how to tweak it to improve its successful trade ratio (with the conditioned factors already in place) Thank you for taking the time to do this video...
I've always been interested in binary options, but I never knew where to start. Thanks to you and this video, I finally felt confident enough to give it a try.
Hint: on a recent macbook you can use all its cores by: import joblib N_CORES = joblib.cpu_count(only_physical_cores=True) ... model = RandomForestClassifier(n_estimators='your value', min_samples_split='your other value', random_state=1, n_jobs=N_CORES) The speedup is amazing
Vik, I echo the compliments on the excellent video. I was able to use my own bespoke weekly market timing signals aligned with weekly S&P closes to finally get a grounded statistical "opinion" on the predictability of forward returns - as only my second Python exercise! Thanks!
I'm hoping you can do a follow up video to this. Would be great to see how you would incorporate macro data into your model, such as news or interest rates.
when you split the data into the training and testing dataset, you are actually performing what is called Simple Random Sampling, this will cause the training data to have the same elements/characteristics of the testing dataset. If you were to calculate the means of each predictor variable in the testing and training dataset it will roughly be the same due to random sampling. The point I am trying to make is that you cannot claim the model has not "seen" the testing data, yet it managed to capture the majority of its properties due to simple random sampling, how about you train the model using the first 70% rows then leave the remaining 30% at the bottom for predictions? In that way the model does not have any idea what's happening with the remaining 30% (though there is an argument one can put forward about this), I think that approach would be the most realistic. I have used the simple random sampling before and I have gotten results which seemed to be accurate, it was not until I used this method I am suggesting to you that I obtained a little bit higher errors.
Wow, the concept of predicting the stock market using machine learning and Python is such a fascinating topic! The blend of finance and technology is always an area ripe for innovative approaches. It's impressive how machine learning can analyze vast amounts of data to find patterns that might not be obvious at first glance. Python, with its extensive libraries and community support, is an excellent choice for such complex computations. It's exciting to think about how these tools can provide insights into market trends and possibly even predict future movements. The intersection of machine learning and finance is definitely a space to watch! 📈💡🤖
Excellent video and you are above average by all means. You made things easier for me who is new to Python. At 65 yrs old I tried to work your script and it worked beautifully. So, I tried with TSLA ticker and it gave me no obj to concatenate error and I have no idea how to fix that error.
This is very nice way to get started using data science with the markets. This gives a nice framework to get started. And attempt to expand the predictors (on RSI based or Change in Open Interest , some correlation with the major stocks composing that index) . Thank you for sharing.
Congratulations for your explanation and it was very clear. I would like to suggest you to prepare a vide including news about the stock into this model. Thanks
Buying a stock is easy, but buying the right stock without a time-tested strategy is incredibly hard. I’ve been trying to grow my portfolio of $160K for sometime now, my major challenge is not knowing the best entry and exit strategies
Investors should be cautious about their exposure and be wary of new buys, especially during inflation. Such high yields in this recession is only possible under the supervision of a professional or trusted advisor.
I have been speaking with a coach for a long time now mostly because I lack the background knowledge and mental toughness to handle these reoccurring market conditions. I made over $220K during this drop, which proved that there is more to the market than the average person is aware of.
Actually you forgot to measure the expectancy of a trade in the case it has a precision of 42%. Because what makes a strategy profitable is bit the win rate but rather the expectancy of the trades. Although it is a great video and a good tutorial about programming. Thanks and keep up the good work.
Hi Vik, I love the way you work and describe at lightning speed so effortlessly. As you said, you have worked for a long time on this and I am quite sure you wouldn't give away your personal final model. But since you probably already included other stock markets and in particular news feeds, could you give me a hint to which kind of accuracy you got up to? I am not asking for any kind of code, I would love to do that on my own - but it would be so great if you could give me a hint where I might get...
Thanks for your great video. Im curious to read more about the whole issue of predicting actual prices versus only the direction. Do you have a good source on this? I can see why the latter is more robust, but once you start accounting for transaction costs, the magnitude of the direction is also important. curious to get your thought on this too.
No one does what he did because it’s stupid. It’s been common practice for over 40 years to calculate the logged odds of the derivative of the price (logged odds of the returns).
Machine learning is artificial learning from a geat many individual experiences. And like the wize man said "Experience is a lantern that we carry on our back and which only ever illuminates the path traveled". The experience of the Stock Market was done with a dominant economic model. Who knows what other models creative minds will come up in the future.
You would only care about directionality if brokers fees where not a thing. As soon as you are trading via a broker knowing the the price will go up is great but if it goes up by 10 cents and you would stand to gain $10 in absolute value is great but if the brokers fee is $5.99 you are not winning at all as you will still need to offload the stock thus you are loosing money and only your broker is winning here. This is why both price and direction are relevant you need to not only know you will see a price increase but also know that it will be enough to actually make sense to buy. Sure you could bet on the gains on average outpacing the fee's thus ending up net positive but if you are into betting like that leveraged options trading might be a better fit than stock trading.
The last row regenerated by the backtest is 2022-05-17. If the latest existing close is 2022-05-18 (assuming we’re in the morning of 2022-05-19), how is it we can predict the close of 2022-05-19? I suppose this has something to do with dropping rows with NaN…
Hi! Maybe you can compare your algorithm with the real optimal decision in every season, so you could "asign points" to this algorithm and compare with others!
Hello there. thank you for this excellent education. just I have one problem. my yfinance does not work and gives some errors. please introduce another library that I can catch the data
Do you think it's a good time to consider selling some stocks, or is it better to hold onto them for the long term? I’m considering rebalancing my $2M portfolios, So I'm curious about the best strategies for potential market downturns
I guess it's important to reassess your investment strategies based on current market conditions. You should also consider a market expert to guide you.
You're right mate! I’ve been using a fin-market expert for two years now and I own a 7figure diversified portfolio from investing in stocks. Currently, my portfolio is worth over $900k.
How can I participate in this? I sincerely aspire to establish a secure financial future and am eager to participate. Who is the driving force behind your success?.
Nicole Desiree Simon is the licensed fiduciary I use. Just research the name. You’d find necessary details to work with a correspondence to set up an appointment.
Ok, you have a model that when it says the market is likely to go up the market does go up 57% of the time. That's great. Now today is May 10, 2023. How do I know what the model thinks is going to happen tomorrow?
So a question, it’s currently 04-11 and I’m only getting the predictions for 04-10. As in I’m not getting the predictor for 04-12, it is also currently past 4pm so I’m assuming it is because the tomorrow price is = shift(-1). How is this a predictor then if it only gives me the prediction for day of and not day after?
I request you to create a video considering Fundamental Analysis news integration prediction model as its happening behind the scenes to change the values. Its just a request if possible.
Hi, great lesson, I have a question. I'm still new to data science. But why didn't you use the data as a predictor? Im asking because say we want to predict what happens in the next day. How do i pass it to the model when i didn't train with it
Amazing work! Although I have a few doubts. I selected 18 features - from global stock indices, currency, and commodity - to predict daily directional changes in Nifty 50. 1. I'm not using the closing price for input variables rather I'm using the difference in previous close and current close. Is this a correct approach. 2. Also, can I split the target variable into 5 category (Up, Down, Neutral, Extended Up, Extended Down).
1) wouldn’t that be the same as using closing values? 2) interesting idea but it will probably reduce the over all effectiveness of the model because it reduces the amount of training data that meets the 5 categories vs 2. I don’t know about India exchanges, but in the US, for example, Fidelity charges $0 trade fee and keeps $0 from market makers for order flow. It all goes to the customer as price improvement. This is an extreme case, but my point is that I’m 2023, there should be markets you can trade for little to no cost. The brokers want your limit orders because it provides their other customers more liquidity without having to execute through a market maker. Also, they sell the limit order data to hedge funds that use that extra level of info to have an edge on the markets.
Excellen video. I think you have a great teaching ability. I'm surprised you did not start with the usual "THIS IS NOT FINANCIAL ADVICE..." disclaimer 😇
WOW, That was GREAT! Thanks man. It was perfect and very professional in every aspect. But I think you missed a little thing: (maybe I'm wrong ...) model.predict_proba has 2 classes, so to say: model.predict_proba(test[predictors])[:,1] for UpDays and: model.predict_proba(test[predictors])[:,0] for DownDays. This way, I think there is an issue with these lines … preds=model.predict_proba(test[predictors])[:,1] preds[preds >= .6]=1 preds[preds < .6]=0 # !? I'm new to machine learning and struggling with my model for now 6 months I've learned so many things from you 😊👍👍 Viele Grüße Kourosh
I get the Error "index 1 is out of bounds for axis 1 with size 1" on the first line here. with my "preds" being a list (so one dimensional). Do you have any clue what I missed?
My spouse and I are adding a variety of stocks/ETF to my present holdings for the long term, We've set aside $250k to start following inflation-indexed bonds and stocks of companies with solid cash flows, I believe it is a good time to capitalize on the market for long-term gains, but it wouldn't hurt to know means of actualizing short term profit.
The current market might give opportunities to maximize profit within a short term, but in order to execute such strategy , you must be a skilled practitioner.
Having an lnvestment advser is the best way to go about the market right now, especially for near-retirees, I've been in touch with a coach for a year now mostly because I lack the depth knowledge and mental fortitude to deal with these recurring market conditions, I nettd over $320K in profits so far, Its clear there's more to the market that we avg joes don't know that Investment advisors know.
Thank you for this tip. it was easy to find your coach. Did my due diligence on her before scheduling a phone call with her. She seems proficient considering her résumé.
16:35 A rough estimate of this model's real world accuracy is 10 - 20%. Fascinatingly, we get the same accuracy when doing Vedic astrology + ML. Maybe this is the reason why no "super AI traders" or "super astrologers" have popped up in the last 5 years of this tech being around. ML is better for data points that is far too abstract for humans, like pixels in an image. Hence AI image recognition has amazed the world and not "AI trade prediction".
Vik thank you for this video! Greetings from Poland. Please explain to me how to connect the model so that operating on a virtual server bought and sold instruments? How do you combine it?
Thank you so much for the tutorial and for taking the time to explain each piece of code in such a clear manner. I have two quick questions: 1.) What is the purpose of the .csv file ? 2.) Broadly speaking, what would be the steps to using a different API? Thanks !!
@@adamfrench4587 You need to save the result of model.predict_proba to another variable. add probs = preds before changing "preds" with 0.6 condition. And then add "probs" to the array inside pd.concat.
On September 1st, 2022, I ran this exactly as it is shown in video. It pulls data till August 31st, 2022 which is what I expected. However, the tomorrow column, the predicted price for Sept 1st does not show. At the end of the script, it pulls data up to August 30th and predicts price for August 31st which is not correct because that is previous close. It should predict September 1st because the market is not closed yet. Something wrong somewhere and I am still learning this script.
It will remove some rows from training because we need to have an actual tomorrow price value to use the data for training purposes. You can feed future data into the predict methods the same way you feed in the test set. This will let you make future predictions.
@@Dataquestio Would be able to provide an example of code how you would feed future data into the predict method? I think that example would be great for people to understand the process
How would you use the volume column? Not sure how to use the volume, can we build some relative volume indicator? Can you give a hint, or maybe a link to a video, where you use volume somehow to improve your model? Volume should influence the model significantly.
To see similar results as the tutorial, you would want to add the following lines to limit the period max --> sp500 = sp500.history(period="max") end_date = "2022-05-19" sp500 = sp500[sp500.index
Hi Jeevan - the easiest way to do it is to scrape daily headlines from say the new york times, and create a "sentiment" model to indicate confidence in the market. The output of that model could then be a predictor column. Of course, you could get a lot more complicated than this :)
T Pred 2022-09-06 0.0 1.0 2022-09-07 1.0 1.0 2022-09-08 1.0 0.0 For example this case. The prediction of 2022-09-08 is 0, does it mean that the price will go down tomorrow 2022-09-09? Great work!! Thank you so much Vik!
I’m trying to figure out which kind of career gets you working on these kind of predictions on a daily basis? I’m on the fence whether to go the BBA route and major in finance or to major in computer science. I know I’ll need both, but I’m unsure which area is more important. Does a financial analyst do this primarily or is it a data scientists job?
Pure math. Eventually physics. Companies like people who know how to think and have good problem solving skills that could be applied to anything. Also in math uni you learn a lot of programming these days
Backtesting/optimizing on historical data is merely "curve fitting". I know from firsthand experience working with TradeStation, creating a model that blew away the market on 4 years of recent historical S&P Futures data, but then failing miserably going forward on live data.
Anytime you use historical data - - and you optimize your trading algo's on historical data - - you are merely curve fitting. I highly recommend paper trading your algo on current data for at least 3 to 6 months to see it's real world performance, the longer the better.
Ill take a notes: the model without hyperparameter tuning. if hyperparamter tuning is done, when backtesting we no longer need to look for the best parameters. In contrast to cross-validation which requires more tuning
Hello, thank you very much for the video, I am new to ML, I would like to know how to use the model? How do I see the prediction for the next day? thanks and greetings
Hi everyone! You can find the code for this tutorial here - github.com/dataquestio/project-walkthroughs/tree/master/sp_500 .
Thanks Vik!
Thanks Vic, However your F1 score is at 0.5. How does that factor in?
Thanks, but it's incomplete.
Hey Viki. You should have used the pd.dropna(inplace=True).
Great video. Will you or can you provide additional information on other useful classifiers and also how to merge other data sources like news and sentiment into this code?
First of all, this whole economic chaos was powered by optimism that the FED is done with hiking interest rates. Now that interest rate crash is the situation, where do we go from here? How would you advise I safely allocate $250k funds at this point?
The market is volatile at this time, But doesn't the Federal Reserve's monetary policy and low interest rates contribute to the current valuations? hence I will advice you get yourself a financial advisor that can provide you with entry and exit points on the share/ETF you focus on.
Agreed, my portfolio is well-matched for every market season yielding 85% from early last year to date. I and my CFP are working on a 7 figure ballpark goal, tho this could take another year. IMO, financial advisors are the most sought-after professionals after doctors.
This sounds interesting. My portfolio is in the red. Can you recommend your analyst, please?
Mine's Rebecca Noblett Roberts. She turned out to be better and smarter than all the advisors I ever worked with till date, I’ve never met anyone with as much conviction.
I ran an online search on her name and came across her websiite; pretty well educated. thank you for sharing.
Clear and to the point. I hate super long videos full of things that don't provide much value. This one was great. I like that he walked through general data science/machine learning steps. In particular the data cleansing which many skip over, but it is actually an important step. Also, a pet peeve of mine is audio quality. This video you can hear the presenter clearly and he doesn't sound like his is working from a tin can.
The features used for the random forest cannot be the high, close, low , open values directly without any transformation because what the model is essentially doing is creating a overfit of non linear decisions to certain prices ranges. It is basically memorizing that when the close was above X value and open below Y value predict 1 or 0. You need to normalize the predictors in some way so that the model can use them independently of how high the value the stock is and truly create generalizable rules. Ratios are good since they use percentage instead of using absolute values and allow the model to use information of multiple candles as well.
Quite important comment.
Excellent. This tutorial corrects an error that pretty much every other video from others that I have seen has made. Don't seek MSE precision in your target as your goal. That's not what practitioners are looking for. Do what this educator has done instead. This model gets it right as used in the real world. Solid base to work with. Well done!
No, this is not even close to how practitioners have approached the problem in the last 30 years…
Watched up to 2:26 and I already know this is going to be excellent.
Clear and concise explanation from the start and you know this is going to be more than your ordinary YT tutorial
It's not excellent, you can't beat the market as regular person. You basically compete with Harvard graduates with math, computer science, etc. Degrees. Again, one UA-cam video won't make you beat the market
@@alang.2054 someone had to break this kids dreams of being rich off a youtube vid
@alang.2054 Where'd you get that she said she would beat the market from her comment?
I read an observation just stating that, this video is higher quality than most YT videos that claim to teach you something specific yet just give you fluff..
I’m new to coding but have always been an avid market watcher and looking for opportunities. Best video I’ve seen since I started scouring the depths of UA-cam for this content last week. Thank you sir!
This is awesome, instead of showing what you need to learn or try it shows how to actually build a model. This is very usefull. Thank you!
Could we get a similar video bus featuring a deep learning model instead?
What are you talking about? Do you really think this guy would show you real ways to make money? On market you compete with professionals in multi billion hedge funds with degrees, you can't beat them with UA-cam video
Before you start investing, it's crucial to understand the basics of investing, different asset classes (stocks, bonds, real estate, etc.), and the associated risks. are you investing for retirement, buying a home, or building an emergency fund? Your goals will help shape your investment strategy.
Don't put all your eggs in one basket. Diversification means spreading your investments across different asset classes (stocks, bonds, real estate) and within those classes (different companies or industries). This helps reduce risk.
I Invest in low-cost index funds or ETFs that track broad market indices, such as the S&P 500. These funds offer diversification across a wide range of stocks and can be a more passive, low-maintenance investment option. They are suitable for investors who prefer a hands-off approach and want exposure to the overall market.
I try to consider a mix of bonds and fixed-income securities to provide stability to my portfolio but i need solid advise.
I'm very cautious about giving specific recommendations as everyone's situation varies. Consider independent financial advisors like "Vivian Jean Wilhelm" I've worked with her for some years and highly recommend her. Check if she meets your criteria.
Thanks a lot for this suggestion. I needed this myself, I looked her up, and I have sent her an email. I hope she gets back to me soon.
I've heard the stock market often does well during election years. I recently inherited a lump sum and want to invest it wisely to take advantage of this potential upswing. Do you have any tips or strategies?
A good portfolio should have three basic things: ETFs for diversification, dividend stocks for cash flow, and leading tech stocks. When starting especially with a lump sum, it's a good idea to talk to a fiduciary advisor for expert advice.
I believe every Investor should start with ETFs for a solid foundation, then diversify across asset classes and maintain disciplined, regular investing to minimize risks and maximize growth.
You don't need to find the next NVDA to succeed in investing. Just choose top-notch ETFs and partner with a financial advisor like I did. I turned $100k into $40,000 in annual dividends-a significant milestone for me today.
I've been considering getting one, but haven't been proactive about it. Can you recommend your advisor? I could really use some assistance.
I'm cautious about giving specific recommendations as everyone's situation varies. Consider independent financial advisors like "Melissa Elise Robinson" I've worked with her for 6 years and highly recommend her. Check if she meets your criteria.
Very nice video and a great explanation . You didn’t mention finally how to get stock price predictions for tomorrow
Impressive video, and very well narrated !!!
I have certainly learnt more about PY and ML...
I am curious as to taking the model and learning, knowing how to tweak it to improve its successful trade ratio (with the conditioned factors already in place)
Thank you for taking the time to do this video...
I've always been interested in binary options, but I never knew where to start. Thanks to you and this video, I finally felt confident enough to give it a try.
Searched & watched a LOT of videos. This is the best. Well done man.
have you tried them? do they work on real data?
Great video, would you consider doing a follow up on some of the stuff you mentioned that would further enhance it?
thank yiu so much fir the video. I have taken varius courses in different places, and your video and teaching style are certainly the best !
Hint: on a recent macbook you can use all its cores by:
import joblib
N_CORES = joblib.cpu_count(only_physical_cores=True)
...
model = RandomForestClassifier(n_estimators='your value', min_samples_split='your other value', random_state=1, n_jobs=N_CORES)
The speedup is amazing
you don't need any information about the system to do this, n_jobs = -1 will use all the available cores with no imports or extra lines :)
Vik, I echo the compliments on the excellent video. I was able to use my own bespoke weekly market timing signals aligned with weekly S&P closes to finally get a grounded statistical "opinion" on the predictability of forward returns - as only my second Python exercise! Thanks!
I'm hoping you can do a follow up video to this. Would be great to see how you would incorporate macro data into your model, such as news or interest rates.
Sir your explaining skills are top notch
your ability to hide though is not....
when you split the data into the training and testing dataset, you are actually performing what is called Simple Random Sampling, this will cause the training data to have the same elements/characteristics of the testing dataset. If you were to calculate the means of each predictor variable in the testing and training dataset it will roughly be the same due to random sampling. The point I am trying to make is that you cannot claim the model has not "seen" the testing data, yet it managed to capture the majority of its properties due to simple random sampling, how about you train the model using the first 70% rows then leave the remaining 30% at the bottom for predictions? In that way the model does not have any idea what's happening with the remaining 30% (though there is an argument one can put forward about this), I think that approach would be the most realistic. I have used the simple random sampling before and I have gotten results which seemed to be accurate, it was not until I used this method I am suggesting to you that I obtained a little bit higher errors.
Wow, the concept of predicting the stock market using machine learning and Python is such a fascinating topic! The blend of finance and technology is always an area ripe for innovative approaches. It's impressive how machine learning can analyze vast amounts of data to find patterns that might not be obvious at first glance. Python, with its extensive libraries and community support, is an excellent choice for such complex computations. It's exciting to think about how these tools can provide insights into market trends and possibly even predict future movements. The intersection of machine learning and finance is definitely a space to watch! 📈💡🤖
Excellent video and you are above average by all means. You made things easier for me who is new to Python. At 65 yrs old I tried to work your script and it worked beautifully. So, I tried with TSLA ticker and it gave me no obj to concatenate error and I have no idea how to fix that error.
Hey
I'm facing the similar issue.
You got any solutions?
@@bvspa Did not work for me yet
@@SuperVIN786 you have to alter the start an step count as per the dataset
@@bvspa Thanks I will try that
@@bvspa Finally ran it after I made start and step number change, watched the video again which helped. Thanks
I cannot thank you enough! It's very straight to point and I've learned more in this video than in n online courses and articles.
This is very nice way to get started using data science with the markets. This gives a nice framework to get started. And attempt to expand the predictors (on RSI based or Change in Open Interest , some correlation with the major stocks composing that index) . Thank you for sharing.
Congratulations for your explanation and it was very clear. I would like to suggest you to prepare a vide including news about the stock into this model. Thanks
Buying a stock is easy, but buying the right stock without a time-tested strategy is incredibly hard. I’ve been trying to grow my portfolio of $160K for sometime now, my major challenge is not knowing the best entry and exit strategies
Investors should be cautious about their exposure and be wary of new buys, especially during inflation. Such high yields in this recession is only possible under the supervision of a professional or trusted advisor.
I have been speaking with a coach for a long time now mostly because I lack the background knowledge and mental toughness to handle these reoccurring market conditions. I made over $220K during this drop, which proved that there is more to the market than the average person is aware of.
I just started a few months back, I'm going for long term, I'm still trying to wrap my head around it, who’s this advisor you work with?
Credits to *Sharon Louise Count* one of the best portfolio manager;s out there. she;s well known, you should look her up
I Found her online page by searching her full name, I wrote her an email and scheduled a call, hopefully she responds soon. Thanks
My man is doing noble work. Kudos!
Actually you forgot to measure the expectancy of a trade in the case it has a precision of 42%. Because what makes a strategy profitable is bit the win rate but rather the expectancy of the trades. Although it is a great video and a good tutorial about programming. Thanks and keep up the good work.
Great video. Thank you for the insights. Going to be tuning into more of your work.
I suggest you google the semi strong efficient market hypothesis. Would save a lot of time.
Is he doing classification? (I wonder because most people do Regression) Thank you for your reply.
Hi Vik, I love the way you work and describe at lightning speed so effortlessly.
As you said, you have worked for a long time on this and I am quite sure you wouldn't give away your personal final model. But since you probably already included other stock markets and in particular news feeds, could you give me a hint to which kind of accuracy you got up to?
I am not asking for any kind of code, I would love to do that on my own - but it would be so great if you could give me a hint where I might get...
Thanks for your great video. Im curious to read more about the whole issue of predicting actual prices versus only the direction. Do you have a good source on this? I can see why the latter is more robust, but once you start accounting for transaction costs, the magnitude of the direction is also important. curious to get your thought on this too.
No one does what he did because it’s stupid. It’s been common practice for over 40 years to calculate the logged odds of the derivative of the price (logged odds of the returns).
Great tutorial dude!
I love Dataquest learning method. Thanks for the video
Machine learning is artificial learning from a geat many individual experiences. And like the wize man said "Experience is a lantern that we carry on our back and which only ever illuminates the path traveled". The experience of the Stock Market was done with a dominant economic model. Who knows what other models creative minds will come up in the future.
The stock market always existed.
Very thorough and loved it sir. Thanks for the video lesson.
You would only care about directionality if brokers fees where not a thing. As soon as you are trading via a broker knowing the the price will go up is great but if it goes up by 10 cents and you would stand to gain $10 in absolute value is great but if the brokers fee is $5.99 you are not winning at all as you will still need to offload the stock thus you are loosing money and only your broker is winning here.
This is why both price and direction are relevant you need to not only know you will see a price increase but also know that it will be enough to actually make sense to buy. Sure you could bet on the gains on average outpacing the fee's thus ending up net positive but if you are into betting like that leveraged options trading might be a better fit than stock trading.
Which SOFTWEAR used for run this code ?
The last row regenerated by the backtest is 2022-05-17. If the latest existing close is 2022-05-18 (assuming we’re in the morning of 2022-05-19), how is it we can predict the close of 2022-05-19?
I suppose this has something to do with dropping rows with NaN…
Hi! Maybe you can compare your algorithm with the real optimal decision in every season, so you could "asign points" to this algorithm and compare with others!
Hello there. thank you for this excellent education. just I have one problem. my yfinance does not work and gives some errors. please introduce another library that I can catch the data
Hi, how do I predict the next , for instance in a new data.
Do you think it's a good time to consider selling some stocks, or is it better to hold onto them for the long term? I’m considering rebalancing my $2M portfolios, So I'm curious about the best strategies for potential market downturns
I guess it's important to reassess your investment strategies based on current market conditions. You should also consider a market expert to guide you.
You're right mate! I’ve been using a fin-market expert for two years now and I own a 7figure diversified portfolio from investing in stocks. Currently, my portfolio is worth over $900k.
How can I participate in this? I sincerely aspire to establish a secure financial future and am eager to participate. Who is the driving force behind your success?.
Nicole Desiree Simon is the licensed fiduciary I use. Just research the name. You’d find necessary details to work with a correspondence to set up an appointment.
She appears to be well-educated and well-read. I ran an online search on her name and came across her website; thank you for sharing.
Ok, you have a model that when it says the market is likely to go up the market does go up 57% of the time. That's great.
Now today is May 10, 2023. How do I know what the model thinks is going to happen tomorrow?
Never mind, I see it is explained in some of the other comments. I need to look at the prediction column in the last row of the up-to-date data set.
Excellent video, thank you for sharing this. Hopefully I can see more ML related videos going forward.
So a question, it’s currently 04-11 and I’m only getting the predictions for 04-10. As in I’m not getting the predictor for 04-12, it is also currently past 4pm so I’m assuming it is because the tomorrow price is = shift(-1). How is this a predictor then if it only gives me the prediction for day of and not day after?
I request you to create a video considering Fundamental Analysis news integration prediction model as its happening behind the scenes to change the values. Its just a request if possible.
Would be great to see an updated or enhanced version that incorporated a LLM to show how easy data manipulation can be…
Hi, great lesson,
I have a question.
I'm still new to data science.
But why didn't you use the data as a predictor?
Im asking because say we want to predict what happens in the next day.
How do i pass it to the model when i didn't train with it
Amazing work! Although I have a few doubts. I selected 18 features - from global stock indices, currency, and commodity - to predict daily directional changes in Nifty 50.
1. I'm not using the closing price for input variables rather I'm using the difference in previous close and current close. Is this a correct approach.
2. Also, can I split the target variable into 5 category (Up, Down, Neutral, Extended Up, Extended Down).
1) wouldn’t that be the same as using closing values?
2) interesting idea but it will probably reduce the over all effectiveness of the model because it reduces the amount of training data that meets the 5 categories vs 2. I don’t know about India exchanges, but in the US, for example, Fidelity charges $0 trade fee and keeps $0 from market makers for order flow. It all goes to the customer as price improvement. This is an extreme case, but my point is that I’m 2023, there should be markets you can trade for little to no cost. The brokers want your limit orders because it provides their other customers more liquidity without having to execute through a market maker. Also, they sell the limit order data to hedge funds that use that extra level of info to have an edge on the markets.
Excellen video. I think you have a great teaching ability. I'm surprised you did not start with the usual "THIS IS NOT FINANCIAL ADVICE..." disclaimer 😇
This was an amazing walkthrough. I have learned so much!
Great, have you tried to improve the model ?
Hi can we use this for Indian stock markets?❤
Explaining is on top. Thank you!
I am getting a zero division error while calculating precision score so please help
WOW, That was GREAT! Thanks man. It was perfect and very professional in every aspect.
But I think you missed a little thing: (maybe I'm wrong ...)
model.predict_proba has 2 classes, so to say:
model.predict_proba(test[predictors])[:,1] for UpDays and:
model.predict_proba(test[predictors])[:,0] for DownDays.
This way, I think there is an issue with these lines …
preds=model.predict_proba(test[predictors])[:,1]
preds[preds >= .6]=1
preds[preds < .6]=0 # !?
I'm new to machine learning and struggling with my model for now 6 months
I've learned so many things from you 😊👍👍
Viele Grüße
Kourosh
I get the Error "index 1 is out of bounds for axis 1 with size 1" on the first line here. with my "preds" being a list (so one dimensional). Do you have any clue what I missed?
Could you make another video like this but with Forex pairs like EURUSD, GBPUSD & AUDUSD please? 👍
My spouse and I are adding a variety of stocks/ETF to my present holdings for the long term, We've set aside $250k to start following inflation-indexed bonds and stocks of companies with solid cash flows, I believe it is a good time to capitalize on the market for long-term gains, but it wouldn't hurt to know means of actualizing short term profit.
The current market might give opportunities to maximize profit within a short term, but in order to execute such strategy , you must be a skilled practitioner.
Having an lnvestment advser is the best way to go about the market right now, especially for near-retirees, I've been in touch with a coach for a year now mostly because I lack the depth knowledge and mental fortitude to deal with these recurring market conditions, I nettd over $320K in profits so far, Its clear there's more to the market that we avg joes don't know that Investment advisors know.
Salvatore Fortunato Sofia. You can easily look her up, she has years of financiaI market experience.
Thank you for this tip. it was easy to find your coach. Did my due diligence on her before scheduling a phone call with her. She seems proficient considering her résumé.
This was an excellent presentation.
16:35 A rough estimate of this model's real world accuracy is 10 - 20%. Fascinatingly, we get the same accuracy when doing Vedic astrology + ML. Maybe this is the reason why no "super AI traders" or "super astrologers" have popped up in the last 5 years of this tech being around. ML is better for data points that is far too abstract for humans, like pixels in an image. Hence AI image recognition has amazed the world and not "AI trade prediction".
Do we have any latest updates to this model? Adding extended logic for improvements?
Nothing beats decades of trading experience.
centuries of trading experience
@@noahschuler6388
Strictly speaking, millennia is ideal.
Vik thank you for this video! Greetings from Poland. Please explain to me how to connect the model so that operating on a virtual server bought and sold instruments? How do you combine it?
Thank you so much for the tutorial and for taking the time to explain each piece of code in such a clear manner. I have two quick questions: 1.) What is the purpose of the .csv file ? 2.) Broadly speaking, what would be the steps to using a different API? Thanks !!
If you can fit the data from the API into a data-frame it would be very easy.
@@FlisB thanks for replying. Would you by any chance know how get (in addition to 1 or 0 when proba >.6) a column with the actual probability?
@@adamfrench4587 You need to save the result of model.predict_proba to another variable. add probs = preds before changing "preds" with 0.6 condition. And then add "probs" to the array inside pd.concat.
Legend, thank you so much!
I get how we can predict for one day, but can we predict with this model for several days, or what the trend will be for the next week?
DUDE THIS IS SO HELPFUL
Shouldn't you normalize the data first? It's an excellent tutorial btw.
On September 1st, 2022, I ran this exactly as it is shown in video. It pulls data till August 31st, 2022 which is what I expected. However, the tomorrow column, the predicted price for Sept 1st does not show. At the end of the script, it pulls data up to August 30th and predicts price for August 31st which is not correct because that is previous close. It should predict September 1st because the market is not closed yet. Something wrong somewhere and I am still learning this script.
It will remove some rows from training because we need to have an actual tomorrow price value to use the data for training purposes. You can feed future data into the predict methods the same way you feed in the test set. This will let you make future predictions.
@@Dataquestio Would be able to provide an example of code how you would feed future data into the predict method? I think that example would be great for people to understand the process
@@cdvllcI agree with u, I also stuck in the code to feed future value in prediction phase
What a deep voice
How would you use the volume column?
Not sure how to use the volume, can we build some relative volume indicator? Can you give a hint, or maybe a link to a video, where you use volume somehow to improve your model?
Volume should influence the model significantly.
To see similar results as the tutorial, you would want to add the following lines to limit the period max
-->
sp500 = sp500.history(period="max")
end_date = "2022-05-19"
sp500 = sp500[sp500.index
Is there next project where you improved the accuracy of the model to a higher percentage
Brilliant video Vik! Towards the end, you mentioned adding news to the model. Could you share how one could integrate that?
Thanks!
Hi Jeevan - the easiest way to do it is to scrape daily headlines from say the new york times, and create a "sentiment" model to indicate confidence in the market. The output of that model could then be a predictor column. Of course, you could get a lot more complicated than this :)
Can you fit this model for all stocks or just this one?
T Pred
2022-09-06 0.0 1.0
2022-09-07 1.0 1.0
2022-09-08 1.0 0.0
For example this case.
The prediction of 2022-09-08 is 0, does it mean that the price will go down tomorrow 2022-09-09?
Great work!!
Thank you so much Vik!
Hi Leo - the prediction is for the next trading day, yes. So the row labeled 9-08 is the prediction for 9-09.
@@vikasparuchuri Thank you so much!
Very good explanation, thanks.
Great tutorial!
What did you use for the risk rate as there is no such thing that exists in finance
I’m trying to figure out which kind of career gets you working on these kind of predictions on a daily basis? I’m on the fence whether to go the BBA route and major in finance or to major in computer science. I know I’ll need both, but I’m unsure which area is more important. Does a financial analyst do this primarily or is it a data scientists job?
Pure math. Eventually physics. Companies like people who know how to think and have good problem solving skills that could be applied to anything. Also in math uni you learn a lot of programming these days
A BS in Data Science, not CS or SE
What is missing is comparing it to another strategy. What about Buy & Hold? What about "if the price increases yesterday, it will increase today"?.
What are the Profits after trading every day compared to SPY performance as the benchmark.
Thats a really good video and it seems you really know what you are talking about. Thanks!
cool went threw the whole process on mini conda.
Backtesting/optimizing on historical data is merely "curve fitting". I know from firsthand experience working with TradeStation, creating a model that blew away the market on 4 years of recent historical S&P Futures data, but then failing miserably going forward on live data.
Known as the overfitting problem.
How can we measure whether this model is overfitted or not ?
Anytime you use historical data - - and you optimize your trading algo's on historical data - - you are merely curve fitting. I highly recommend paper trading your algo on current data for at least 3 to 6 months to see it's real world performance, the longer the better.
Amazing video!! Have yiou looked at the performances of other ML techniques, e.g, MLPregressor?
Bro your accent is so good, yet better than Americans
Ill take a notes: the model without hyperparameter tuning. if hyperparamter tuning is done, when backtesting we no longer need to look for the best parameters. In contrast to cross-validation which requires more tuning
This was very well delivered. Thank yo sharing.
I will consider the suggestions you made and see how this works.
Very exciting with a bit of 😅.....
Very useful man, thanks for show us the way!
dude your look in the thumbnail is straight Tim and Eric
It's free real estate
so how many people have been able to predict using this technique and what are the success and fail use cases ?
How has the model done this year? Does it show a topping formation?
What a great framework to ML time-series data for prediction. Thanks for sharing!
Great video but where is the clarification that it will go up or down tomorrow?
Hello, thank you very much for the video, I am new to ML, I would like to know how to use the model? How do I see the prediction for the next day? thanks and greetings
Excellent video!
What is the definition of you can get pretty far? 51, 60, 70, 80, 90 % accuracy?