We’ve run hundreds of A/B tests in the past 3 years of being in business. These were some of the most impactful ones. I hope you got value from my sharing these results. Would you like a video where I cover how to set up A/B testing using No Code? Next up ➝ Learn how to validate your SaaS idea Watch it here: ua-cam.com/video/aSMxdBVTcLg/v-deo.html Be sure to hit the like button on the video if you got value from it! Talk to you in the comments 👇
Simon this video was absolute our gold. Thea insights are huge and it has definitely got me motivated to run my own AB tests. We are currently stuck around $11k MRR and I think this is what we are missing. Thanks 👊🏻
I'm pumped to hear that 🙌 Congratulations on $11K MRR. That's a huge win! I think you'll gain a ton of new insights if you start running more experiments and A/B testing in your product.
This is MASSIVE! Crazy to see these results. The onboarding ARPU was interesting! Data doesn't care about your feelings... 😂 and great work on the editing in this video 👏
Can you do a video on the failed tests that didnt bring any meaningful results ? Otherwise many entrepreneurs might get survivor-bias and think that A/B is a perfect tool that should always always be used. I've had to explain in the past that A/B is really good once you get a certain scale, but if you haven't even had some product market fit it might actually hurt your team speed to iterate and add value
It's a super good point 👌 You do need at certain volume in order for A/B tests to become statistically significant. Although, I still think teams should practice A/B testing from an early stage, but the results should definitely be taken lighter in the beginning.
This is great Simon!! I built a script where I’m able to choose how much traffic is routed to each feature flag of my application. I think I’m going to use it to run some a/b tests. Thank you very much for sharing this! 😊
The 8:40 experiment seemed obvious and was quite obvious. It’s just the endowment effect, where option b it seems like you’re losing something. And in option a it feels like your getting a discount. Because of the anchors.
I think its a simple situation. People who don't understand your product fully (the ones left to their own devices) are not going to appreciate what it is that makes your product critical for their use case. When you onboard someone, you expose them to many possibilities and features of your product that may or may not be assumed to exist before the user signed up. Having said this, onboarding is usually done in a very hamfisted way by most companies. Its very important you dont piss off your users while they are forming an opinion.
I check the Feedhive page every couple of weeks and look what has changed, what works and where I can find inspiration. But I mean, for newcomers definitely a great inside. Between: 1.) It would be cool if you could share your UA-cam setup / I know vou want to stick with vour topic brother, but maybe like „my setup for a personal brand in the SaaS space" e and how you do the effects. 2.) where do you learn all that stuff, which Podcasts, communities or topics are you looking at a weekly basis.
Haha sneaky 😉 And yes, I know. You're definitely not the first to ask! But as you say, it's a bit off topic for this channel. I'll see if I can get these topics incorporated into something that works soon 😊 Thanks a ton for the feedback.
For Test #1: Could it be that scenario B did better because some of those users forgot to un-sub from the plan that they were forced to enter credit card info for? If so, then does scenario B have a high number users churning than scenario A on the 2nd month?
It's a good observation, but no, I don't consider that to be the case. It happens occasionally (though not very often) that some users forget to opt out during the trial, but they quickly reach out to support and we obviously refund the ones who didn't intent to continue on a paid plan. So no, overall the churn isn't different in the B-Group.
Hi Simon, thank you for sharing this. There are great insights here. One clarification: In test #4, is the scroll stopping measurement needed? Could it be said that, since the comparison is with lack of content, any content could have caused the scroll stop and not necessarily the G2 badges? CTR measurement alone would be fine to come to the same conclusion, right? (say, CTR of 3% of total visitors etc.)
@simonhoiberg why you don't make a course on how to make a saas from start to finish and take a real case study like replicating some of the most known saas or websites for exemple : starting with something simple ass like Socialblade and newsletters to something complex
Amazing video Simon I didn't fully understand the ARPU difference, was that because less people churn / will continue after trial who did the onboarding flow? Thanks!
What is the retention i.e. when people fill in the credit card they might forget they did so? Have you considered an option (C) and just doing better onboarding for A and not forcing the credit card part?
Retention is roughly the same in both groups. It does happen occasionally that people forgot about the trial, but we almost always refund them. Still, it happens rarely enough that it doesn't affect the numbers. No noticeable difference in retention between the groups.
Hey Simon, Have been following your videos since the last 1 year. Could you make an updated video on SAAS tech stack. I'm building an Applicant Tracking System SAAS and my tech stack is highly inspired by your serverless stack. I'm curious to learn more specially about DynamoDB since I'm using it looking at its advantages but considering the relational nature of my product, I'm quite nervous as well.
Hi Simon, Love your videos! 2 more A/B experiment requests! Can you: - Simply add the "add your CC to start your free trial" step to the first version? (Without the extra onboarding questions?) - If you believe the conversion is higher due to an increase of perceived value due to increased friction and thereby creating a Loss Aversion effect, can you add the onboarding steps to the *No CC* user journey? Very curious :) Also, question: the ARPU per MONTH was higher, but what about the Average Cutstomer Lifetime Value (ACLV)? Hypothesis: - More people may upgrade when CC is forced - However, on average, people stay a customer *longer* if they upgraded from intrinsic motivation, after genuinly experiencing the product's value and make a conscious decision that it's worth paying for. If true, at first glance it may seem like you increase your revenue with 52% in the short term, while it may well be that you're actually decreasing revenue with 52% in the long term. Lastly, to me it just seems like *usage and success-based pricing* will always have the least friction to make a sale. Therefore they will always convert the highest. If not, whatever the CX is in between the Awareness and Revenue step in the customer journey is needs to be improved. AKA: how well are you guiding their experience and decrease Time to Value? What story do people build up in their mind as they use your product?
Take this with a grain of salt. Industry SaaS metrics on Product Led Growth are the opposite of what he says. Onboarding should be frictionless and easy. Show free value fast to get the user “hooked”, then upsells follow.
Absolutely! I also make that very clear in the end of the video. These are _our_ results. You may very well get different results, so you are highly encouraged to do your own A/B tests 👌
I'd love to up the pace! Working on getting some operational parts of the video creation flow nailed, but hopefully I can push more videos in the upcoming year 🙌
We’ve run hundreds of A/B tests in the past 3 years of being in business.
These were some of the most impactful ones.
I hope you got value from my sharing these results.
Would you like a video where I cover how to set up A/B testing using No Code?
Next up ➝ Learn how to validate your SaaS idea
Watch it here: ua-cam.com/video/aSMxdBVTcLg/v-deo.html
Be sure to hit the like button on the video if you got value from it!
Talk to you in the comments 👇
Yes please
I was just about to ask for That since I'm not really technical.
Thank you
I would definitely appreciate such a video Simon ! Thank you
Simon this video was absolute our gold. Thea insights are huge and it has definitely got me motivated to run my own AB tests. We are currently stuck around $11k MRR and I think this is what we are missing. Thanks 👊🏻
I'm pumped to hear that 🙌
Congratulations on $11K MRR. That's a huge win!
I think you'll gain a ton of new insights if you start running more experiments and A/B testing in your product.
Your video editor needs a raise! It has to be the most amazing video on AB testing. 🔥
Haha... That would be me 😅
Thanks man! Glad you liked it.
This is outstanding!! I'm a full time writer with a history of failed SAAS, this is exactly why I'm here learning, I love this!
Well done man!
I'm glad you're getting value from this 🙌
As always you make an astonishing content Simon 🔥🔥🔥 🚀
This is MASSIVE! Crazy to see these results. The onboarding ARPU was interesting! Data doesn't care about your feelings... 😂 and great work on the editing in this video 👏
Thanks a lot, man!
Yes, indeed. Hard to argue with data 🤷
I watch every video you upload and this one was on another level. Well done Simon!
Wow Simon, what a gem !! Thank you for this video.
Another golded video from Simon!! 👏
Glad you liked it, buddy 🙌
Can you do a video on the failed tests that didnt bring any meaningful results ?
Otherwise many entrepreneurs might get survivor-bias and think that A/B is a perfect tool that should always always be used.
I've had to explain in the past that A/B is really good once you get a certain scale, but if you haven't even had some product market fit it might actually hurt your team speed to iterate and add value
It's a super good point 👌
You do need at certain volume in order for A/B tests to become statistically significant.
Although, I still think teams should practice A/B testing from an early stage, but the results should definitely be taken lighter in the beginning.
Super useful
Glad you liked it, Ali!
This is great Simon!! I built a script where I’m able to choose how much traffic is routed to each feature flag of my application. I think I’m going to use it to run some a/b tests.
Thank you very much for sharing this! 😊
Another awesome video! Thank you @simon
The 8:40 experiment seemed obvious and was quite obvious. It’s just the endowment effect, where option b it seems like you’re losing something. And in option a it feels like your getting a discount. Because of the anchors.
Awesome video Simon!
Glad you enjoyed it 😄
This is so much fun, thank you!
The pleasure is all mine 😁
I think its a simple situation. People who don't understand your product fully (the ones left to their own devices) are not going to appreciate what it is that makes your product critical for their use case. When you onboard someone, you expose them to many possibilities and features of your product that may or may not be assumed to exist before the user signed up. Having said this, onboarding is usually done in a very hamfisted way by most companies. Its very important you dont piss off your users while they are forming an opinion.
Super valild points! 🔥
@@SimonHoiberg i do this for a living :). I quite enjoyed that video
Great great video thank you. New subscriber 😊
This is crazy!! You saved our tons of money with these AB tests
I check the Feedhive page every couple of weeks and look what has changed, what works and where I can find inspiration. But I mean, for newcomers definitely a great inside.
Between: 1.) It would be cool if you could share your UA-cam setup / I know vou want to stick with vour topic brother, but maybe like „my setup for a personal brand in the SaaS space" e and how you do the effects. 2.) where do you learn all that stuff, which Podcasts, communities or topics are you looking at a weekly basis.
Haha sneaky 😉
And yes, I know. You're definitely not the first to ask!
But as you say, it's a bit off topic for this channel.
I'll see if I can get these topics incorporated into something that works soon 😊
Thanks a ton for the feedback.
Can you create a video on the tools you used for ab testings?
For Test #1: Could it be that scenario B did better because some of those users forgot to un-sub from the plan that they were forced to enter credit card info for? If so, then does scenario B have a high number users churning than scenario A on the 2nd month?
It's a good observation, but no, I don't consider that to be the case.
It happens occasionally (though not very often) that some users forget to opt out during the trial, but they quickly reach out to support and we obviously refund the ones who didn't intent to continue on a paid plan.
So no, overall the churn isn't different in the B-Group.
Btw @SimonHoiberg, any updates on the building in public video? Still in the making?
Hi Simon, thank you for sharing this. There are great insights here. One clarification: In test #4, is the scroll stopping measurement needed? Could it be said that, since the comparison is with lack of content, any content could have caused the scroll stop and not necessarily the G2 badges? CTR measurement alone would be fine to come to the same conclusion, right? (say, CTR of 3% of total visitors etc.)
Simon what tool do you recommend for the actual A/B logic and testing?
@simonhoiberg why you don't make a course on how to make a saas from start to finish and take a real case study like replicating some of the most known saas or websites for exemple : starting with something simple ass like Socialblade and newsletters to something complex
Amazing video Simon
I didn't fully understand the ARPU difference, was that because less people churn / will continue after trial who did the onboarding flow?
Thanks!
Yes. Higher trial-to-paid conversion, and higher paying users 😊
What is the retention i.e. when people fill in the credit card they might forget they did so? Have you considered an option (C) and just doing better onboarding for A and not forcing the credit card part?
Retention is roughly the same in both groups.
It does happen occasionally that people forgot about the trial, but we almost always refund them.
Still, it happens rarely enough that it doesn't affect the numbers. No noticeable difference in retention between the groups.
Really amazing video!!
what app do you use or software to know how many people clicked ?plz help
Hey Simon, Have been following your videos since the last 1 year.
Could you make an updated video on SAAS tech stack. I'm building an Applicant Tracking System SAAS and my tech stack is highly inspired by your serverless stack. I'm curious to learn more specially about DynamoDB since I'm using it looking at its advantages but considering the relational nature of my product, I'm quite nervous as well.
btw.. I'm using the single table architecture of dynamoDB as suggested by the AWS team in their articles.
Are you also doing this?
Great video!
Thank you 🙌
Hi Simon,
Love your videos! 2 more A/B experiment requests! Can you:
- Simply add the "add your CC to start your free trial" step to the first version? (Without the extra onboarding questions?)
- If you believe the conversion is higher due to an increase of perceived value due to increased friction and thereby creating a Loss Aversion effect, can you add the onboarding steps to the *No CC* user journey?
Very curious :)
Also, question: the ARPU per MONTH was higher, but what about the Average Cutstomer Lifetime Value (ACLV)?
Hypothesis:
- More people may upgrade when CC is forced
- However, on average, people stay a customer *longer* if they upgraded from intrinsic motivation, after genuinly experiencing the product's value and make a conscious decision that it's worth paying for.
If true, at first glance it may seem like you increase your revenue with 52% in the short term, while it may well be that you're actually decreasing revenue with 52% in the long term.
Lastly, to me it just seems like *usage and success-based pricing* will always have the least friction to make a sale. Therefore they will always convert the highest. If not, whatever the CX is in between the Awareness and Revenue step in the customer journey is needs to be improved.
AKA: how well are you guiding their experience and decrease Time to Value? What story do people build up in their mind as they use your product?
Simon I was wondering what kind of no code saas builder would you suggest when building ai saas?
Probably Bubble.
Wouldn't you be attached to bubble? Or is this not a big problem for starters? @SimonHoiberg
Jumped on the video faster than the light 😂
Haha, you're awesome Mehdi 💪
great video
You are a Boss....Pls help Young SaaS entrepreneurs like me, I have no laptop Brother 😢
Well B option was not my bet 😅 Interesting
Take this with a grain of salt. Industry SaaS metrics on Product Led Growth are the opposite of what he says. Onboarding should be frictionless and easy. Show free value fast to get the user “hooked”, then upsells follow.
Absolutely! I also make that very clear in the end of the video.
These are _our_ results. You may very well get different results, so you are highly encouraged to do your own A/B tests 👌
Thank you
You're most welcome! Pleasure is all mine 🙌
A/B Tests: Avoid Assumptions! Test!
UA-cam channel selling
Are you Austrian? Because sometimes you sound like Arnold 😅
@@kikky7 Danish 😂 But close.
Whenever you don't upload, as an Indian, am I allowed to say 'Simon come back'?
Haha, I guess so 😁
@@SimonHoiberg hehehe thanks..
Hahah, got u. 😂😂
Make more videos
I'd love to up the pace!
Working on getting some operational parts of the video creation flow nailed, but hopefully I can push more videos in the upcoming year 🙌
First
Priceless