Quick comments after releasing this video. 1. Browser session and auth data is highly time-dependent. A logged-in session might be invalid if the tests don't run for a while. Therefore, placing the generated storage state in `.gitignore` and regenerating it when the tests run is recommended. 2. You can and should only reuse session / login data when your tests aren't modifying the existing user. Otherwise, your tests will conflict with each other. In these scenarios, it's better to create a new user for each test.
Neat way to reuse logged-in state, when you don't create unique users for each test and can't (or don't want to) auth via API calls! Great content! Thanks 👍
Thanks for great video. I have already had API authentication in place that bypassed login of every single test. So I tried your storageState approach and did very basic measuring between those two approaches. I ran a single suite of 10 tests, with 3 workers on my workstation. I did 10 repeats and calculated average run time. API authentication in before each: Average run time: 14486.2 ms Setup with storageState: Average run time: 17121.5 ms So the storageState is on average slower by 2,635.3 ms. The setup itself seems to be on average ~ 1,700 ms This means that while each steps executes less amount of code, I guess the setting the of the storageState for each test takes a bit of time. And rough comparison would be they run by 0.1 s longer per test. I would judge that negligible. So to sum it up. It seems storageState does not provide any performance advantage over calling auth API and setting auth tokens before every test. But storageState definitely reduces the amount of code that is in your test file and improves readability. I would be interesting hearing your experience or anyone else on this topic. :)
@PakoVere This is great! 💙 Thank you so much for this investigation, this is super interesting. Let me summarize if I got this correctly. In your benchmark, you only apply the already created storage state (`setup` was already done?) and this approach is slower than making an api call and setting the value to your test runs? If so, this is interesting and I woudln't have expected this outcome. Could you share slimed down example code somewhere? I'd love to look into it! That said, I think it's a great investigation already, but in my experience sometimes logins need a browser and authenticating via API isn't always possible. But of course, if your API approach works and is faster - that's great stuff! 🎉
You could still have a single "setup" project in which you write and create different storage state files. You can do this similar to how it's shown in the video for a single user. Then you could define different projects ("User A project" and "User B project"), both relying on "setup" being a dependency but reading in the particular storage state file. "User A project" could rely on the storage state "user-a-state.json" or whatever. Then you could control what files will belong to which project (and as a result to which user) by specifying `testMatch` on the project level (playwright.dev/docs/api/class-testproject#test-project-test-match) Here's an untested example. ``` projects: [ { name: "setup", use: { ...devices["Desktop Chrome"] }, testMatch: /.*\.setup\.ts/, }, { name: user-a", use: { storageState: ".auth/user-a.json", }, testMatch: "*/user-a/*.spec.ts", dependencies: ["setup"], }, { name: user-b", use: { storageState: ".auth/user-b.json", }, testMatch: "*/user-b/*.spec.ts", dependencies: ["setup"], }, ], ``` Does this make sense?
@@ChecklyHQ Thank you so much. I will try this out. My set up is little more complex. I need to run all the tests in Desktop And in Mobile. If I follow the above suggestion, I will have to set up multiple user A desktop and user A mobile?
@ No, you should be able to have User A and User B but than would have 4 projects: - User A desktop - User A mobile - User B desktop - User B mobile But, of course, there are multiple ways to do things, but that's how I would start without seeing code. 😅
Great video, everything worked like a charm! Quick question, could you theoretically use similar logic to store login state when running the scraper in a production environment with Playwright Library?
I haven't tested it, but it should work the same when using the Playwright lib. :) The only thing to keep in mind then is that you'll have to run the setup script (it's done via projects in the video) somehow yourself. :)
@@ChecklyHQ I can confirm this works! I took the following steps: 1. Created an async function called login.js that returns the cookies, or sets and returns them if needs be - it's important to make sure this is just a normal async function, not a self invoking one 2. Created an async getCookies function to handle the cookie logic for cleanliness. It's just a try catch statement that either returns the cookies array or an empty array 3. Call the async login function in my index.js file on each run, so now it's able to handle the cookie logic depending on whether the cookies array is empty or has length Perhaps there's a dryer way to do this with browser contexts etc but I'm a little new to this. Everything seems to be working fine tho so I'll optimise at a later date! Thanks again for the tutorial-couldn't have reached here otherwise!
Hey Stefan, I really love your videos - they've helped me a lot! I’m currently facing an issue while testing a webpage (Web3) that interacts with the blockchain. I need to connect to a wallet before running my tests. I followed your approach of setting up the wallet connection through a setup file, as you explained in the video, but unfortunately, it doesn’t seem to be working. I think the connection isn't getting stored properly in the JSON file. Any advice on what might be going wrong? Thanks a lot!
Just a quick follow-up, Stefan - I was also wondering if you could consider making some videos about testing with extensions and Web3 in general? There's really not much material out there on this topic, and it would be super helpful to get your insights on it. Thanks again for all the great content!
Thanks man, cool tip! You said we can remove tick from setup, so auth step is not running before each test. But, what happens, when tests are running remotely in pipeline? How could I make setup test to run only once, and then take all info from storatestate? For example, I have 100 tests. I need setup step to run only before first step, not before all of them.
Thank you! 💙 Yes, what I meant in the video is that during development you can run the setup step once and than reuse the storageState for as long as it's valid. When running things in a CI pipeline Playwright should figure out everything for you automatically. When you run `npx playwright test` it'll figure out the project dependencies and run `setup` for each worker. As long as all your tests run in the same worker, `setup` should only run once. If you're parallizing things (which you most likely do), `setup` will run for each worker because workers can potentially run in different invironments. I hope this makes sense. :)
I have a question , for example if my web application has multiple user types to login (for example customer and back office user) every one of user has its own portal (with different url) , how can I use different logins (with different storageStates) within the same test ?
Playwright fixtures should be able to help out here. You could create different fixtures that rely on on different storage state. We have some fixture videos here on the channel: - ua-cam.com/video/2O7dyz6XO2s/v-deo.html - ua-cam.com/video/k488kAtT-Pw/v-deo.html The docs also include some help here: playwright.dev/docs/auth#testing-multiple-roles-together I'll put this case on the list of future videos. Thanks!
Hi, thanks for this very useful tip to increase the speed of running tests in Playwright. One issue though I encountered is that if we have several tests and after 20 minutes when running other tests, it failed because the token already expired. Do you have any best practice or recommendation in alleviating this issue?
Thanks for the question. Hmm... This is an interesting case. How many tests are we talking about here? I'd look into ways to speed up your test first, because 20min seems like a reasonable long time for an authentication token. Could you parallelize your tests more so that more things run to reuse the token and browser state?
Assuming that you're running the tests sequentially in every worker (i think this is recommended in the docs somewhere? 🤔) you could parallelize even more to decrease the overall time? If that doesn't work, unfortunately, I can't give any better advice because "refreshing storage state" seems wrong and is also missing the point of the whole concept. 😅
It works until my session is active i .e.if the maximum time for a session is 15 mins . If I try to access my stored. Json file after 15 mins it would show session time out . How to resolve it
I'm not sure if I understand your question correctly, but the session storage to be invalid at some point is expected. In CI/CD it would always run before your tests so it still speeds up your tests. In development you'd have to rerun `setup`.
I am working on an application where the login flow is not based on the user's input. Instead the user is redirected to an Okta auth page which internally uses kerberos and eventually redirects the user back to the application. The redirect logic and local storage update is handled by Okta auth js and okta react packages. In this scenario, how can I stub my auth state to a temporary json file? When I try to run my tests, the auth flow breaks as Okta server doesn't recognize the user when the application is running via playwright's browser.
Heyoooo. 👋 Unfortunately, I haven't tested this flow yet and neither do I have a quick test setup to play around with it. So, unfortunately, I can't give guidance here. :/
Great question! The `checkly.config` isn't a default replacement for your Playwright configuration because additionally it allows you to configure your Checkly monitoring infrastructure. If you want to reuse or define values in your `playwright.config` you can do that, too. :) www.checklyhq.com/docs/browser-checks/playwright-test/#global-configuration I hope this helps. 🦝
@@ChecklyHQ I have test cases need to be interacted between user and admin in one test. This is how I set it: const adminContext = await browser.newContext({ storageState: '.auth/admin.json' }) const adminPage = await adminContext.newPage()
const userContext = await browser.newContext({ storageState: '.auth/user.json' }) const userPage = await userContext.newPage() await adminPage.goto(commonData.stgUrl) await userPage.goto(commonData.stgUrl) But I need to paste it on each test, that is my issue. Hope that you give a suggestion
Unfortunately, we can't help with individual issues. But 5min init time indeed sounds very wrong. In these cases, trace files and UI-mode are a big help to indentify what's taking so long. Good luck!
Unfortunately neither have I done OKTA testing nor do I have an OKTA setup available. So generally, I can't give advice here right now. But if you make it to pass the OKTA 2fa flow storage state should still work. :)
7:46 this step is confusing because I’m not using the UI when configuring the CI tests right? Is this only required with the UI and otherwise tests respect the dependency array? Or would this also run setup every time before the each test?
UI mode is generally only for development. :) When you run the tests in CI or via the command line the dependency array will be respected either way and setup will run for each executed worker.
I can create the storage using setup, but it's not passing to next test. In the browser, it was showing" about:blank". But on your video pass to another test.
I'm sorry to hear that. Without additional info and example code, it's impossible to help, though. You can check the working example on GitHub: github.com/checkly/playwright-examples/
I have 7 tests to run, after using this, first, it's taking only 4 workers instead of 7 workers, and after that, the other 3 workers started. Do I need to change anything?
What are you `fullyParallel` and `workers` settings? By default, PWT tried to parallelize as much as possible. The `dependency` setting shown in this video only controls the order of projects run. :)
@@ChecklyHQ Thank you for your response! I wanted to clarify-should we include that type of information in the repo as well? Additionally, what would be the best approach to manage credentials securely when using Playwright in a CI/CD pipeline? I’d appreciate any suggestions or best practices you could share. Thanks again!
@@BranZafiroDk Sure thing. :) No, you wouldn't put these credential files in your repo and they should live in gitignore. (they'll also expire eventually) In CI/CD, you run the setup step as shown, generate a new credentials file, and your tests will read it to set the state. This way, everything is always fresh and you don't need to check in additional files. I hope this makes sense. :)
Just worth to mention that this method is only viable when you do not change any data on a user or in the system that is shared between tests. In most cases you need to create new user and log in via api.
Did someone try it with auth0, does it work fine? I loose login after a 2-4 tests... UPDATE: works fine, it's just logout of one of the tests cancelled the state (what a surprise) No logout - runs fine
Quick comments after releasing this video.
1. Browser session and auth data is highly time-dependent. A logged-in session might be invalid if the tests don't run for a while. Therefore, placing the generated storage state in `.gitignore` and regenerating it when the tests run is recommended.
2. You can and should only reuse session / login data when your tests aren't modifying the existing user. Otherwise, your tests will conflict with each other. In these scenarios, it's better to create a new user for each test.
Your tips have everything I was looking for, thank you so much.
I wouldn't have started my test automation project without your channel.
@@suda-nz thank you! Happy the videos are valuable. 🦝
I cannot imagine that this amazing concept exists! Thanks sir.
Neat way to reuse logged-in state, when you don't create unique users for each test and can't (or don't want to) auth via API calls!
Great content! Thanks 👍
This is useful for login with MFA functionalities.
Thanks for great video.
I have already had API authentication in place that bypassed login of every single test. So I tried your storageState approach and did very basic measuring between those two approaches.
I ran a single suite of 10 tests, with 3 workers on my workstation. I did 10 repeats and calculated average run time.
API authentication in before each: Average run time: 14486.2 ms
Setup with storageState: Average run time: 17121.5 ms
So the storageState is on average slower by 2,635.3 ms. The setup itself seems to be on average ~ 1,700 ms
This means that while each steps executes less amount of code, I guess the setting the of the storageState for each test takes a bit of time. And rough comparison would be they run by 0.1 s longer per test. I would judge that negligible.
So to sum it up. It seems storageState does not provide any performance advantage over calling auth API and setting auth tokens before every test. But storageState definitely reduces the amount of code that is in your test file and improves readability.
I would be interesting hearing your experience or anyone else on this topic. :)
@PakoVere This is great! 💙 Thank you so much for this investigation, this is super interesting.
Let me summarize if I got this correctly. In your benchmark, you only apply the already created storage state (`setup` was already done?) and this approach is slower than making an api call and setting the value to your test runs? If so, this is interesting and I woudln't have expected this outcome.
Could you share slimed down example code somewhere? I'd love to look into it!
That said, I think it's a great investigation already, but in my experience sometimes logins need a browser and authenticating via API isn't always possible. But of course, if your API approach works and is faster - that's great stuff! 🎉
This is great tip! Thanks!
this guide is fantastic! just want to ask how can we implement it in CircleCI for our test environments
In CI the setup instructions would automatically run for every run. I think it should just work. Did you encounter issues?
Спасибо за видео и время которое вы потратили на его создание. Вы очень помогли
What would be the best way to set up the same but for multiple login users and different files uses different user data?
You could still have a single "setup" project in which you write and create different storage state files. You can do this similar to how it's shown in the video for a single user.
Then you could define different projects ("User A project" and "User B project"), both relying on "setup" being a dependency but reading in the particular storage state file. "User A project" could rely on the storage state "user-a-state.json" or whatever.
Then you could control what files will belong to which project (and as a result to which user) by specifying `testMatch` on the project level (playwright.dev/docs/api/class-testproject#test-project-test-match)
Here's an untested example.
```
projects: [
{
name: "setup",
use: { ...devices["Desktop Chrome"] },
testMatch: /.*\.setup\.ts/,
},
{
name: user-a",
use: {
storageState: ".auth/user-a.json",
},
testMatch: "*/user-a/*.spec.ts",
dependencies: ["setup"],
},
{
name: user-b",
use: {
storageState: ".auth/user-b.json",
},
testMatch: "*/user-b/*.spec.ts",
dependencies: ["setup"],
},
],
```
Does this make sense?
@@ChecklyHQ Thank you so much. I will try this out. My set up is little more complex. I need to run all the tests in Desktop And in Mobile. If I follow the above suggestion, I will have to set up multiple user A desktop and user A mobile?
@ No, you should be able to have User A and User B but than would have 4 projects:
- User A desktop
- User A mobile
- User B desktop
- User B mobile
But, of course, there are multiple ways to do things, but that's how I would start without seeing code. 😅
Did you have a note about git ignoring the storage state?
Great call. I forgot about it because it seemed obvious to me that session data invalidates over time. I adjusted the pinned comment. Thanks.
Great video, everything worked like a charm! Quick question, could you theoretically use similar logic to store login state when running the scraper in a production environment with Playwright Library?
I haven't tested it, but it should work the same when using the Playwright lib. :)
The only thing to keep in mind then is that you'll have to run the setup script (it's done via projects in the video) somehow yourself. :)
@@ChecklyHQ Amazing, thank you so much for the info and quick response! I’ll report back when I cross this bridge 🙏🏾
@@stephena8965 Looking forward to hear back!
@@ChecklyHQ I can confirm this works! I took the following steps:
1. Created an async function called login.js that returns the cookies, or sets and returns them if needs be - it's important to make sure this is just a normal async function, not a self invoking one
2. Created an async getCookies function to handle the cookie logic for cleanliness. It's just a try catch statement that either returns the cookies array or an empty array
3. Call the async login function in my index.js file on each run, so now it's able to handle the cookie logic depending on whether the cookies array is empty or has length
Perhaps there's a dryer way to do this with browser contexts etc but I'm a little new to this. Everything seems to be working fine tho so I'll optimise at a later date!
Thanks again for the tutorial-couldn't have reached here otherwise!
Nice trick, now do it with session storage :3
Haha, while possible with page.evaluate() i'll leave that for another time. 🫣
@@ChecklyHQ ugh i need this!! was bummed when i only saw localStorage being saved
Thanks for the tips!
Hey Stefan, I really love your videos - they've helped me a lot! I’m currently facing an issue while testing a webpage (Web3) that interacts with the blockchain. I need to connect to a wallet before running my tests. I followed your approach of setting up the wallet connection through a setup file, as you explained in the video, but unfortunately, it doesn’t seem to be working. I think the connection isn't getting stored properly in the JSON file. Any advice on what might be going wrong? Thanks a lot!
Just a quick follow-up, Stefan - I was also wondering if you could consider making some videos about testing with extensions and Web3 in general? There's really not much material out there on this topic, and it would be super helpful to get your insights on it. Thanks again for all the great content!
@MarkusGarmeister Thanks for the kind words! 🦝 Unfortunately, I don't have any Web3 experience so I doubt we'll cover these topics any time soon. :/
Thank you, sir!
Thanks man, cool tip! You said we can remove tick from setup, so auth step is not running before each test. But, what happens, when tests are running remotely in pipeline? How could I make setup test to run only once, and then take all info from storatestate?
For example, I have 100 tests. I need setup step to run only before first step, not before all of them.
Thank you! 💙 Yes, what I meant in the video is that during development you can run the setup step once and than reuse the storageState for as long as it's valid. When running things in a CI pipeline Playwright should figure out everything for you automatically. When you run `npx playwright test` it'll figure out the project dependencies and run `setup` for each worker. As long as all your tests run in the same worker, `setup` should only run once. If you're parallizing things (which you most likely do), `setup` will run for each worker because workers can potentially run in different invironments. I hope this makes sense. :)
@@ChecklyHQ thanks for your reply
I have a question , for example if my web application has multiple user types to login (for example customer and back office user) every one of user has its own portal (with different url) , how can I use different logins (with different storageStates) within the same test ?
Playwright fixtures should be able to help out here. You could create different fixtures that rely on on different storage state. We have some fixture videos here on the channel:
- ua-cam.com/video/2O7dyz6XO2s/v-deo.html
- ua-cam.com/video/k488kAtT-Pw/v-deo.html
The docs also include some help here: playwright.dev/docs/auth#testing-multiple-roles-together
I'll put this case on the list of future videos. Thanks!
@ChecklyHQ the main issue I didn't fix it yet is to use different storage files during the test, function: test.use is not working within test
@@romag5605 Understood. The docs links should help out there. :)
Hi, thanks for this very useful tip to increase the speed of running tests in Playwright. One issue though I encountered is that if we have several tests and after 20 minutes when running other tests, it failed because the token already expired. Do you have any best practice or recommendation in alleviating this issue?
Thanks for the question.
Hmm... This is an interesting case. How many tests are we talking about here?
I'd look into ways to speed up your test first, because 20min seems like a reasonable long time for an authentication token.
Could you parallelize your tests more so that more things run to reuse the token and browser state?
@ChecklyHQ 98 tests to be exact, its running in 4 workers already
Assuming that you're running the tests sequentially in every worker (i think this is recommended in the docs somewhere? 🤔) you could parallelize even more to decrease the overall time?
If that doesn't work, unfortunately, I can't give any better advice because "refreshing storage state" seems wrong and is also missing the point of the whole concept. 😅
It works until my session is active i .e.if the maximum time for a session is 15 mins . If I try to access my stored. Json file after 15 mins it would show session time out . How to resolve it
I'm not sure if I understand your question correctly, but the session storage to be invalid at some point is expected. In CI/CD it would always run before your tests so it still speeds up your tests. In development you'd have to rerun `setup`.
Is it possible to use for API testing?
Short answer: yes. 🦝
👉 playwright.dev/docs/api-testing#reusing-authentication-state
I am working on an application where the login flow is not based on the user's input. Instead the user is redirected to an Okta auth page which internally uses kerberos and eventually redirects the user back to the application. The redirect logic and local storage update is handled by Okta auth js and okta react packages.
In this scenario, how can I stub my auth state to a temporary json file? When I try to run my tests, the auth flow breaks as Okta server doesn't recognize the user when the application is running via playwright's browser.
Heyoooo. 👋
Unfortunately, I haven't tested this flow yet and neither do I have a quick test setup to play around with it. So, unfortunately, I can't give guidance here. :/
By default we have checkly.config.ts, not playwright.config.ts right? We just need to replace it?
Great question! The `checkly.config` isn't a default replacement for your Playwright configuration because additionally it allows you to configure your Checkly monitoring infrastructure. If you want to reuse or define values in your `playwright.config` you can do that, too. :) www.checklyhq.com/docs/browser-checks/playwright-test/#global-configuration I hope this helps. 🦝
It' great. Could you make a video to guideline "Testing multiple roles with POM fixtures", please?
Could you explain a bit more? Are you asking for an example showing how to have e.g. an admin/user test case? What issues are you facing?
@@ChecklyHQ I have test cases need to be interacted between user and admin in one test. This is how I set it:
const adminContext = await browser.newContext({ storageState: '.auth/admin.json' })
const adminPage = await adminContext.newPage()
const userContext = await browser.newContext({ storageState: '.auth/user.json' })
const userPage = await userContext.newPage()
await adminPage.goto(commonData.stgUrl)
await userPage.goto(commonData.stgUrl)
But I need to paste it on each test, that is my issue. Hope that you give a suggestion
Initialization steps are taking around 5 minutes for each test case, can you please help me to resolve this situation ?
Unfortunately, we can't help with individual issues. But 5min init time indeed sounds very wrong. In these cases, trace files and UI-mode are a big help to indentify what's taking so long. Good luck!
Is there a workaround if your app is integrated with OKTA authentication?
Unfortunately neither have I done OKTA testing nor do I have an OKTA setup available. So generally, I can't give advice here right now. But if you make it to pass the OKTA 2fa flow storage state should still work. :)
7:46 this step is confusing because I’m not using the UI when configuring the CI tests right? Is this only required with the UI and otherwise tests respect the dependency array? Or would this also run setup every time before the each test?
UI mode is generally only for development. :) When you run the tests in CI or via the command line the dependency array will be respected either way and setup will run for each executed worker.
I can create the storage using setup, but it's not passing to next test. In the browser, it was showing" about:blank". But on your video pass to another test.
Without seeing your code unfortunately it's impossible to help. A link to the code is in the description.
I did the exact setup but on next load it again lands on login page.
I'm sorry to hear that. Without additional info and example code, it's impossible to help, though. You can check the working example on GitHub: github.com/checkly/playwright-examples/
I have 7 tests to run, after using this, first, it's taking only 4 workers instead of 7 workers, and after that, the other 3 workers started. Do I need to change anything?
What are you `fullyParallel` and `workers` settings? By default, PWT tried to parallelize as much as possible. The `dependency` setting shown in this video only controls the order of projects run. :)
How can we handle this on CICD ?
Heyoo. :) Can you expand the question a bit more? This approach should work in CICD without a problem. :)
@@ChecklyHQ Thank you for your response! I wanted to clarify-should we include that type of information in the repo as well? Additionally, what would be the best approach to manage credentials securely when using Playwright in a CI/CD pipeline? I’d appreciate any suggestions or best practices you could share. Thanks again!
@@BranZafiroDk Sure thing. :)
No, you wouldn't put these credential files in your repo and they should live in gitignore. (they'll also expire eventually)
In CI/CD, you run the setup step as shown, generate a new credentials file, and your tests will read it to set the state. This way, everything is always fresh and you don't need to check in additional files.
I hope this makes sense. :)
Just worth to mention that this method is only viable when you do not change any data on a user or in the system that is shared between tests. In most cases you need to create new user and log in via api.
That's a great call thank you! And for sure, it depends on the tests and what they perform!
I made a comment and pinned it. Thanks again for this valuable comment!
Did someone try it with auth0, does it work fine? I loose login after a 2-4 tests... UPDATE: works fine, it's just logout of one of the tests cancelled the state (what a surprise) No logout - runs fine
Great to hear you figured it out!