Etienne, thank you for publishing this content. I have found it to be very useful. Running Selenium in a pipeline is for me is a must and having used Selenium in the past I hadn't really got handle on getting it to work as you have shown, in a pipeline and within a hosted agent. This has been a great time saver - with your detailing not only the method of packaging the application but also the necessary steps in the pipeline to get the tests published, all invaluable. Thanks again.
@@TegaaaSolutions I was trying to perform "failed test case", were i can see in azure devops "Test" Tab (say e.g. 2 Pass, 1 Fail). So considering your example of google i couldn't perform, Requesting to please add a Failed test case video, whenever you get time. (I am basic learner in Selenium .Net Core)
Hello. Thanks. In release it is sending me the error: "Dotnet command failed with non-zero exit code on the following projects : D:\ag3\_w 127\a\_PruebasSelenium_ASP.NET_Core-CI\drop\SeleniumDemo.dll in the Run Selenium Tests task
Awesome video! Thank you for going through these steps. Do you have any good resources on how to better utilize Selenium for testing web applications? Thanks again!
Justin Miller do you specific questions the seleniumhq site has lots of information to author actual tests. If you have a specific problem I can maybe help you better
Hello Etienne! In the 0:44 seconds of your video you ask us to click on new solution but I don't have this option, can you please help to enable it ? thanks
Thank you for the great video! I have a question that I got " Project file(s) matching the specified pattern were not found" message, how to fix it, please?
first of all - great video. I wonder if this a dotnetcore version issue but when i create the build and dotnet publish, the chrome driver exe doesn't go to the artifact staging directory directly, it's being copied to the source directory even when it says publish chrome driver in the project file
Alexandre Salomon hey Alex thanks, did you add the bit in the csproj to publish the chrome driver at publish time. Can’t remember exactly where it is in the video. I had the same issue before adding the bit to the proj
Etienne Tremblay hey yup its there, its actually executing that task it from what i see in the logs and copying the exe from the nuget dir. i basically had to use the artifact staging directory as the output location for the build for all the driver executables to be there in the final published artifacts
Hi Etienne, Nice video, I still have a question though, You could have "run" the test in the BUILD pipeline , after building everything right ? the extra step of the RELEASE is not mandatory if you just want to run the test and not actually deploying anything. is it right to say that ? Thanks
You can absolutely run the test in the build, BUT, Selenium tests (UI/Functional Tests) are usually run against a deployed instanced. So you normally would want to run this after you released. But if you are deploying inside the build then sure you could run those test there.
Great video! One question: Is this possible to run without the headless option? My tests require some "outside the browser" interaction so I think with the headless option some tests might fail.
Well you certainly can but your build server would need to run the agent in interactive mode and you’d need to keep a session alive on the build server with the build account to do it. Check docs.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops&tabs=browser#interactive-or-service and docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows?view=azure-devops#run-interactively
Thanks. Detailed explanation. I setup selenium c# test in Azure now, but unfortunately, when the release pipeline runs, getting error in as "##[error]Project file(s) matching the specified pattern were not found." unable to solve it, any help, will be highly appreciated. Thank you,
@@TegaaaSolutions Thanks for quick response. I using windows and first build pipeline to build, restore and publish ran successfully but the release pipeline with dotnet core job using vstest fails with above mentioned error..
@@bobbysiddhu1599 ok so it’s the same agent for both build and release then. Next thing is the test task parameters are they pointing to get all test dlls? Or is there a filter that does not match the name off your test dll...
Thanks for this initial setup, very clear. I do have some questions since I did get the tests to run, but they all fail with a chromedriver timeout error. Now our application requires you to log in via AD account. Which works locally, when I'm logged in with a Pulse VPN connection. How would I approach this? Would I need to change the way I log into the account or is there another way?
Hooo, ya connecting with Windows Creds (which are different then the account running the tests I presume) is a bit tricky, you need to launch the chrome driver with another user. Check this code as a start gist.github.com/tegaaa/b48a2d0b79808b4cbb05aae850326ffe
Hi , thanks for the explanation. But it is working absolutely fine for passed test cases. However, if one test case is failing ..the release is failing..how to deal with this?
Well it depends. The intent is that if you have tests that fail you should not continue with the release since there is an issue. You should validate why the test failed, is it the test? is it the application? if it's the test, then you should fix it and retry, if it's the app then you should fix the app. In either case, something is not well with your application and that release should be rolled back or rolled forward depending on how you deal with bug at your company.
You are correct Kat but selenium test usually need the application deployed in order to run so that’s why you usually run them as part of the deployment steps
Hey Etienne, great tutorial! Could you maybe give me a hint about getting an error at the end of my test steps. The tests are running perfectly and will be published to the test section within the release, but however i receive an error which says: Total tests: 4. Passed: 3. Failed: 1. Skipped: 0. Test execution time: 50.6905 Seconds Error: The process 'C:\Program Files\dotnet\dotnet.exe' failed with exit code 1 Dotnet command failed with non-zero exit code on the following projects : d:\a 1\a\TestProject\drop\TestProject.dll
Thanks for this useful upload. I just use NUnit framework with dotnet core. I receive the same error while Releasing. It ran all my test cases but while completing the run test task it says the error above error. error Dotnet command failed with non zero exit code on the following projects. Also I checked the Tests tab, it says no Runs. Plz help me out here tried all possible hacks.. Thanks in Advance..
Let's say I have one solution and two projects: My automation (with mapping and what not) and then a unit test project (the one with my tests). How would that work? I'm trying to use these steps and it seems I'm running into an error "project files matching the specified pattern were not found" and I'm assuming it's because I have two project files?
@@TegaaaSolutions Thank you very much for responding, I'm running through the video and when it comes to Azure, I'm not able to see the same setup in the video, they must have made changes since the video but I think the problem is that this project I'm doing is .net 4.6.1 and I'm assuming maybe that's where there is a problem? I'm still a bit new to Azure but essentially when I follow the video and go to create the release, it's saying that it fan't find the path to the .dll
@@TegaaaSolutions I can't tell you how much I appreciate the help. I went back and re-did it using .net core and it's getting way further but on the run selenium tests portion of the release it's kicking me an error "The process 'CProgram Files\dotnet\dotnet.exe' failed with exit code 1" have you ever seen this happen?
Moses A which agent are you using for the release part it looks like dotnet core is not installed on that machine. Also it may be that I was pointing to it but you sont need to now
I have a run settings in my project and I can't run my test because of that, I try a lot of commands like "--Settings:Configuration/chrome.dev.runsettings" but I don't why the artefact can't find the path
@@TegaaaSolutions So, the problem wasn't that. I find the path and everything, but the problem was when to do in this way with .runsettings on release, the path couldn't be found, but if we do this in the build, everything goes fine and I have the same dashboard, graphics, and results like you, I don't find the reason to create and have this results just on release, I'll create an article or tutorial someday about it. If you can talk more about ping me here
While creating the "empty" but no empty repo, he picked the options to add a README and a .gitignore files. In any case, you should still be able to do all further steps without having those files. Cheers.
Well since cordova is built on node.js, you can deploy your app to it's destination (is it a phone application or a web site) and then run your selenium tests from a browser (if web site). If it's a phone application you'd have to setup a subscription on Visual Studio App Center and then you can use the mobile test infrastructure. But I'm not sure if that supports Selenium or you'd need to write some new tests.
Can you Create an asp.net web application which can run the selenium test you just created? For eg. We can go to this website and click on a button which says search for cheeze and on button click the script will be executed in azure and we will get the results in a textbox
Sorry for the late reply, I'm not sure I understand, you want an application that starts Azure DevOps Release Pipeline to run test and then get the result of the google query in the textbox? Not sure that makes sense...
I am getting error like ##[error]Error: There was an error when attempting to execute the process 'C:\Program Files\dotnet\dotnet.exe'. This may indicate the process failed to start. Error: spawn C:\Program Files\dotnet\dotnet.exe ENOENT, any clue?
Have you installed dotnet core on your build server? The same version that you are running locally. You'll need to install the SDK for that version on your build server.
@@TegaaaSolutions I did some searching and got a solution for this, attached the code ChromeDriver(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location), chromeoptions); ChromeOptions chromeOptions = new ChromeOptions(); var driverPath = Path.Combine(Directory.GetCurrentDirectory()); var envChromeWebDriver = Environment.GetEnvironmentVariable("ChromeWebDriver"); if (!string.IsNullOrEmpty(envChromeWebDriver) && File.Exists(Path.Combine(envChromeWebDriver, "chromedriver.exe"))) { driverPath = envChromeWebDriver; } ChromeDriverService defaultService = ChromeDriverService.CreateDefaultService(driverPath); defaultService.HideCommandPromptWindow = true; driver = (IWebDriver)new ChromeDriver(defaultService, chromeOptions); driver.get(....)
It was a nice video for starting up with Azure and able to successfully create a project, build and release. Thanks Etienne.
Etienne, thank you for publishing this content. I have found it to be very useful. Running Selenium in a pipeline is for me is a must and having used Selenium in the past I hadn't really got handle on getting it to work as you have shown, in a pipeline and within a hosted agent. This has been a great time saver - with your detailing not only the method of packaging the application but also the necessary steps in the pipeline to get the tests published, all invaluable. Thanks again.
Jon Brookes glad I could help
JUST "WOW", so clean and crisp video, explaining the selenium test as well as integrating in Azure Devops CI/CD
Thanks Satyen
@@TegaaaSolutions I was trying to perform "failed test case", were i can see in azure devops "Test" Tab (say e.g. 2 Pass, 1 Fail). So considering your example of google i couldn't perform, Requesting to please add a Failed test case video, whenever you get time.
(I am basic learner in Selenium .Net Core)
You just made my day with this video. Thank you.
Very interesting and clear thank you!
That's a good one. Thanks for sharing.
great explanation , very clear and consist, thanks for sharing.
thank you! amazing
Thanks you so much for this video man !
Thank you Sir.
Hello. Thanks. In release it is sending me the error: "Dotnet command failed with non-zero exit code on the following projects : D:\ag3\_w
127\a\_PruebasSelenium_ASP.NET_Core-CI\drop\SeleniumDemo.dll in the Run Selenium Tests task
Awesome video! Thank you for going through these steps. Do you have any good resources on how to better utilize Selenium for testing web applications? Thanks again!
Justin Miller do you specific questions the seleniumhq site has lots of information to author actual tests. If you have a specific problem I can maybe help you better
Hello Etienne! In the 0:44 seconds of your video you ask us to click on new solution but I don't have this option, can you please help to enable it ? thanks
HI, Thank you for this video , How to run those VStest in order ? can you tell me suppose if i use Nunit what will be the test result format?
Target junit you can then upload the Azure DevOps with the upload test result task
Thank you for the great video! I have a question that I got " Project file(s) matching the specified pattern were not found" message, how to fix it, please?
first of all - great video. I wonder if this a dotnetcore version issue but when i create the build and dotnet publish, the chrome driver exe doesn't go to the artifact staging directory directly, it's being copied to the source directory even when it says publish chrome driver in the project file
Alexandre Salomon hey Alex thanks, did you add the bit in the csproj to publish the chrome driver at publish time. Can’t remember exactly where it is in the video. I had the same issue before adding the bit to the proj
Etienne Tremblay hey yup its there, its actually executing that task it from what i see in the logs and copying the exe from the nuget dir. i basically had to use the artifact staging directory as the output location for the build for all the driver executables to be there in the final published artifacts
Create stuff Etienne. Thanks. Please where can we get the code you used?
I uploaded to github here github.com/tegaaasolutions/SeleniumDemo
@@TegaaaSolutions Thanks alot.
Hi Etienne,
Nice video, I still have a question though, You could have "run" the test in the BUILD pipeline , after building everything right ?
the extra step of the RELEASE is not mandatory if you just want to run the test and not actually deploying anything.
is it right to say that ?
Thanks
You can absolutely run the test in the build, BUT, Selenium tests (UI/Functional Tests) are usually run against a deployed instanced. So you normally would want to run this after you released. But if you are deploying inside the build then sure you could run those test there.
Great video! One question: Is this possible to run without the headless option? My tests require some "outside the browser" interaction so I think with the headless option some tests might fail.
Well you certainly can but your build server would need to run the agent in interactive mode and you’d need to keep a session alive on the build server with the build account to do it. Check docs.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops&tabs=browser#interactive-or-service and docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows?view=azure-devops#run-interactively
Ref: Selenium.WebDriver, Selenium.Support, Selenium.WebDriver.ChromeDriver 2.40.0, Microsoft.TestPlatform.TestHost
ChromeOptions, headless, donetcore vstest ,
csporj: true
Build: Uncheck Project deploy,
Release: --logger:trx;logfilename=TEST.xml
merge test results
Thanks. Detailed explanation. I setup selenium c# test in Azure now, but unfortunately, when the release pipeline runs, getting error in as "##[error]Project file(s) matching the specified pattern were not found." unable to solve it, any help, will be highly appreciated. Thank you,
Are you using a windows agent or a Linux one? If you are using a Linux agent you likely need to build on a Linux agent as well or vise versa
@@TegaaaSolutions Thanks for quick response. I using windows and first build pipeline to build, restore and publish ran successfully but the release pipeline with dotnet core job using vstest fails with above mentioned error..
@@bobbysiddhu1599 ok so it’s the same agent for both build and release then. Next thing is the test task parameters are they pointing to get all test dlls? Or is there a filter that does not match the name off your test dll...
@@TegaaaSolutions . Thanks,, u rightly pointed out. Pipeline was not pointed to right artifacts,..once i corrected that, it executed. Thanks
@@bobbysiddhu1599 I'm facing the same issue now what artifacts path did you change ?
Thanks for this initial setup, very clear. I do have some questions since I did get the tests to run, but they all fail with a chromedriver timeout error. Now our application requires you to log in via AD account. Which works locally, when I'm logged in with a Pulse VPN connection. How would I approach this? Would I need to change the way I log into the account or is there another way?
Hooo, ya connecting with Windows Creds (which are different then the account running the tests I presume) is a bit tricky, you need to launch the chrome driver with another user. Check this code as a start gist.github.com/tegaaa/b48a2d0b79808b4cbb05aae850326ffe
@@TegaaaSolutions thanks for the suggestion! I'll give it a try!
Hi , thanks for the explanation. But it is working absolutely fine for passed test cases. However, if one test case is failing ..the release is failing..how to deal with this?
Well it depends. The intent is that if you have tests that fail you should not continue with the release since there is an issue. You should validate why the test failed, is it the test? is it the application? if it's the test, then you should fix it and retry, if it's the app then you should fix the app. In either case, something is not well with your application and that release should be rolled back or rolled forward depending on how you deal with bug at your company.
You don't really need a release pipeline to run Selenium tests. DevOps picks up your unit test runner (xUnit, NUnit, MSTest) rather than Selenium.
You are correct Kat but selenium test usually need the application deployed in order to run so that’s why you usually run them as part of the deployment steps
@@TegaaaSolutions You can separate the automation pipeline from the deploy pipeline btw.
@@KatRollo oh y’a I know that. :)
Hey Etienne, great tutorial! Could you maybe give me a hint about getting an error at the end of my test steps. The tests are running perfectly and will be published to the test section within the release, but however i receive an error which says:
Total tests: 4. Passed: 3. Failed: 1. Skipped: 0.
Test execution time: 50.6905 Seconds
Error: The process 'C:\Program Files\dotnet\dotnet.exe' failed with exit code 1
Dotnet command failed with non-zero exit code on the following projects : d:\a
1\a\TestProject\drop\TestProject.dll
MrDanTheGentleman84 that means you have on of your test that fails and you should be able to see witch on in the tests tab in you environment log
Thanks for this useful upload. I just use NUnit framework with dotnet core. I receive the same error while Releasing. It ran all my test cases but while completing the run test task it says the error above error. error Dotnet command failed with non zero exit code on the following projects.
Also I checked the Tests tab, it says no Runs. Plz help me out here tried all possible hacks.. Thanks in Advance..
Let's say I have one solution and two projects: My automation (with mapping and what not) and then a unit test project (the one with my tests). How would that work? I'm trying to use these steps and it seems I'm running into an error "project files matching the specified pattern were not found" and I'm assuming it's because I have two project files?
No you should have two projects, your main application and one for your tests. In which project are you seeing this error?
@@TegaaaSolutions Thank you very much for responding, I'm running through the video and when it comes to Azure, I'm not able to see the same setup in the video, they must have made changes since the video but I think the problem is that this project I'm doing is .net 4.6.1 and I'm assuming maybe that's where there is a problem? I'm still a bit new to Azure but essentially when I follow the video and go to create the release, it's saying that it fan't find the path to the .dll
Moses A ahh yes this video is for .net core a little different. Also azure DevOps has for sure changed in a year
@@TegaaaSolutions I can't tell you how much I appreciate the help. I went back and re-did it using .net core and it's getting way further but on the run selenium tests portion of the release it's kicking me an error "The process 'CProgram Files\dotnet\dotnet.exe' failed with exit code 1" have you ever seen this happen?
Moses A which agent are you using for the release part it looks like dotnet core is not installed on that machine. Also it may be that I was pointing to it but you sont need to now
I have a run settings in my project and I can't run my test because of that, I try a lot of commands like "--Settings:Configuration/chrome.dev.runsettings" but I don't why the artefact can't find the path
Can’t find the path of the chrome driver? Did you add the publish section in you project?
@@TegaaaSolutions So, the problem wasn't that. I find the path and everything, but the problem was when to do in this way with .runsettings on release, the path couldn't be found, but if we do this in the build, everything goes fine and I have the same dashboard, graphics, and results like you, I don't find the reason to create and have this results just on release, I'll create an article or tutorial someday about it. If you can talk more about ping me here
I can't get to your starting point in the video. After I create an empty git repo, I don't see the files that you see in 0:35.
i can help you out if needed. PM me
they are just basic files when you initialize a repository
While creating the "empty" but no empty repo, he picked the options to add a README and a .gitignore files. In any case, you should still be able to do all further steps without having those files. Cheers.
Currently I am looking automation tool for our cordova app to be test
can i use selenium + integrate with Azure DevOps pipeline for our cordova app
Well since cordova is built on node.js, you can deploy your app to it's destination (is it a phone application or a web site) and then run your selenium tests from a browser (if web site). If it's a phone application you'd have to setup a subscription on Visual Studio App Center and then you can use the mobile test infrastructure. But I'm not sure if that supports Selenium or you'd need to write some new tests.
What is the target framework version that you used sir?
Rakesh Chanamala dotnet core 2.2 i think
Can you Create an asp.net web application which can run the selenium test you just created? For eg. We can go to this website and click on a button which says search for cheeze and on button click the script will be executed in azure and we will get the results in a textbox
Sorry for the late reply, I'm not sure I understand, you want an application that starts Azure DevOps Release Pipeline to run test and then get the result of the google query in the textbox? Not sure that makes sense...
Great video. Can selenium tests written in java run on azure?
Yes they can.
What are the settings in "Publish artifacts" ?
Roman Tsymbal look around 12:30 you should see them
Can We use Selenium JAva test project instead .Net Core in this process
M B yes you can Azure DevOps supports running java applications
@@TegaaaSolutions can you share any link for Azure devops with selenium Java
I am getting error like ##[error]Error: There was an error when attempting to execute the process 'C:\Program Files\dotnet\dotnet.exe'. This may indicate the process failed to start. Error: spawn C:\Program Files\dotnet\dotnet.exe ENOENT, any clue?
Have you installed dotnet core on your build server? The same version that you are running locally. You'll need to install the SDK for that version on your build server.
@@TegaaaSolutions it is resolved actually
chromedriver.exe is not getting created in artifact, where I am going wrong?
Did you add a publish task in your csproj?
@@TegaaaSolutions sorry for the late reply yes I did
@@TegaaaSolutions I did some searching and got a solution for this, attached the code
ChromeDriver(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location), chromeoptions);
ChromeOptions chromeOptions = new ChromeOptions();
var driverPath = Path.Combine(Directory.GetCurrentDirectory());
var envChromeWebDriver = Environment.GetEnvironmentVariable("ChromeWebDriver");
if (!string.IsNullOrEmpty(envChromeWebDriver) &&
File.Exists(Path.Combine(envChromeWebDriver, "chromedriver.exe")))
{
driverPath = envChromeWebDriver;
}
ChromeDriverService defaultService = ChromeDriverService.CreateDefaultService(driverPath);
defaultService.HideCommandPromptWindow = true;
driver = (IWebDriver)new ChromeDriver(defaultService, chromeOptions);
driver.get(....)
Why are you testing google? Setup will be complicated for any projects we develop.
Kapil Sinha its an example you can do tests for your own projects as well. You can go to the selenium site for information on how to find controls.