Announcing UI Tests in CI/CD for Desktop App Developers

For many years the ability to run UI tests in CI/CD has provided great value to web developers. This past Microsoft Build 2019 we were excited to announce desktop app developers can now also run UI tests in Azure DevOps! Desktop applications can now run automated UI tests in CI/CD on Azure DevOps using hosted or private agents, and setup is simple with a new Pipeline task.

Why UI tests in CI will help App Developers

Continuous Integration (CI) enables you to run automated tests of your application every time there’s a code change, and typically on servers so you’re not tying up desktop machines for testing. App developers have had the ability to run UI tests using WinAppDriver, and adding these tests to CI is important for a couple key reasons:

1. UI tests need interactive test machines. If you attempt to move the mouse, type on the keyboard, on a machine while a UI test is running, you will likely invalidate the test. This is because the tests perform automated user interactions in a way that does not differentiate from your physical actions.
2. CI is the right time to run UI Tests. There’s great value in having UI tests confirm if code changes break the user experience in your application. But equally important to having UI tests, is running them for every code change and pull request so you know exactly which commit broke the behavior. A key tenet of CI is to run every time the code changes.

How do UI tests work?

UI tests are automated tests designed to test an application by interacting with its UI in the same way a normal user would. These tests are different from unit/functional tests which test functional components of the application. A UI test typically involves going through an application in common scenarios just like a user would, verifying a user can accomplish common tasks. UI tests also typically record screenshots and often video of the test so the developer can review how the application appeared at various moments in the test.

What work was done to enable this?

For app developers to run UI tests in CI, the machine running the tests must be in an interactive state. This means the operating system has a logged in user and is listening for user input. You also need a test runner such as WinAppDriver on the test machine. Setting up a private agent in Azure DevOps with this configuration was possible, but it required significant manual configuration. Many customers requested enabling the hosted agent pool with a simple solution. The HostedVS2017 and HostedVS2019 agents are now updated with the following to enable UI tests in CI:

1. Support for interactive mode
2. Addition of WinAppDriver which executes the UI test commands

Both agents are included for free in the base Azure Pipelines plan.

In addition to the hosted agent additions there’s now a task in Azure Pipelines for easy addition of UI tests and added test configuration options. A preview of the WinAppDriver task is available today on the DevOps Marketplace. If you’re using the Classical Editor, search for “WinAppDriver” from the Add tasks menu:

For more information on the task, check out the updated CI with Azure Pipelines section on our Wiki.

Live Example: Windows Calculator

The recently open-sourced Windows Calculator project has UI Tests running in Azure DevOps as part of their CI. These tests are helping the Calculator Team catch PRs that break the user experience.

You can check out their DevOps CI/CD Pipeline fully integrated with the new WinAppDriver Task seen here:

Call to Action

Please take a look at Calculator and evaluate if adding UI tests in CI/CD will help your desktop application projects.

We’d love to hear from you. Please reach out on the WinAppDriver Issues Board on GitHub, or email [email protected]. For product announcements follow @mrhassanuz and @yosefdurr.