This blog post was authored by Craig Horsfield, a Senior SDET on the Windows Phone Test and Operations team.
This is the final post in a 3-part series that outlines key test areas that you should consider before submitting your app to the Windows Phone Store. In part 1 and part 2 we describe specific testing areas you should consider while you develop and test your app. In part 3, we describe additional areas and testing approaches to consider, including network resources, device-specific tests for hardware variations, display resolution, app upgrade, common Store test cases, and real-world testing.
Testing helps ensure that your app has a high level of performance and quality, and it helps you make sure your app is effectively represented in the Store.
Most apps depend on network access through Wi-Fi, cellular, or Bluetooth networks. If your app relies on one of these network types, you need to test the app with valid network state and invalid network state (airplane mode). You can use the Visual Studio Simulation Dashboard, which you access through the Tools menu, to control network, speed, type, and signal strength within the emulator. It’s important that your app handles any of the related app exceptions that can arise with network and web requests. Check the API in MSDN for the exception types thrown by the specific API, and then do targeted try/catch blocks. Avoid catching all exception types, for example, HttpWebRequest throws WebException – catch this and not System.Exception.
1. Verify that the app can access web resources in the active network state.
Your app should be able to transfer data as expected in normal network scenarios across the network.
2. Test that the app handles the case where the network is not available.
Consider having your app prompt the user one time to either recognize that the network is not available or enable the network. For example, “This app requires Wi-Fi or Bluetooth.”
3. Verify that the app handles dropped network connection and network request timeouts.
4. Verify that the app handles switching from Wi-Fi to cellular.
5. If your app uses Bluetooth, verify that it handles these cases:
The app should present the correct notification to the user in each state.
6. Test the app with all possible network states from the emulator.
App should present the correct notification to the user in each network state.
Not every Windows Phone has the same capabilities. Different devices may have different processor speeds, physical memory, screen resolution, and sensors.
1. Test memory scenarios.
Use the emulator to test a low-memory device scenario. When possible, Test your app on a low-memory device as well.
2. Test sensor scenarios.
3. Test resolution (more critical for native and XNA apps)
Verify the following on all supported resolutions:
Some devices have lower memory in the device. It is critical to reduce memory usage as much as possible on these phones.
1. Verify that your app’s peak memory usage is below the cap for the device you are running on.
2. Verify that your app agent’s memory usage is below the memory cap for that specific agent.
For more info about your app’s performance and memory usage, see App profiling for Windows Phone.
Windows Phone has three distinct resolutions that you should test. This is easy to do in the emulator.
1. Look for clipping of the UI that may reduce functionality or a user’s understanding of how the app works.
It is often the case that text items are set to clip and not wrap.
2. Look at how the UI layout flows on lower-resolution devices.
3. Verify the touch area of the controls.
For more info about display resolution, see Multi-resolution apps for Windows Phone 8.
Not all devices contain all sensors. Apps should check capabilities of the device and handle cases as needed. For example, not all phones have a compass. If your app uses the compass, verify that the compass is support by the device. If it is not supported, handle this case appropriately.
1. Verify that the app can start and detect hardware, and use only available hardware.
2. Test that the app does not fail if the sensor is not present.
If you release an updated version of your app, a user will interact with your app in a new way when they update the app in the Store. App data is preserved in the app ISO store when your app starts, and the new version of the app consumes data saved from the previous version of the app. If your app has changed the format of the data or the meaning of the data, the app needs to handle this when it is started and the data is loaded.
These are the steps to implement this test scenario in Visual Studio:
1. Install app version 1 using the XAPDeploy tool.
2. Run the app to generate your data that persisted in ISO store.
3. Capture the data using the ISETool.
4. Install app version 2 using XAPDeploy.
5. Restore the data from version 1 using the ISETool.
6. Test the app data in version 2 of the app against app data in version 1.
Verify that the app does one of the following:
These tests won’t affect the usability of the app or result in an app crash.
This section defines key areas you should test before submitting your app, in addition to running the Windows Phone Store Test Kit, which is available in Visual Studio with the phone development tools.
1. Verify that app screen shots look correct when uploaded as part of the app submission process.
Don’t include the debug counter in screen shots taken from the emulator.
2. Verify that app icons and Tile images are not the default images created by Visual Studio.
You can incorporate a batch of tests in your development process that covers a wider range of scenarios, from device manipulation to user language.
1. Verify orientation.
Test each page in multiple orientations. Avoid enabling orientation change in Pivot and Panorama pages. This can impact performance of the app. For more info, see Quickstart: Screen orientation for Windows Phone.
2. Verify credentials.
Valid and invalid sign-in credentials if required by the app.
3. Test text entry.
4. International: test the app using different language settings.
5. Verify app in light and dark theme.
6. App beta testing
We hope you’ve found this series on Windows Phone app testing valuable. For more info about testing your apps, see Testing apps for Windows Phone.