June 5, 2013

Testing your Windows Phone app – Part 3

Adam Denning

This blog post was authored by Craig Horsfield, a Senior SDET on the Windows Phone Test and Operations team.

– Adam

This is the final post in a 3-part series that outlines key test areas that you should consider before submitting your app to the Windows Phone Store. In part 1 and part 2 we describe specific testing areas you should consider while you develop and test your app. In part 3, we describe additional areas and testing approaches to consider, including network resources, device-specific tests for hardware variations, display resolution, app upgrade, common Store test cases, and real-world testing.

Testing helps ensure that your app has a high level of performance and quality, and it helps you make sure your app is effectively represented in the Store.

Network resources

Most apps depend on network access through Wi-Fi, cellular, or Bluetooth networks. If your app relies on one of these network types, you need to test the app with valid network state and invalid network state (airplane mode). You can use the Visual Studio Simulation Dashboard, which you access through the Tools menu, to control network, speed, type, and signal strength within the emulator. It’s important that your app handles any of the related app exceptions that can arise with network and web requests. Check the API in MSDN for the exception types thrown by the specific API, and then do targeted try/catch blocks. Avoid catching all exception types, for example, HttpWebRequest throws WebException – catch this and not System.Exception.

Test scenario	Details
1. Verify that the app can access web resources in the active network state.	Your app should be able to transfer data as expected in normal network scenarios across the network.
2. Test that the app handles the case where the network is not available.	Consider having your app prompt the user one time to either recognize that the network is not available or enable the network. For example, “This app requires Wi-Fi or Bluetooth.”
3. Verify that the app handles dropped network connection and network request timeouts.	None.
4. Verify that the app handles switching from Wi-Fi to cellular.	None.
5. If your app uses Bluetooth, verify that it handles these cases: Bluetooth enabled and disabled. Device paired and not paired. Connection lost to the Bluetooth device.	The app should present the correct notification to the user in each state.
6. Test the app with all possible network states from the emulator.	App should present the correct notification to the user in each network state.

Device-specific tests for hardware variations

Not every Windows Phone has the same capabilities. Different devices may have different processor speeds, physical memory, screen resolution, and sensors.

Test scenario	Details
1. Test memory scenarios.	Use the emulator to test a low-memory device scenario. When possible, Test your app on a low-memory device as well.
2. Test sensor scenarios.	Verify that the app sets the correct capabilities so that unsupported capabilities prevent the app from installing. For example, if the app requires the compass and the device does not support this type of sensor, the app does not install. If the sensor is not critical to the app, test to verify that the app handles sensor API exceptions.
3. Test resolution (more critical for native and XNA apps)	Verify the following on all supported resolutions: UI layout and appearance. Touch input on the app controls.

Memory

Some devices have lower memory in the device. It is critical to reduce memory usage as much as possible on these phones.

Test scenario	Details
1. Verify that your app’s peak memory usage is below the cap for the device you are running on.	None.
2. Verify that your app agent’s memory usage is below the memory cap for that specific agent.	None.

For more info about your app’s performance and memory usage, see App profiling for Windows Phone.

Display resolution

Windows Phone has three distinct resolutions that you should test. This is easy to do in the emulator.

Test scenario	Details
1. Look for clipping of the UI that may reduce functionality or a user’s understanding of how the app works.	It is often the case that text items are set to clip and not wrap.
2. Look at how the UI layout flows on lower-resolution devices.	None.
3. Verify the touch area of the controls.	None.

For more info about display resolution, see Multi-resolution apps for Windows Phone 8.

Sensors

Not all devices contain all sensors. Apps should check capabilities of the device and handle cases as needed. For example, not all phones have a compass. If your app uses the compass, verify that the compass is support by the device. If it is not supported, handle this case appropriately.

Test scenario	Details
1. Verify that the app can start and detect hardware, and use only available hardware.	None.
2. Test that the app does not fail if the sensor is not present.	None.

App upgrade

If you release an updated version of your app, a user will interact with your app in a new way when they update the app in the Store. App data is preserved in the app ISO store when your app starts, and the new version of the app consumes data saved from the previous version of the app. If your app has changed the format of the data or the meaning of the data, the app needs to handle this when it is started and the data is loaded.

These are the steps to implement this test scenario in Visual Studio:

Test step	Details
1. Install app version 1 using the XAPDeploy tool.	None.
2. Run the app to generate your data that persisted in ISO store.	None.
3. Capture the data using the ISETool.	None.
4. Install app version 2 using XAPDeploy.	None.
5. Restore the data from version 1 using the ISETool.	None.
6. Test the app data in version 2 of the app against app data in version 1.	Verify that the app does one of the following: Loads and converts the data. Discards the data and reinitializes the app. These tests won’t affect the usability of the app or result in an app crash.

Common Windows Phone Store test cases

This section defines key areas you should test before submitting your app, in addition to running the Windows Phone Store Test Kit, which is available in Visual Studio with the phone development tools.

Test scenario	Details
1. Verify that app screen shots look correct when uploaded as part of the app submission process.	Don’t include the debug counter in screen shots taken from the emulator.
2. Verify that app icons and Tile images are not the default images created by Visual Studio.	None.

Real-world testing

You can incorporate a batch of tests in your development process that covers a wider range of scenarios, from device manipulation to user language.

Test scenario	Details
1. Verify orientation.	Test each page in multiple orientations. Avoid enabling orientation change in Pivot and Panorama pages. This can impact performance of the app. For more info, see Quickstart: Screen orientation for Windows Phone.
2. Verify credentials.	Valid and invalid sign-in credentials if required by the app.
3. Test text entry.	Verify that proper input scope is set for text boxes. Verify that the correct keyboard is the default for the required input type. Make sure pages that require a lot of text entry support landscape mode. Verify cut-and-paste input for text input.
4. International: test the app using different language settings.	Ensure the correct resource strings are loading for each user language. Test Left to Right and Right to Left text input. Verify that the app can handle non ANSII input characters. Test Date and Time Input if applicable; format differs by locale.
5. Verify app in light and dark theme.	Verify that all text and visual context is visible. Change phone default accent color and ensure that the visual state of the app is acceptable in both light and dark themes.
6. App beta testing	Use the app in real network conditions. Use the app with real data, not just test data. Verify the app in low-battery scenarios. Verify the app when connected to Bluetooth audio.

We hope you’ve found this series on Windows Phone app testing valuable. For more info about testing your apps, see Testing apps for Windows Phone.

Tags:

Developing Apps

Testing Apps