Developers frequently send us questions asking for the best publishing option to test their apps, benefits and limitations of beta publishing, as well as how to test in-app purchase. Testing is an important step in the development process to minimize problems, and this in turn can lead to higher ratings potentially generate more downloads.
Today I want to share a framework for how to think about the different publishing and testing options, which can help you identify the best alternative for your project. I’ll then share details for testing in-app purchase.
Which options are available?
The Windows Phone Store offers a wide range of publishing options, for different levels of app testing:
1. Run app in the emulator – test app functionality using the emulator 2. Run app on a phone – test app performance using a developer-unlocked phone 3. Publish a beta app – test app with a select group of testers 4. Publish a hidden app – distribute your app to many users, in a production environment 5. Publish a visible app – distribute your app to all users in a production environment, in the public Store 6. Enterprise deployment – distribute an app to employees in your company
1. Run app in the emulator – test app functionality using the emulator
2. Run app on a phone – test app performance using a developer-unlocked phone
3. Publish a beta app – test app with a select group of testers
4. Publish a hidden app – distribute your app to many users, in a production environment
5. Publish a visible app – distribute your app to all users in a production environment, in the public Store
6. Enterprise deployment – distribute an app to employees in your company
This table summarizes the options, and accompanying benefits and limitations:
1. Run app in the emulator
2. Run app on a phone
3. Publish a beta app
4. Publish a hidden app
5. Publish a visible app
6. Enterprise deployment
Development, functionality testing
Development, performance testing
Testing with selected group of users
Testing the production ready app, confidentiality is not that critical or Microsoft account of users is unknown
Employees in company
Who can use the app?
Developer that has the code
Only one person on that phone
Up to a few thousand users that the developer whitelisted based on Microsoft account email
Any user that has the URL link
All consumers in any of the 191 markets your app is published to
Employees of that company
Is app available in the public store?
Yes (can be downloaded only by whitelisted users, and cannot be discovered through search or browse)
Yes (cannot be discovered through search or browse)
Does the app go through certification?
Is the app visible to users or crawlers?
Partial (metadata could be visible: app name, icon, description, screenshots)
Test ads shown through Microsoft Ad SDK?
No (only ‘test ads’ can be shown)
Test in-app purchase using WP API?
Yes (see next section for details)
Yes – in-app purchase always free
No – in-app purchase is not supported because app is not public
Rapid code development and testing
Unit testing on a production phone, all features operational, validating app speed and performance
Only whitelisted users can access the full app. More rapid turnaround from submission availability due to no certification validation.
Test in production environment, and take app through certification process, testing large number of anonymous users
Controlled deployment. Apps can be removed from phone by IT. No time required for public app cert.
Can’t test speed and performance. Can’t test some features easily (multi touch, compass, gyro, ads).
Only one phone at a time, no global testing
Need the Microsoft account emails for every whitelisted users. Need to copy all metadata and In-App purchase again to the production app.
Not use for those who need confidential publishing: users can share the link and crawlers can find the app. Turn-around time is the same as a publically visible app.
Not good for testing as consumers will expect a final product, and will rate the app as a finished app
Enterprise set up requires more time (need key, Symantec cert, central IT process, Company Hub…). Apps only available to employees.
As I develop apps, my typical workflow is the following:
This example can help you decide what will be your own development/publishing/testing workflow for your apps.
Create your app in Visual Studio, then select the emulator that you want to test on and click the green arrow to run. The emulators available can show how the app will look and run in Windows Phone 7, Windows Phone 8, and on different memory and screen resolution phones. The emulator has a different performance and memory profiling than a real phone, so the emulator is ideal for feature testing, but not good for performance testing. Learn more about this option.
Developer-unlock your phone. If you have a Windows Phone 7 device, connect it to your development machine, and run the Zune client. If you are using a Windows Phone 8 device, this step is not needed. Then, run the app “Windows Phone Developer Registration” tool, and unlock your device. If you have too many devices already registered, go to Dev Center, and then Account, and remove one of the devices you are not using.
Connect your developer-unlocked phone, and in Visual Studio, select Device and click the green arrow to run. You can even debug an app on the phone.
Compile your app and generate a XAP. The XAP file will be found in your project folder, Bin, Release (or Debug) folder. Then in Dev Center create a new product. During the submission process, select BETA publishing, and then add up to 10,000 Microsoft account emails for beta testing. Learn more about beta publishing. Once the beta app has been published, you’ll need to send the link to your beta testers so they can install the app on their phones.
Compile your app and generate a XAP. The XAP file will be found in your project folder, Bin, Release (or Debug) folder. Then in Dev Center publish your app. During the submission process, select PUBLIC STORE, and turn on “Hide from users browsing or searching the Store”. Be careful not to mark an existing public app as hidden; this would make your existing app no longer appear for users who are searching for, or want to download your app.
Once you finish the publishing process, the app goes through the certification validation. When the app is live, you can use the Link to the Store link, under App properties, to share that link with all the users that you want.
This is the standard way to publish your app through Dev Center. Make sure you select as many markets as needed to ensure maximum reach of your app. You can do that in the App info section, and then choose individual markets in Market selection and custom pricing. See how to publish your first app.
Use this option to distribute apps to your employees, without publishing your app in the public store. Read more about the steps and process.
When you add in-app purchase to your app, it is always important that you test your apps for the most common failure scenarios. As you develop your test plan, there are several different mechanisms that you should understand:
1. Test using the emulator
2. Test using a beta app
3. Test using a production app
4. Test using the emulator with real in-app purchase
Rapid prototyping, testing complex in-app purchase scenarios
Testing by more than one user
Testing production code, with zero or non-zero prices
Rapid testing using the production in-app purchase
How to enable it
Use the mock library
Publish a beta app, and add in-app purchase items to the beta app
Publish your app, hidden, in only one market (to minimize risks). Set in-app purchase prices to $0. Once it is published, test on a real phone. Once testing is finished, update the app, remove the ‘hidden’ flag, set the correct prices, and publish in more markets.
Publish an app, then take the Product ID of the published app and add it to the Visual Studio project. This allows the project to use the production in-app purchase products.
Allows resetting purchases
Requires money to purchase
No (always zero price)
Yes (if price is set to > $0)
Very rapid prototyping
Some delay to publish beta app
Delay to wait until app is published
Complexity of deployment
High, requires adding mock library
Medium, test like a real app, though requires publishing all the app and in-app purchase again to a production app after testing is finished.
Low, test like a real app
Requires changes to the code
Requires publishing app and metadata again
Hidden app could be found by users or crawlers
In my own apps, I usually test using the emulator (1), then publish a hidden app and test using a production app (3), and then continue development using the live in-app purchase (4) once the first version of the app has been published – this lets me test with production apps, and I’m not very concerned about the confidentiality of my apps. Every case is different, so you might find the first two options better because they allow faster prototyping and higher confidentiality in testing.
To make option 4 work, take the product ID of your published app from the app details page in Dev Center, and add it in your Visual Studio project, in the manifest packaging. This way the Visual Studio development product is the same as the ID of the published app.
Another scenario that developers want to test is upgrading an app from Windows Phone 7.x XAP to 8.0 XAP. More developers are adding an 8.0 XAP version of the app to take advantage of new functionality available only in Windows Phone 8 XAPs. The app will have two XAPs: 7.x and 8.0, and Windows Phone 8 devices running the 7.x XAP will be able to upgrade to the 8.0 XAP. It is useful to confirm that your app will function correctly during the upgrade.
To test this behavior, publish a beta app that only has the 7.x XAP. Download and use the app. Then add the 8.0 XAP in Dev Center, and publish the app again. After it is published, the beta app will download as an 'upgrade' in the phone, and you can validate that the upgrade scenario will work correctly before taking your upgrade live.
I hope this framework can help you make the best decision for testing your apps, and make sure in-app purchase works as well as possible. We’d love to hear your questions and suggestions for publishing and testing Windows Phone apps. Leave a comment here, or tweet at @wpdev.