Options for publishing Windows Phone beta apps and testing in-app purchase

Developers frequently send us questions asking for the best publishing option to test their apps, benefits and limitations of beta publishing, as well as how to test in-app purchase. Testing is an important step in the development process to minimize problems, and this in turn can lead to higher ratings potentially  generate more downloads.

Today I want to share a framework for how to think about the different publishing and testing options, which can help you identify the best alternative for your project. I’ll then share details for testing in-app purchase.

Which options are available?

The Windows Phone Store offers a wide range of publishing options, for different levels of app testing:

1. Run app in the emulator – test app functionality using the emulator

2. Run app on a phone – test app performance using a developer-unlocked phone

3. Publish a beta app – test app with a select group of testers

4. Publish a hidden app – distribute your app to many users, in a production environment

5. Publish a visible app – distribute your app to all users in a production environment, in the public Store

6. Enterprise deployment – distribute an app to employees in your company

Comparing the options

This table summarizes the options, and accompanying benefits and limitations:

 

Characteristics

1. Run app in the emulator

2. Run app on a phone

3. Publish a beta app

4. Publish a hidden app

5. Publish a visible app

6. Enterprise deployment

Best for

Development, functionality testing

Development, performance testing

Testing with selected group of users

Testing the production ready app, confidentiality is not that critical or Microsoft account of users is unknown

Global availability

Employees in company

Who can use the app?

Developer that has the code

Only one person on that phone

Up to a few thousand users that the developer whitelisted based on Microsoft account email

Any user that has the URL link

All consumers in any of the 191 markets your app is published to

Employees of that company

Is app available in the public store?

No

No

Yes (can be downloaded only by whitelisted users, and cannot be discovered through search or browse)

Yes (cannot be discovered through search or browse)

Yes

No

Does the app go through certification?

No

No

No

Yes

Yes

No

Is the app visible to users or crawlers?

No

No

Partial (metadata could be visible: app name, icon, description, screenshots)

Yes

Yes

No

Test ads shown through Microsoft Ad SDK?

No (only ‘test ads’ can be shown)

Yes

Yes

Yes

Yes

Yes

Test in-app purchase using WP API?

Yes (see next section for details)

Yes (see next section for details)

Yes – in-app purchase always free

Yes

Yes

No – in-app purchase is not supported because app is not public

Key advantages

Rapid code development and testing

Unit testing on a production phone, all features operational, validating app speed and performance

Only whitelisted users can access the full app. More rapid turnaround from submission availability due to no certification validation.

Test in production environment, and take app through certification process, testing large number of anonymous users

Global use

Controlled deployment. Apps can be removed from phone by IT. No time required for public app cert.

Key disadvantages

Can’t test speed and performance. Can’t test some features easily (multi touch, compass, gyro, ads).

Only one phone at a time, no global testing

Need the Microsoft account emails for every whitelisted users. Need to copy all metadata and In-App purchase again to the production app.

Not use for those who need confidential publishing: users can share the link and crawlers can find the app. Turn-around time is the same as a publically visible app.

Not good for testing as consumers will expect a final product, and will rate the app as a finished app

Enterprise set up requires more time (need key, Symantec cert, central IT process, Company Hub…). Apps only available to employees.

 

What is a typical workflow for developing and publishing a new app?

As I develop apps, my typical workflow is the following:

  • Test in the emulator – Iterative development and functional testing using the emulators. I test using the multiple resolutions of the Windows Phone 8 emulators, as well as Windows Phone 7.
  • Test performance on a phone – Once the core functionality is ready I validate the app performance and memory usage on a variety of phones to ensure the app works well on both Windows Phone 8 and Windows Phone 7 devices with different memory configurations. I run the performance analysis tools to understand, on a real phone, what functions and that are taking more time or using more memory.
  • Private beta - Once I finish the development, I sometimes publish a beta app. I do this only for apps that are brand-new, and that explore brand-new concepts that I have not tried before, and where I need feedback. I invite to the beta a selected number of trusted users. This way I can get feedback from a (small) sample of users from a variety of phones. Their feedback always helps me fix important issues before the app is offered to all users.
  • Public beta (hidden) – I then publish the app hidden, and validate that everything works correctly, that the app has the right name and I used the correct pricing, and that the trial version is working. I also do thorough testing on in-app purchase.
  • Production – Once I finalize all the tests, I publish the app in production, visible, in as many markets as possible.

This example can help you decide what will be your own development/publishing/testing workflow for your apps.

Steps to implement each of the different options

Run project in the emulator

Create your app in Visual Studio, then select the emulator that you want to test on and click the green arrow to run. The emulators available can show how the app will look and run in Windows Phone 7, Windows Phone 8, and on different memory and screen resolution phones. The emulator has a different performance and memory profiling than a real phone, so the emulator is ideal for feature testing, but not good for performance testing. Learn more about this option.

clip_image002

Run project on a phone

Developer-unlock your phone. If you have a Windows Phone 7 device, connect it to your development machine, and run the Zune client. If you are using a Windows Phone 8 device, this step is not needed. Then, run the app “Windows Phone Developer Registration” tool, and unlock your device. If you have too many devices already registered, go to Dev Center, and then Account, and remove one of the devices you are not using.

clip_image004

Connect your developer-unlocked phone, and in Visual Studio, select Device and click the green arrow to run. You can even debug an app on the phone.

clip_image006

Publish a beta app

Compile your app and generate a XAP. The XAP file will be found in your project folder, Bin, Release (or Debug) folder. Then in Dev Center create a new product. During the submission process, select BETA publishing, and then add up to 10,000 Microsoft account emails for beta testing. Learn more about beta publishing. Once the beta app has been published, you’ll need to send the link to your beta testers so they can install the app on their phones.

clip_image008

clip_image010

Publish a hidden app

Compile your app and generate a XAP. The XAP file will be found in your project folder, Bin, Release (or Debug) folder. Then in Dev Center publish your app. During the submission process, select PUBLIC STORE, and turn on “Hide from users browsing or searching the Store”. Be careful not to mark an existing public app as hidden; this would make your existing app no longer appear for users who are searching for, or want to download your app.

clip_image012

clip_image014

Once you finish the publishing process, the app goes through the certification validation. When the app is live, you can use the Link to the Store link, under App properties, to share that link with all the users that you want.

clip_image016

Publish a visible app

This is the standard way to publish your app through Dev Center. Make sure you select as many markets as needed to ensure maximum reach of your app. You can do that in the App info section, and then choose individual markets in Market selection and custom pricing. See how to publish your first app.

clip_image018

Enterprise deployment

Use this option to distribute apps to your employees, without publishing your app in the public store. Read more about the steps and process.

Testing in-app purchase

When you add in-app purchase to your app, it is always important that you test your apps for the most common failure scenarios. As you develop your test plan, there are several different mechanisms that you should understand:

 

Characteristics

1. Test using the emulator

2. Test using a beta app

3. Test using a production app

4. Test using the emulator with real in-app purchase

Best for

Rapid prototyping, testing complex in-app purchase scenarios

Testing by more than one user

Testing production code, with zero or non-zero prices

Rapid testing using the production in-app purchase

How to enable it

Use the mock library

Publish a beta app, and add in-app purchase items to the beta app

Publish your app, hidden, in only one market (to minimize risks). Set in-app purchase prices to $0. Once it is published, test on a real phone. Once testing is finished, update the app, remove the ‘hidden’ flag, set the correct prices, and publish in more markets.

Publish an app, then take the Product ID of the published app and add it to the Visual Studio project. This allows the project to use the production in-app purchase products.

Allows resetting purchases

Yes

No

No

Yes

Requires money to purchase

No

No (always zero price)

Yes (if price is set to > $0)

No (always zero price)

Testing speed

Very rapid prototyping

Some delay to publish beta app

Delay to wait until app is published

Very rapid prototyping

Complexity of deployment

High, requires adding mock library

Medium, test like a real app, though requires publishing all the app and in-app purchase again to a production app after testing is finished.

Low, test like a real app

Low, test like a real app

Risks

Requires changes to the code

Requires publishing app and metadata again

Hidden app could be found by users or crawlers

None

In my own apps, I usually test using the emulator (1), then publish a hidden app and test using a production app (3), and then continue development using the live in-app purchase (4) once the first version of the app has been published – this lets me test with production apps, and I’m not very concerned about the confidentiality of my apps. Every case is different, so you might find the first two options better because they allow faster prototyping and higher confidentiality in testing.

To make option 4 work, take the product ID of your published app from the app details page in Dev Center, and add it in your Visual Studio project, in the manifest packaging. This way the Visual Studio development product is the same as the ID of the published app.

clip_image020

Testing behavior of an app upgrading from Windows Phone 7.X to 8.0

Another scenario that developers want to test is upgrading an app from Windows Phone 7.x XAP to 8.0 XAP. More developers are adding an 8.0 XAP version of the app to take advantage of new functionality available only in Windows Phone 8 XAPs. The app will have two XAPs: 7.x and 8.0, and Windows Phone 8 devices running the 7.x XAP will be able to upgrade to the 8.0 XAP. It is useful to confirm that your app will function correctly during the upgrade.

To test this behavior, publish a beta app that only has the 7.x XAP. Download and use the app. Then add the 8.0 XAP in Dev Center, and publish the app again.  After it is published, the beta app will download as an ‘upgrade’ in the phone, and you can validate that the upgrade scenario will work correctly before taking your upgrade live.

 

 

I hope this framework can help you make the best decision for testing your apps, and make sure in-app purchase works as well as possible. We’d love to hear your questions and suggestions for publishing and testing Windows Phone apps. Leave a comment here, or tweet at @wpdev.