Instrumenting Your App for Telemetry and Analytics

Publishing your app through the Windows Store and the Windows Phone Store is just the first step in your app’s relationship with its customers. Certainly you hope that many customers will find and install your app, and also leave ratings and reviews. The Store dashboards will also give you information like average app usage per day with comparisons to other apps in your category, quality reports (crash dumps, unresponsive events, and exceptions), ratings and reviews, adoption rates, download history, in-app purchases, and app sales. More details, including how to enable certain data collection, is found on Collecting telemetry data from your apps. Third party providers such as Distimo and AppFeds also produce Store-wide analytics, which is very valuable for tracking overall market trends and activities.

However, none of this really tells you about users’ specific activity within the app itself. Such activity cannot be generalized or collected automatically. Collecting that data is the purpose of instrumenting your app from the inside for telemetry and analytics.

Logging versus telemetry

You might be thinking, “Hey, no problem—I’ve littered my code with all kinds of logging. Won’t that work?” Well, logging is a good start, but it is generally a tool that you use during development so that you can diagnose errors and code flows. Logging is typically oriented around the internal structure of your app rather than reflecting real-world customer usage. In short, logging is how you collect data about your app in the lab; instrumenting your app for telemetry, on the other hand, is how you collect data once the app is released into the wild.

Telemetry, or tele-metering, is automated remote measurement and data collection. It’s used in all kinds of industries, from tracking spacecraft, tracking wildlife, medical monitoring, law enforcement, and so on. The electrical meter attached to my house, for example, automatically sends readings to the utility company, which has eliminated the need for human meter readers. I especially like to think of telemetry in the context of spacecraft, where it is the way in which mission controllers monitor the health and operation of a very expensive piece of hardware that is completely unreachable any other way.

Putting an app into the global market is, in many ways, similar to launching a spacecraft: without telemetry, you’re flying blind. Your app represents a significant investment, and once it gets out to more than a handful of customers the main issue is not so much distance, but scale. And what you need is a way to both collect the data and then condense it into useful analytics, or reports that human beings can read to make decisions.

A gold mine

These analytics are a gold mine. They let you look over your user’s shoulders (respecting privacy of course!) and know what they’re really doing. This enables you to answer many questions, such as:

• How are customers really engaging with the app? Do people use the features you thought they wanted?
• Where are they spending (or not spending) their time? How long do users spend in each app session, between sessions, and between suspend/resume?
• What device configurations do your users prefer (view size, screen orientation, display types, input modalities).
• How are customers using options you provide via Settings?
• What’s going on when crashes happen? How often do users encounter non-crashing errors such as failed HTTP requests, failed syncs, timeouts, etc.?
• How successful and engaging are the various features of the app? Are social features being used? Are there usage patterns that you can reward or discourage in some way to drive behavior?
• How successful are your trial versions and/or in-app purchases? Where are trial conversion reminders and/or in-app purchase options most effectively displayed? How many users look at purchase options but don’t buy anything?
• Are users clicking ads (or converting to a paid version to get rid of them)?
• How often do users run online vs. offline?
• Where should you concentrate future investments for quality, performance and feature updates? What ideas could you test in your next update or in a separate app?
• Is the app truly serving your business goals?

I can’t imagine that anyone who is serious about their apps business would not want answers to these kinds of questions, which is why many app developers consider telemetry a must-do for their apps, starting with any beta-testing or early previews they might put out.

Designing telemetry for your business goals

When you publish an app, it really means you’re going into business whether you realize it or not! Telemetry becomes very important, then, because it tells you a lot about how successfully your app is fulfilling your business goals. I like to think of those goals as the four “F’s”—the four motivations for writing apps. Here’s how telemetry is typically used for each:

Business Goal	Use of Telemetry
Fame	Priority: Growing a customer base. Are you acquiring new customers? How are you acquiring them? What strategies in the app lead to new customers?
Fortune	Priority: Monetization. Assuming you’ve done market analysis on the competition, are your differentiating features succeeding? Are your monetization strategies being employed? What use patterns can you take advantage of to further monetize?
Fun	Priority: Sharing your joy. Are your customers enjoying themselves and perhaps sharing that joy with their social network?
Philanthropy	Priority: Promoting a message. Are customers inspired by the message and the cause? Are customers willing to support the cause?

The questions you want to answer with your app will lead directly into how you instrument your app to gather telemetry. That is, your questions lead to telemetry design, and your telemetry design leads to what are called events. Events are broad, human-readable verbs or actions that you want to track—generally just strings that you use to categorize your telemetry data. This way they become the basic unit of organization for the analytics that you’ll eventually get from your telemetry.

Typical apps log about 30 distinct events, which often include (but are not limited to) app start, app exist, registration, login/log off, settings changes, content sharing, recoverable errors, non-recoverable exceptions, view content, mark as favorite, comment on content, viewing and item or category in a catalog, search or filter, add to wishlist, begin/complete/abandon checkout, invite friend, accept a friend’s invitation, game started, game completed, hint requested, and so on.

The relatively small number of events is driven by the fact that each one surfaces separately in the analytics reports that are generated from the telemetry—think of each event translating to a chart. Too many (or too few) events makes it difficult to harvest actionable information from the analytics, so when you think about events try to think clearly about the insights you want to gain. For example, it’s more important to track what content items are being tapped or clicked rather than lower-level pointer events (except perhaps to gather secondary data about the type of input, whether mouse, touch, or stylus).

Events help identify users’ flow through the app and per-page feature usage. As such, they are usually static (not dynamically generated) and are usually somewhat generic, such as “Article read.” Event attributes or properties (think adjectives or subject nouns) then provide specific details, such as the URI or title of the article being read. This way the top-level events answer one level of questions about app usage, with the attributes providing another level. This becomes very helpful when generating charts from all this data, where the overall chart is for an event with attributes and properties providing the individual data points, bars, lines, and so on.

If you think about charting—which is probably the most common way to consume analytics—it’s also important that the properties and attributes you assign to events will chart well. For this reason, numerical attributes that can vary widely are best grouped into buckets or ranges, rather than reported as discrete values. For example, you could track the time spent on a given page or the time it took to complete a game level in terms of 0–5s, 6–10s, 11–20s, 21–60s, 61s–120s, and so on, so that a pie chart or bar chart has a reasonable and meaningful number of elements.

It’s also best to report events at the end of an action when you have all the data you might attach to it, thereby making each event as rich as you can and reducing clutter in your analytics.

Instrumenting your app

I hope that I’ve convinced you that instrumenting your app for telemetry is a good idea! So how then do you go about it? Well, you have two choices:

• The hard way: Implement some kind of tracing/logging API of your own, sprinkle calls to that API throughout your app, implement a backend service to which you regularly upload your collected data (and make sure it can scale to thousands or tens of thousands of users), and then implement an entire analytics engine to process that data into meaningful reports.
• The easy way: Register with a third-party analytics provider who is in the business of doing most of the hard work for you. Incorporate their SDK into your app, make calls to their API as needed, and spend most of your time using the analytics to create better apps!

Unless you already have an infrastructure in place, I suspect that you’ll choose the easy way! If you go to the Windows Partner Directory and click the Analytics filter on the left, you’ll see a number of companies that offer great tools in this regard. These companies are in the business of turning your raw telemetry data into useful analytics, a job that I’m more than happy to defer to them!

Do note that the Partner Directory intermixes partners that provide telemetry services, like we’re talking about here, and those who provide services like error tracking, market performance tracking, or marketwide trend reports (which are also useful, but different). Just spend a little time looking through the list and you’ll find the right ones such as Localytics, MarkedUp, AppFireworks, mtiks, Adobe Omniture, and Parse. (Note that Flurry, the most popular analytics SDK for Windows Phone and other platforms, is not listed because it’s not available for Windows Store apps at present. Many of the others in this list work with both Windows Store and Windows Phone apps. Furthermore, check that the provider’s SDK supports the language you’re working in.) Microsoft is also busy creating an analytics platform of its own called Application Insights.

In any case, once you’ve chosen a provider, the first step is to visit its developer portal and create an account. Most offer a free service tier so that you can get started without any up-front cost and start paying only when your app is successful enough to be generating significant data.

Next, download the appropriate SDK and incorporate it into your project, and then register an app for which you receive a unique key. You provide this key when you initialize the SDK’s main object, and then you call that object’s methods to log your events. Depending on the SDK, you might need to request an upload of the data, or it might do it automatically. Either way, after using the app for some time, you visit the provider’s portal and explore the results.

A few tips:

• Use separate development, beta test, and production keys. The purpose of telemetry is to gather actionable data from real-world users, so you always want to keep any data gathered during your development phase separate from real customer data. It’s likely that you’ll also want to separate beta testing data (e.g., for side-loaded versions, which probably has correlations with distinct time periods) from final production data (for the app acquired from the Store). This means creating separate keys through the provider’s portal and make sure to change them when you build each variant of the app.
• Create a telemetry layer in your app. This allows you to easily switch providers at any time, if the need arises, and encapsulates any computation or bucketing logic behind the layer’s interface so that it doesn’t interfere with the rest of the app. A simple enablement flag that you check within each method of your layer also makes it very easy to turn telemetry on and off (as when the user opts out) without touching any other part of your code.
• Test your telemetry by visiting the analytics portal early and often. At the end of the day, you’re doing all of this instrumentation to get actionable insights from the provider’s analytics. So even when you first start doing your instrumentation, gather some data and then visit the provider’s portal to see the results. You’ll quickly find whether the events you defined are meaningful, whether the attributes and properties you include with the events have the right granularity, and whether you’re correctly reporting that data. For example, if you see null showing up in some attribute, you’ll know that you didn’t use the right variable when reporting the event.
• Test when offline. This ensures that your telemetry layer doesn’t make assumptions about connectivity.
• Be sure to turn on the Internet (Client) capability in your manifest. Otherwise nothing will ever make it to the backend! Note that so long as you collect no personal information, email addresses, screenshots, or browsing history, gathering telemetry should not affect your app’s required age rating. However, you will need to include a privacy policy.
• Inform the user and allow opt out. Telemetry data is a form of user information, though not personally identifiable. For this reason, make sure that your app’s listing in the Store and your privacy policy states what you collect and how it will be used. Also provide an option in your Settings to disable telemetry altogether if the user chooses to opt out.

As a short example, I’ve instrumented the 15+Puzzle game that I have in the Windows Store with telemetry so that I can determine usage patterns, especially patterns that affect different monetization strategies. I used the Localytics SDK for JavaScript, but anything specific to that SDK is isolated in my telemetry layer. That layer is implemented in my own Telemetry namespace that’s defined as follows, where the init function gets everything going:

WinJS.Namespace.define("Telemetry", {
    inputType: { touch: 0, mouse: 1, keyboard: 2},

    session: null,
    enabled: true,
    _curPage: null,
    _lastNavTime: 0,
    _settingsTime: 0,

    _timeLastStarted: 0,
    _timeLastCompleted: 0,
    _inputTally: new Array(3),

    _now: function () {
return new Date().getTime();
    },

    _convertToRange: function (value, granularity) {
//For telemetry, we don’t want to log every number, but should group them together
// into ranges. This method returns a string based on the granularity. For 25 for
// example, it’ll create strings "0-24", "25-49", "50-74", etc.

//Special case a true zero
if (value === 0) {
return "0";
        }

var rangeBase = Math.floor(value / granularity);
return (granularity * rangeBase) + "-" + ((granularity * (rangeBase + 1)) — 1);
    },

    init: function () {
var keyDeveloper = "...";
var keyBeta = "...";
var keyProduction = "...";

        Telemetry.session = LocalyticsSession(/* key */);
        Telemetry.session.open();
        Telemetry.session.upload();
        } else {
            Telemetry.session = null;
        }

        Telemetry.enabled = (Telemetry.session !== null);

//Automatically log app errors
        WinJS.Application.addEventListener("error", function I {
            Telemetry.error("WinJS.Application.onerror", { "Line": e.detail.errorLine,
"Character": e.detail.errorCharacter, "File": e.detail.errorUrl,
"Message": e.message });
        });
    },

// Other event methods, such as error
}

You can see here that I have a generic bucketing method, _convertToRange, which I use elsewhere in the layer, whose other methods reflect my event design. These include: error, appStarted, syspending, resuming, visibilityChange, licenseChanged, appResized, newGrid, restarted, pageNav, scoredCleared, scoresViewChanged, gameStarted, gameRestarted, gameCompleted, tallyInput, optionsEnter, optionsExit, settingsEnter, settingsExit, and feedback.

This last one, by the way, is how I log user comments in a Feedback panel that’s part of the app’s Settings. This way I don’t need some other service or web form to collect the data: I can just create whatever form I want directly within Settings.

To show what I mean by encapsulating telemetry logic in this layer, below is the appResized method that’s called on window.onresize events; as you can see, I gather up information about the device characteristics, because I want to know if the customer is playing full screen in portrait or landscape, how large of a screen they’re on, and the size of the app view itself. This data can inform future investments in layout and user experience. For example, if I knowthat customers prefer portrait over landscape, I can focus on that layout first and landscape second:

appResized: function () {
if (Telemetry.session === null || !Telemetry.enabled) { return; }

var wgd = Windows.Graphics.Display; 
var displayInfo = wgd.DisplayInformation.getForCurrentView();
var w = document.body.clientWidth;
var h = document.body.clientHeight;

//Convert orientation enum into a simple string value
var orientation = "landscape";

if (displayInfo.currentOrientation == wgd.DisplayOrientations.portrait
        || displayInfo.currentOrientation == wgd.DisplayOrientations.portraitFlipped) {
        orientation = "portrait";
    }

//Here I group window sizes at a 200px granularity for better analytics.
var data = {
"Display Orientation": orientation,
"Scale": displayInfo.resolutionScale,
"View Orientation": (w < h) ? "portrait" : "landscape",
"Width": Telemetry._convertToRange(w, 200),
"Height": Telemetry._convertToRange(h, 200)
    };

    Telemetry.session.tagEvent("View resized", data);
},

With my gameCompleted event, I also report what kind of input was used to play the game, bucketing the specific counts, of course. This is data that I want to drive further investments in handling input, such as my implementation of keyboard support:

gameCompleted: function (data) {
if (Telemetry.session === null || !Telemetry.enabled) { return; }

    data = data || {};
    Telemetry._timeLastCompleted = Telemetry._now();

//Add a range for inputs used in this game.
    data["InputTally_Touch"] = 
        Telemetry._convertToRange(Telemetry._inputTally[Telemetry.inputType.touch], 25);
    data["InputTally_Mouse"] = 
        Telemetry._convertToRange(Telemetry._inputTally[Telemetry.inputType.mouse], 25);
    data["InputTally_Keyboard"] = 
        Telemetry._convertToRange(Telemetry._inputTally[Telemetry.inputType.keyboard], 25);

//Add an attribute for network connectivity: I don’t use this in the game at present,
//but the data can help me understand whether to add network-related features.
var connected = false;
//Returns null if offline
var profile = Windows.Networking.Connectivity.NetworkInformation.getInternetConnectionProfile();

if (profile != null) {
var level = profile.getNetworkConnectivityLevel();

//internetAccess or constrainedInternetAccess is probably acceptable; none or limitedAccess is not.
                connected = (level == Windows.Networking.Connectivity.NetworkConnectivityLevel.internetAccess) ||
                    (level == Windows.Networking.Connectivity.NetworkConnectivityLevel.constrainedInternetAccess);
            }

            data["Has Connectivity"] = connected;
    Telemetry.session.tagEvent("Game completed", data);

},

Closing: telemetry as an investment

Collecting telemetry and watching the analytics that get generated from them is certainly important for improving an app and addressing customer needs. Beyond that, every app that you instrument for telemetry becomes a long-term investment for all of your app building efforts: your learnings from each app you create will greatly improve the quality of every subsequent app. You’ll have an increasingly clear understanding of what users expect and what engages them best, and will be able to apply that understanding in each of your designs. Indeed, one reason why you might choose to release one or more free apps, and focus on building up a large user base for each one, is to acquire telemetry data that will then inform a more significant investment in an app you’d like to monetize.

Like I said early on, all the data you gather and what you learn through analytics is truly a gold mine, and the efforts you make today to instrument your app for this purpose will likely pay dividends well into the future.

Kraig Brockschmidt, Senior Program Manager, Windows Ecosystem & Frameworks Team