April 20, 2016 10:00 am

Adding Cortana and Bluetooth to a UWP app for Phillips Hue

Smart devices are becoming increasingly popular. One great device is the Phillips Hue lightbulb – a colorful, internet-connected light you can control with your phone from the comfort of your couch. No longer do you have to get up to toggle a light switch. There are already a number of Windows apps that you can download to interact with the Hue lights, but we wanted to create a sample that demonstrated how a Universal Windows Platform (UWP) app could interact with them in unique ways.

The full source for our sample is available on GitHub here. Feel free to take a look and, if you have some Hue lights, play around.

The basics of the app are simple: find the bridge on your network and talk to it using Hue’s RESTful interface. We created a small library to simplify this part, which lets you do cool things like turn on all your lights with just a few lines of code:



Bridge bridge = await Bridge.FindAsync();
IEnumerable<Light> lights = await bridge.GetLightsAsync(); 
foreach (Light light in lights)
{
	light.State.On = true; 
}


That’s the easy part. With that foundation, we built up the app to explore a few other key scenarios. For this release, we focused on:

  • Cortana
  • Bluetooth LE
  • Extended splash screen

In this post, we’ll talk about the basic steps we took to integrate these three features into our app sample, and point you towards relevant code in the sample that shows how to integrate them into your own app. To see Cortana and the Hue sample in action, check out this video on Channel 9:

Cortana

Cortana provides a way for users to naturally interact with their PCs and phones. They accomplish key tasks such as launching apps, searching the web, and getting a joke of the day. Yet that’s only part of the story. The true value of Cortana is seen when you extend the basic functionality to integrate with your UWP apps. By adding Cortana support to an app, you can create really cool experiences for your users – such as turning on Hue lights or changing their color just with voice.

The first step to integrate Cortana with your app is to create a voice command definition (VCD) file. This file defines the structure of voice commands that Cortana should recognize as related to your app. For the Hue sample, these are commands like changing the color of a specific light or turning a light on or off.



    <Command Name="changeLightsState">
      <Example>Turn the lights on</Example>
      <ListenFor>[Turn] [the] lights {state}</ListenFor>
      <Feedback>Trying to turn the lights {state} </Feedback>
      <VoiceCommandService Target="LightControllerVoiceCommandService" />
    </Command>

    <Command Name="changeLightsColor">
      <Example>Change the lights color</Example>
      <ListenFor>Change [the] [lights] color</ListenFor>
      <VoiceCommandService Target="LightControllerVoiceCommandService" />
    </Command>


You’ll notice that a command specifies a name as well as the background task that should be used when it’s detected. That allows Cortana to route the command to your app and for you to handle it appropriately. You can learn more about the XML schema on MSDN in the article Voice Command Definition (VCD) elements and attributes and see our full VCD file here on GitHub,  but at a high level the schema is made up of VoiceCommands and PhraseLists.

If you look at our full VCD file for the Hue sample, you’ll notice that while all the commands you’d expect are listed, the PhraseLists aren’t quite complete. The “name” list, for example, is empty and no colors are specified. That’s because we wanted to dynamically create those lists, ensuring that the names displayed in our UI matched the names of the lights enumerated on the network and that the full set of system colors was available for specification. To accomplish that, while our extended splash screen was showing, we dynamically modified the VCD file before registering it with the OS as seen in the InitalizeCortanaAsync() method below.



/// <summary>
/// Prepares Cortana for background use. 
/// </summary>
private async Task InitalizeCortanaAsync()
{
	// You can't write to application files by default, so we need to create a 
	// secondary VCD file to dynamically write Cortana commands to.
	StorageFile dynamicFile = await ApplicationData.Current.RoamingFolder.CreateFileAsync(
		"VoiceCommands.xml", CreationCollisionOption.ReplaceExisting);

	// Load the base file and parse the PhraseList we want from it.
	StorageFile baseFile = await StorageFile.GetFileFromApplicationUriAsync(
		new Uri("ms-appx:///VoiceCommands.xml"));
	XDocument xml = XDocument.Load(baseFile.Path);
	XElement state = xml.Descendants().First(x => x.Name.LocalName == "PhraseList" && null != x.Attribute("Label") && x.Attribute("Label").Value == "state");

	// A ColorMapping is a RGB and HSV compliant representation a system color.
	// ColorMapping.CreateAll() returns a ColorMapping for all system colors available to UWP apps.
	// For each ColorMapping, add it to the list of phrases Cortana knows.
	foreach (HsbColor color in HsbColor.CreateAll())
	{
		state.Add(new XElement("Item", color.Name));
	}

	// Add the light names.
	XElement names = xml.Descendants().First(x => x.Name.LocalName == "PhraseList" && null != x.Attribute("Label") && x.Attribute("Label").Value == "name");
	foreach (Light light in _lights)
	{
		names.Add(new XElement("Item", light.Name));
	}

	// Save the file, and then load so Cortana recognizes it.
	using (Stream stream = await dynamicFile.OpenStreamForWriteAsync())
	{
		xml.Save(stream);
	}
	Await VoiceCommandDefinitionManager.InstallCommandDefinitionsFromStorageFileAsync(dynamicFile);
}


Now that the commands and phrases are defined and registered, all you need to do to finish integrating Cortana support is to create a background task to handle the commands. There are some complexities here if you want to handle the disambiguation of responses or query the user for additional information, but it’s all generally straightforward – parse the user’s speech, then send the appropriate command to the Hue lights.

The following code shows the Run method of the background task and how it determines the Cortana command that was spoken by the user.



        /// <summary>
        /// Entry point for the background task.
        /// </summary>
        public async void Run(IBackgroundTaskInstance taskInstance)
        {
            var triggerDetails = taskInstance.TriggerDetails as AppServiceTriggerDetails;
            if (null != triggerDetails && triggerDetails.Name == "LightControllerVoiceCommandService")
            {
                BackgroundTaskDeferral _deferral = taskInstance.GetDeferral();
                taskInstance.Canceled += (s, e) => _deferral.Complete();
                if (true != await InitalizeAsync(triggerDetails))
                {
                    return;
                }
                // These command phrases are coded in the VoiceCommands.xml file.
                switch (_voiceCommand.CommandName)
                {
                    case "changeLightsState": await ChangeLightStateAsync(); break;
                    case "changeLightsColor": await SelectColorAsync(); break;
                    case "changeLightStateByName": await ChangeSpecificLightStateAsync(); break;
                    default: await _voiceServiceConnection.RequestAppLaunchAsync(
                        CreateCortanaResponse("Launching HueLightController")); break;
                }
                // keep alive for 1 second to ensure all HTTP requests sent.
                await Task.Delay(1000);
                _deferral.Complete();
            }
        }


Notice that a BackgroundTaskDeferral is created, since our background task for handling Cortana commands runs asynchronous methods. Without the deferral, it is possible for the Run method to return before the background task has completed its work, which might result in the suspension or termination of the background task host process and prevent the completion of any asynchronous operations started by the background task. For more details about deferrals, see the IBackgroundTaskInstance.GetDeferral method documentation on MSDN.

The majority of the commands we implemented were rather simple, but we did choose to have one command that required Cortana to prompt the user for additional information. When a user asks Cortana to change the lights’ color, but didn’t specify one, we wanted Cortana to suggest some colors. That was accomplished through a request for disambiguation as shown in the following code.



        /// <summary>
        /// Handles an interaction with Cortana where the user selects 
        /// from randomly chosen colors to change the lights to.
        /// </summary>
        private async Task SelectColorAsync()
        {
            var userPrompt = new VoiceCommandUserMessage();
            userPrompt.DisplayMessage = userPrompt.SpokenMessage = 
                "Here's some colors you can choose from.";

            var userReprompt = new VoiceCommandUserMessage();
            userReprompt.DisplayMessage = userReprompt.SpokenMessage = 
                "Sorry, didn't catch that. What color would you like to use?";

            // Randomly select 6 colors for Cortana to show
            var random = new Random();
            var colorContentTiles = _colors.Select(x => new VoiceCommandContentTile
            {
                ContentTileType = VoiceCommandContentTileType.TitleOnly,
                Title = x.Value.Name
            }).OrderBy(x => random.Next()).Take(6);

            var colorResponse = VoiceCommandResponse.CreateResponseForPrompt(
                userPrompt, userReprompt, colorContentTiles);
            var disambiguationResult = await 
                _voiceServiceConnection.RequestDisambiguationAsync(colorResponse);
            if (null != disambiguationResult)
            {
                var selectedColor = disambiguationResult.SelectedItem.Title;
                foreach (Light light in _lights)
                {
                    await ExecutePhrase(light, selectedColor);
                    await Task.Delay(500);
                }
                var response = CreateCortanaResponse($"Turned your lights {selectedColor}.");
                await _voiceServiceConnection.ReportSuccessAsync(response);
            }
        }


And really that’s all there is to it. Those are the basics of implementing Cortana support in an app.

Bluetooth LE

Manipulating the lights with UI controls is a vast improvement over physical switches, but we also want to explore ways to control the lights using proximity – when the user comes within reasonable range of the lights with their phone, the lights should automatically turn on. When they leave, the lights turn off. To achieve this effect, we decided to use Bluetooth Low Energy (LE) because it’s power-friendly and can easily run in the background.

Bluetooth LE is based around publisher and watcher objects. A publisher constantly broadcasts signals for nearby listening watchers to receive. A watcher, on the other hand, listens for nearby publishers and fires events when it receives a Bluetooth LE advertisement so that the app can react accordingly. In our case, we presume there’s a device acting as a publisher in close proximity to the Hue bridge. The app (running on a phone in the user’s pocket) assumes the role of the watcher.

Seem interesting? To get a Bluetooth watcher set up, first you’ll need to register a background task that listens for a BluetoothLEAdvertisementWatcherTrigger.



private IBackgroundTaskRegistration _taskRegistration;
private BluetoothLEAdvertisementWatcherTrigger _trigger;
private const string _taskName = "HueBackgroundTask";
private async Task EnableWatcherAsync()
{
	_trigger = new BluetoothLEAdvertisementWatcherTrigger();

	BackgroundAccessStatus backgroundAccessStatus = 
		await BackgroundExecutionManager.RequestAccessAsync();
	var builder = new BackgroundTaskBuilder()
	{
		Name = _taskName,
		TaskEntryPoint = "BackgroundTasks.AdvertisementWatcherTask"
	};
	builder.SetTrigger(_trigger);
	builder.AddCondition(new SystemCondition(SystemConditionType.InternetAvailable));
	_taskRegistration = builder.Register();
}


You can configure this trigger to fire only when an advertisement with specific publisher and signal strength information is received; that way, you don’t have to worry about handling signals intended for other apps.




// Add manufacturer data.
var manufacturerData = new BluetoothLEManufacturerData();
manufacturerData.CompanyId = 0xFFFE;
DataWriter writer = new DataWriter();
writer.WriteUInt16(0x1234);
manufacturerData.Data = writer.DetachBuffer();
_trigger.AdvertisementFilter.Advertisement.ManufacturerData.Add(manufacturerData);

// Add signal strength filters and sampling interval.
_trigger.SignalStrengthFilter.InRangeThresholdInDBm = -65;
_trigger.SignalStrengthFilter.OutOfRangeThresholdInDBm = -70;
_trigger.SignalStrengthFilter.OutOfRangeTimeout = TimeSpan.FromSeconds(2);
_trigger.SignalStrengthFilter.SamplingInterval = TimeSpan.FromSeconds(1);


Next, you’ll need to add the background task to the app’s manifest file. As with other background processes, this step is required for UWP apps to execute code when the user doesn’t have the app on screen. For more information on registering background tasks in the manifest, see Create and register a background task.

The final step is to create the actual background task entry point. This code, which lives in a separate Windows Runtime Component within the solution, contains the method that fires when a Bluetooth LE beacon is received. In the Hue sample’s case, that entails checking whether the user is moving closer to the publisher (signal strength increasing, user is coming home – turn on the lights!) or leaving range (signal strength dropping off – turn off the lights).



private IBackgroundTaskInstance backgroundTaskInstance;
private BackgroundTaskDeferral _deferral;
private Bridge _bridge; 
private IEnumerable<Light> _lights;

public async void Run(IBackgroundTaskInstance taskInstance)
{
	backgroundTaskInstance = taskInstance;
	var details = taskInstance.TriggerDetails as BluetoothLEAdvertisementWatcherTriggerDetails;
	if (details != null)
	{
		_deferral = backgroundTaskInstance.GetDeferral();
		taskInstance.Canceled += (s, e) => _deferral.Complete();

		var localStorage = ApplicationData.Current.LocalSettings.Values;
		_bridge = new Bridge(localStorage["bridgeIp"].ToString(), localStorage["userId"].ToString()); 
		try
		{
			_lights = await _bridge.GetLightsAsync();
		}
		catch (Exception)
		{
			_deferral.Complete();
			return; 
		}
		foreach(var item in details.Advertisements)
		{
			Debug.WriteLine(item.RawSignalStrengthInDBm);
		}

		// -127 is a BTLE magic number that indicates out of range. If we hit this, 
		// turn off the lights. Send the command regardless if they are on/off
		// just to be safe, since it will only be sent once.
		if (details.Advertisements.Any(x => x.RawSignalStrengthInDBm == -127))
		{
			foreach (Light light in _lights)
			{
				light.State.On = false;
				await Task.Delay(250);
			}
		}
		// If there is no magic number, we are in range. Toggle any lights reporting
		// as off to on. Do not spam the command to lights already on. 
		else
		{
			foreach (Light light in _lights.Where(x => !x.State.On))
			{
				light.State.On = true;
				await Task.Delay(250);
			}
		}
		// Wait 1 second before exiting to ensure all HTTP requests have sent.
		await Task.Delay(1000);
		_deferral.Complete();
	}
}


Now, the next time the user fires up the app and enables the background watcher, it’s ready to start listening and can control the Hue lights accordingly.

Extended splash screen

Windows provides a default splash screen that works for most scenarios. However, if your app needs to perform long-running initialization tasks before it shows its main page (like searching for Hue lightbulbs on the network), users might get annoyed staring at a static image for too long (or worse, think that your app is frozen). To counteract this problem, developers can choose to show a customizable extended splash screen after the general app splash screen, where you can display anything from a basic progress wheel to intricate loading animations.

The Hue sample doesn’t need much extra time, so we chose something simple, but the framework is the same either way. When the app starts, it first displays the regular splash screen. After a few seconds though, instead of taking the user to a page with controls or text, it instead navigates to an intermediate empty page containing a copy of the default splash screen image overlaid with a spinning progress ring. This extended splash screen continues to display until all the initialization code is finished running, at which point the user is navigated to the main light controls. While this behavior is fairly simple, it makes the app seem more responsive.

Getting started with an extended splash screen in your app begins with the creation of a blank page containing only a XAML canvas to hold your splash image. For a smooth transition, we kept this image the same as the app’s standard splash image, but it doesn’t have to be.



    <Canvas Grid.Row="0" Grid.RowSpan="2" x:Name="SplashCanvas"  Background="White">
        <Image x:Name="extendedSplashImage" Source="Assets/splash.png"/>
    </Canvas>


Once you have your page, you need to modify the code-behind to prepare the splash screen and kick off your long-running initialization tasks (in our case, that meant finding the Hue bridge on the network, connecting to it, and then searching for lightbulbs). We also chose to include a bit of extra code to make sure the splash screen looks good on both phone and PC, and that it responds to device orientation.

To see our implementation, take a look on GitHub here.

The last step to hook things up is to modify the app’s App.xaml.cs file so that it navigates to the splash screen when the app starts instead of the MainPage.



        protected override void OnLaunched(LaunchActivatedEventArgs e)
        {
            var initializer = new Initializer(e.SplashScreen);
            Window.Current.Content = initializer;
            Window.Current.Activate();
        }


You can get quite fancy with extended splash screens – we only covered the basics – but even a simple extended splash screen provides a massively improved experience and helps keep impatient users happy.

Additional resources

If you found this app interesting, we’ve got similar app samples out there you might want to check out:

  • TrafficApp – A traffic monitor sample app demonstrating maps and location
  • RSSReader – An RSS aggregator demonstrating basic MVVM usage
  • QuizGame – A peer-to-peer trivia game demonstrating networking and discovery APIs
  • BluetoothAdvertisment – API sample demonstrating sending and receiving Bluetooth Low Energy advertisements
  • CortanaVoiceCommand – API sample showing how to integrate Cortana into an app

Written by Joshua Partlow, Alexander Koren, and Lauren Hughes from the Windows Developer Docs team

Updated April 20, 2016 1:29 pm