Why cover media functionality?
By adding simple JavaScript code, media web apps can easily take advantage of Windows integrations in order to provide a richer experience to users than what is offered through the browser.
Hooking up the System Media Transport Controls
System Media Transport Controls (SMTC) are the common play, pause, back, and forward buttons exposed through Windows 10. When an app hooks into these controls, it will receive the events following user interaction. It will have the ability to handle them appropriately allowing users to accomplish actions such as skip to next track or pause the current video all from the OS.
Whether the app is using the <audio> or <video> HTML element, the controls attribute needs to be added as follows:
| <video id="mediaVideo" src="img/Westminster_high.mp4" controls></video> |
To handle the events from the System Media Transport , developers need to instantiate the controls by calling the proper Windows API in JavaScript. By doing feature detection for the Windows namespace to determine that the code is running in a Windows app it allows the same code to run error free in the browser.
| if (typeof Windows !== 'undefined') { | |
| var systemMediaControls = Windows.Media.SystemMediaTransportControls.getForCurrentView(); | |
| systemMediaControls.addEventListener("buttonpressed", systemMediaControlsButtonPressed, false); | |
| systemMediaControls.isPlayEnabled = true; | |
| systemMediaControls.isPauseEnabled = true; | |
| systemMediaControls.isStopEnabled = true; | |
| systemMediaControls.playbackStatus = Windows.Media.MediaPlaybackStatus.closed; | |
| } |
Once the DOM is loaded and the <audio> or <video> element is present, subscription to the pause, playing, and ended events is needed. Developers need to call the appropriate functions to update the state of the SMTC so it represents the media state change from the HTML element.
| document.addEventListener("DOMContentLoaded", function () { | |
| var videoElement = document.getElementById("mediaVideo"); | |
| videoElement.addEventListener("pause", mediaPaused); | |
| videoElement.addEventListener("playing", mediaPlaying); | |
| videoElement.addEventListener("ended", mediaEnded); | |
| }); |
Here are the definitions of the functions that need to be included in order to ensure correct SMTC state is reflected:
| function mediaPlaying() { | |
| // Update the SystemMediaTransportControl state. | |
| systemMediaControls.playbackStatus = Windows.Media.MediaPlaybackStatus.playing; | |
| } | |
| function mediaPaused() { | |
| // Update the SystemMediaTransportControl state. | |
| systemMediaControls.playbackStatus = Windows.Media.MediaPlaybackStatus.paused; | |
| } | |
| function mediaEnded() { | |
| // Update the SystemMediaTransportControl state. | |
| systemMediaControls.playbackStatus = Windows.Media.MediaPlaybackStatus.stopped; | |
| } |
Now, moving on to handling user interaction from the SMTC, developers need to ensure that the events raised by the SMTC when a user clicks or touches a button is reflected in the playing media.
Here are the definitions of functions that are invoked in the sample once an event from the SMTC is fired in the app. These need to be implemented in order for the user-intended state change to be applied to the HTML element.
| function playMedia() { | |
| var media = document.getElementById("mediaVideo"); | |
| media.play(); | |
| } | |
| function pauseMedia() { | |
| var media = document.getElementById("mediaVideo"); | |
| media.pause(); | |
| } | |
| function stopMedia() { | |
| var media = document.getElementById("mediaVideo"); | |
| media.pause(); | |
| media.currentTime = 0; | |
| } |
By adding less than 100 lines of JavaScript code, a web app is able to cleanly integrate with OS functionality and give users more control through existing physical and UI style buttons.
For detailed information, check out this MSDN article on the SystemMediaTransportControls class.
Background Audio
No user wants their music to cut out when an app is minimized. By following a few easy steps, this can be avoided altogether on Windows desktop devices.
- Integrate with SMTC (covered above)
- Add the htmlmsAudioCategory=”BackgroundCapableMedia” to the html element
| <video id="mediaVideo" src="img/Westminster_high.mp4" msAudioCategory="BackgroundCapableMedia" controls></video> |
- Add a background task in the manifest. This can be done through the manifest editor:
Or directly in XML:
| <Extensions> | |
| <Extension Category="windows.backgroundTasks" StartPage="index.html"> | |
| <BackgroundTasks> | |
| <Task Type="audio" /> | |
| </BackgroundTasks | |
| </Extension> | |
| </Extensions> |
On Windows mobile devices, the way to play audio in the background is through a separate, in-package background task. The background task has to be local, but it can be written in JavaScript. This allows the platform to keep the background task alive in a separate process from that of the app when the app is in the background. When the app enters in the background, it’s only the background task code that’s able to execute.
The sample shows how to enable background audio on a phone running Windows 10.
For more detailed information on how background audio functions on Windows, be sure to check out the Basics of Background Audio.
Going full screen
Full screen is a very common scenario for apps that support video playback. This functions differently in the browser versus in an app. In an app, developers have control over whether the video should take up the entire app window or the entire monitor when full screen is called. In order to take up the entire screen, developers must listen for the fullscreenchange DOM event. Once this event fires, a corresponding call to the WinRT API that puts the app in full screen needs to be made.
| document.addEventListener("DOMContentLoaded", function () { | |
| var videoElement = document.getElementById("mediaVideo"); | |
| videoElement.addEventListener("msfullscreenchange", fullScreen); | |
| videoElement.addEventListener("webkitfullscreenchange", fullScreen); | |
| videoElement.addEventListener("fullscreenchange", fullScreen); | |
| videoElement.addEventListener("mozfullscreenchange", fullScreen); | |
| }); |
This screen change handling logic calls the correct WinRT based on the fullscreenchange event:
| function fullScreen() { | |
| console.log("fullscreenchange"); | |
| if (typeof Windows !== 'undefined') { | |
| var view = Windows.UI.ViewManagement.ApplicationView.getForCurrentView(); | |
| if(view.IsFullScreenMode) { | |
| console.log("Exiting WinRT Fullscreen"); | |
| view.ExitFullScreenMode(); | |
| } | |
| else { | |
| console.log("Entering WinRT Fullscreen"); | |
| view.TryEnterFullScreenMode(); | |
| } | |
| }; | |
| } |
Listening for voice commands
Simple Cortana integration is easy to add as well. In this sample we show how a user can play a specific video through a voice command.
A Voice Command Definition (VCD) file needs to be added to the server so it can be accessed publicly. This VCD file is used to describe the commands that will be registered with the Windows Speech Platform and invoked when a user utters them to Cortana. Here’s the VCD file used in the sample:
| Platform and invoked when a user utters them to Cortana. Here’s the VCD file used in the sample: | |
| <?xml version="1.0" encoding="utf-8"?> | |
| <VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.2"> | |
| <CommandSet xml:lang="en-us" Name="MediaSample"> | |
| <CommandPrefix>Contoso Video</CommandPrefix> | |
| <Example>Contoso video play elephants</Example> | |
| <Command Name="play"> | |
| <Example>play {message} using media sample</Example> | |
| <ListenFor RequireAppName="BeforeOrAfterPhrase">please play {streamSubject}</ListenFor> | |
| <Feedback>playing {streamSubject} with Stream Demo</Feedback> | |
| <Navigate Target="/playStream.htm"/> | |
| </Command> | |
| <PhraseTopic Label="streamSubject" Scenario="Dictation"></PhraseTopic> | |
| </CommandSet> | |
| </VoiceCommands> |
Adding the meta tag in the HTML page that’s set as the start page for the hosted web app will ensure the VCD file is downloaded and properly registered by the platform. This will be automatically handled when the web app is running in the app host with the meta tag present. The meta tag must include the location of the VCD file in the content attribute.
| <meta name="msapplication-cortanavcd" content="http://seksenov.github.io/HWAsmtc/vcd.xml" /> |
Once the VCD file is created, added to the server, and the meta tag is included in HTML, Cortana voice command activations need to be handled in JavaScript. Developers should always be feature detecting for the Windows namespace to avoid errors when the web app is running in the browser.
| if (typeof Windows !== 'undefined') { | |
| // Subscribe to the Windows Activation Event | |
| Windows.UI.WebUI.WebUIApplication.addEventListener("activated", function (args) { | |
| var activation = Windows.ApplicationModel.Activation; | |
| // Check to see if the app was activated by a voice command | |
| if (args.kind === activation.ActivationKind.voiceCommand) { | |
| var speechRecognitionResult = args.result; | |
| var textSpoken = speechRecognitionResult.text; | |
| // Determine the command type {play} defined in vcd | |
| if (speechRecognitionResult.rulePath[0] === "play") { | |
| // Determine the stream name specified | |
| if (textSpoken.includes('elephants') || textSpoken.includes('Elephants')) { | |
| setupVideo("http://amssamples.streaming.mediaservices.windows.net/b6822ec8-5c2b-4ae0-a851-fd46a78294e9/ElephantsDream.ism/manifest(filtername=FirstFilter)"); | |
| } | |
| else if (textSpoken.includes('Steel') || textSpoken.includes('steel')) { | |
| setupVideo("http://amssamples.streaming.mediaservices.windows.net/bc57e088-27ec-44e0-ac20-a85ccbcd50da/TearsOfSteel.ism/manifest"); | |
| } | |
| } | |
| }; | |
| }); | |
| } |
By creating a Voice Command Definition File, adding the meta element in HTML, and event handling in JavaScript, the app is able to light up voice capabilities and add another dimension of user interaction. Not an overwhelming amount of code in order to integrate with Cortana.
Adding theme
A nice touch is to match the app title bar to the color or color theme of the app. The sample accomplishes this through a few lines of JavaScript code, enhancing the appearance of the app.
Here is the code that needs to be added:
| function setAppBarColors(brandColorHex, brandColorInactiveHex) { | |
| // Detect if the Windows namespace exists in the global object | |
| if (typeof Windows !== 'undefined') { | |
| var brandColor = hexStrToRGBA(brandColorHex); | |
| var brandColorInactive = hexStrToRGBA(brandColorInactiveHex); | |
| // Get a reference to the App Title Bar | |
| var appTitleBar = Windows.UI.ViewManagement.ApplicationView.getForCurrentView().titleBar; | |
| var black = hexStrToRGBA('#000'); | |
| var white = hexStrToRGBA('#FFF'); | |
| appTitleBar.foregroundColor = white; | |
| appTitleBar.backgroundColor = brandColor; | |
| appTitleBar.buttonForegroundColor = white; | |
| appTitleBar.buttonBackgroundColor = brandColor; | |
| appTitleBar.buttonHoverForegroundColor = white; | |
| appTitleBar.buttonHoverBackgroundColor = brandColor; | |
| appTitleBar.buttonPressedForegroundColor = brandColor; | |
| appTitleBar.buttonPressedBackgroundColor = white; | |
| appTitleBar.inactiveBackgroundColor = brandColorInactive; | |
| appTitleBar.inactiveForegroundColor = brandColor; | |
| appTitleBar.buttonInactiveForegroundColor = brandColor; | |
| appTitleBar.buttonInactiveBackgroundColor = brandColorInactive; | |
| appTitleBar.buttonInactiveHoverForegroundColor = brandColor; | |
| appTitleBar.buttonInactiveHoverBackgroundColor = brandColorInactive; | |
| appTitleBar.buttonPressedForegroundColor = brandColor; | |
| appTitleBar.buttonPressedBackgroundColor = brandColorInactive; | |
| } | |
| } |
Note: The code snippet shown above uses RGBA for the color values. Developers need to convert from Hex to RGBA if those are the color value types being used. A helper function can be found here.
For detailed information about the properties exposed through the titleBar, check out this MSDN article on the titleBar property.
Wrapping up
Adding these five easy Windows integrations to a media web app can make it more appealing to customers. There are many more platform features to integrate with so stay tuned and check back for more posts in the Hosted Web Apps series. The best part is, after the app is published to the Windows Store, developers can stick to their existing web workflow should they choose to add new integrations. Pushing new code updates will update the app without the need to re-submit to the Store.
Written by Kiril Seksenov, Engineer on the Web Platform Team
Additional resources
- Full Media Sample
- Hosted Web App Docs
- Hosted Web App Initial Post
- Project Westminster in a Nutshell
- Building and Publishing a Web App:
And don’t forget to check out the other posts in our series: