May 31, 2016 10:00 am

Building an IoT Magic Mirror with Hosted Web Apps and Windows 10

At Build 2016, we demoed a Magic Mirror project powered by a Hosted Web App on Windows 10 IoT Core. This project builds on the inspiration of projects like Michael Teeuw’s open-source Magic Mirror and Nathan Patrick’s Smart Mirror, enhancing the concept with a personalized experience that recognizes each user with facial recognition.

This demo illustrates how Hosted Web Apps in Windows 10 can leverage familiar web technologies to deliver powerful app experiences to all devices, including the Internet of Things. In this post, we’ll walk you through how we went about developing the mirror and how to build one for yourself!

What’s a Magic Mirror?

Our magic mirror is basically a one-way mirror (like you might have seen in Hollywood depictions of interrogation rooms), made “smart” by a simple LCD display which sits behind the mirror and displays white UI elements with a black background. When the display is on, you can see both your reflection and the white elements, allowing software to present relevant information while you get ready for the day.

Mockup showing the Magic Mirror installed in a bathroom.

We designed the mirror to adapt to each person and to work without getting in the way of their daily routine.

We designed the Magic Mirror to be low-cost and simple, so anyone could build it in a couple of hours. We’ve also open-sourced the web app and shared our bill of materials and assembly instructions on our GitHub repository.

To power the mirror, we chose a Raspberry Pi because of its popularity, price point, support, and hardware specs. Our web app, which provides the interface and basic functionality, is a simple Hosted Web App that runs on Windows 10 IoT Core.

Building the interface

We designed the mirror user interface (UI) to be as functional as possible, as both a mirror and an info hub. There are some practical implications to this.  The UI should be simple and easy to visually digest, so we kept adornment light and typography clear. The screen needs to be readable through the mirrored surface, so we used a high contrast ratio of pure white on pure black. Lastly and most importantly, the user needs to see their reflection, so we kept the central area of the mirror clear when the user is logged in.

Annotated screen capture showing the Magic Mirror interface, which is a simple white-on-black screen with relevant information like the current time, weather, and upcoming appointments around the periphery.

The mirror is built to be useful to a person getting ready in the morning. This person is likely on a time crunch, wants to be well-prepared for the day, and is interested in updates, but possibly doesn’t want to be barraged with info before they’re fully awake. With that in mind, we placed more-pressing information (weather, time, and a space reserved for alerts) at the top of the mirror near eye-level, and pushed less-urgent information down at the bottom, where it can be ignored or consciously consumed. Every user will have a slightly different idea about what’s most important, so this is a great project for exploring personalization through tech.

Building the web app

There are multiple pieces at play here. First is what you see displayed behind the mirror: This is a web app created in HTML, CSS, and JavaScript and served from a Node instance hosted on Azure.

Architecture diagram for the Magic Mirror web app, which is served from Azure and powered by the MIcrosoft Cognitive Services APIs. The app runs on a Raspberry Pi 3 on Windows 10 IoT Core.

Using the Hosted Web apps bridge, we turned our web app into a Universal Windows App, which not only give us access to Windows Native APIs but can also run across Windows devices, such as the Raspberry Pi 3 in our case. All the HTML, CSS, and JavaScript comes directly from the server, hence the term hosted.

Making it smart

The most important part of the app and the most delightful experience for the user is the facial recognition capability, which personalizes the mirror’s display based on the individual in front of it. In the past, this was complex technology out of the reach of most web apps, but, with APIs provided by Microsoft’s Cognitive Services, we’re able to build it into our mirror with minimal effort.

Magic Mirror leverages Microsoft’s Cognitive Services Face API to match the user’s face to their profile. The user creates a profile by adding some personal info and taking a selfie, which is then sent to Cognitive Services to get a unique identifier (a face_id) which is then stored in the Magic Mirror’s database.

Once they’ve created a profile, the user can stand in front of the magic mirror, which will take a picture and request Cognitive Services for the user’s face_id. This id is then used to find the user’s profile so the mirror can present the user with relevant info.

Below you can see how our Node instance sends an image as an octet-stream to Microsoft Cognitive Services through their REST API. The Cognitive Services’ cloud then sends back a face_id, which we bind to our user object.

We were very conscious about not wasting resources (CPU cycles, bandwidth, etc). For example, we didn’t want to send every frame to the Cognitive Services API, since most frames don’t have a person in them. To solve this, we used the facedetected event to send images to the Cognitive Services servers only when a face was detected. This event is available to the app since Hosted Web apps can access WinRT APIs through JavaScript.

In the code below, you can see the how we add the listener for the facedetected event once the stream is complete.

Try it out

This is just a small sample of what’s possible with the Hosted Web Apps platform and Cognitive Services APIs, but it’s a great introduction to how Hosted Web Apps on Windows 10 allow you to target the full range of Windows 10 devices, including the Internet of Things, to create compelling experiences with familiar web technologies. We’ve open sourced the application source code and bill of materials on GitHub – try it out for yourself and let us know what you think!

– Andy Pavia, Program Manager
– Stephanie Stimac, Program Manager
– Melanie Richards, Program Manager

Updated June 6, 2016 10:39 am

Join the conversation

  1. The assembly instructions say coming soon on the GitHub page. It doesn’t look like anyone has updated that page in a couple of months. Do you have a timeframe for when those will be available? Thanks. This is really cool.

  2. I also would like the assembly instructions. Rather purchase the Lifescan HD camera mentioned on GitHub, would a an Intel Realsense F200 camera work as well for future proofing?

  3. Could we use an Xbox Kinect with this? I think adding face recognition with a voice recognition would be nice. It might be to much for the PI to handle. My PI 3 is on it’s way. I will be playing with this and 10 IoT.

  4. Looking to replicated this application to demonstrate the possibilities of Hosted Web Applications but having a problem getting my head around the method for registering the ‘selfie’ profile. The on-line demo shows a configuration application to reference the selfie and the associated display capabilities for that profile but I cannot see how to run/access this. Can you point me in the right direction?

  5. photo here is obviously a fake vs. the github instructions which show a medicine cabinet with quite a bit of thickness and also a webcam visible out of the top of the cabinet.

    • The photo used above is a mockup of the concept (with the actual UI of the mirror superimposed) – the GitHub images represent a working, low-cost prototype you can build yourself!

  6. Hi Andy/Stephanie/Melanie,

    Thanks a lot for the awesome instructions and parts list that has links to where to get the parts. I’m not a developer, but I dabble. I’ve got the Magic Mirror up and running except the following:

    – Weather doesn’t reflect my city. It’s stuck at Friday Harbor city’s weather.
    – Stock quote not displaying.
    – Here’s the scoop is not showing the scoop.
    – Travel time is not showing the travel time.

    Everything else works fine. Could you please advise me on where to look to fix? Thanks!

  7. I have downloaded the code from GitHub. Followed the instructions as to how to install the apps but I seem to be stuck at Deploying on a PC.
    I created a new project, Javascript orientated. Then added the http://webreflections.azurewebsites.net/create -> ran the application in Visual Studio and it shows a page which says Set up your Magic Mirror. I fill up the details but get the following error
    “APPHOST9624: The application can’t use script to load the http://webreflections.azurewebsites.net/face/578f2c0de2ddea48398e1455 URL because the URL launches another application. Only direct user interaction can launch another application.”

    The hardware information has good detail. Can the same be done for the software? As I am no expert with js but more of a C# origin.

  8. Hi,

    I have managed to get the initial PC application deployed, but I’m stuck at the capturing a profile picture.

    I’m currently using a Windows 10 laptop with an inbuilt camera, but the camera pane window in the 2nd stage remains black, and I can’t get the application to detect my camera. I have also tried plugging in an old Logitech Webcam C160 to see whether that would work.

    Is there something I should change in the coding or could I possibly create the profile on the Pi using the Pi Camerav2 module which I’ve purchased.

    Thanks in advanced,
    Kevin

  9. Is there any chance you will be updating this project? I heard that Win IoT now supports Cortana. This would allow you to remove the camera and have an interactive mirror. This would be a very interesting update. Think “Mirror, Mirror on the Wall…”. 🙂

  10. Thanks for this, but I am stuck in the image capturing screen. Screen not capturing my web cam and it shows black screen. Can any only able to assist?