See What’s New with Windows Ink in the Windows 10 Creators Update

Windows Ink is about transforming the way we think about computers, from a tool that is great at getting things done, to one that harnesses your personality and your emotions into the things you create. It’s about bringing back the human aspects that a mouse and keyboard (and even touch) cannot express fully, it’s about making personal computers more personal, and an extension of yourself, not just a tool. We want you to feel empowered to create from the moment you pick up the pen, and have the confidence that Windows understands you, knows what you want to do – by understanding your handwriting, your words and your expression. This is the journey we’re on.

With the Creators Update, Windows Ink is now better than ever! When used with the Surface Dial, it allows you to discover new ways to work and interact with Windows. With Windows Ink, we continue to make it possible for you to do more than with pen and paper. Applications like Photos and Maps have added incredible inking functionality in the last year, and continue to evolve and expand. With Paint 3D in the Creators Update, Windows Ink can now create 3D objects! As we evolve what Ink means to users, we’re also introducing new Smart Ink capabilities to Windows Ink. These capabilities allows developers to understand the ink that is being laid down by the user, using AI to help create, connect and complete user actions on ink. We’ve also improved and added features to the building blocks for Windows Ink, introducing new stencils and adding tilt support to create a richer drawing experience.

Devices that support the Pen on Windows have also doubled in the last year, and is on track to double again in the next year! We’re seeing high demand not just for devices, but also for applications that support ink. To make it easier to find compatible pens, Wacom has partnered with us to develop the Bamboo Ink Pen. This pen will be in market in summer and supports almost all Windows PCs that are pen-capable. It features the Microsoft Pen Protocol (MPP), which is based on Surface Pen technology. In addition, we are also excited that the Surface Dial is now available in more countries, like Australia, Canada and New Zealand, giving more people an opportunity to try this incredible new input device. In addition, new hardware from our OEM partners, like the Dell Canvas 27, are shipping soon and takes advantage of the same RadialController APIs that are used for the dial. As a developer building for the Surface Dial today, it means that you are ready for all the new hardware that our OEM partners will bring to the ecosystem.

The progress we’ve made with Windows Ink would not have been possible without the feedback and passion you developers bring to us. With over a thousand inking applications in the store and growing everyday, with well over half of the top 10 paid store apps being ink apps, there is incredible enthusiasm and interest in this space. This is an incredible opportunity that you have embraced with us, and it inspires us to do more in each Windows release.

What’s new with Windows Ink platform?

Ink is the ultimate way humans can express themselves, it opens up new opportunities for application developers to differentiate, and helps make their applications stand out. From the latest fads like adult coloring books to simple games like tic-tac-toe, to applications that help you organize your life, there is just so much opportunity to build the next big thing in the inking space. We also know that people who use Windows Ink are more satisfied with their experience, what they look for, and buy more inking applications. From the platform perspective, we have 2 ways that we help developers:

• Make it as easy and quick for a developer to add inking into their application by providing controls that can be dropped in quickly into any application and get Windows Ink support.
• Provide the most flexible platform building blocks for developers to innovate upon. This gives you the flexibility to choose where to start developing for Windows Ink.

Introducing Smart Ink

Let’s start with a new building block that developers have access to in the Creators Update. Introducing Ink Analysis, this is the first of our family of Smart Ink capabilities that we are bringing to the platform. Smart Ink brings AI technology to not just understand what you write, but also helps connect the dots to what you may want to do. With Ink Analysis, it starts simple, with recognizing shapes and making that square you drew more perfect, but it can also do much more, like understanding you wrote words in squares and making it into an org chart using understanding about your organization. Our goal is to understand user intent and empower developers to turn it into rich digital constructs, as well as to leverage understanding from all parts of the system. Ink Analysis allows any developer to understand the ink they capture, whether it is handwriting, shapes, phone numbers, stock symbols, lists, document structure and more.  This is the same technology we debuted in Sticky Notes in the Window 10 Anniversary Update, and now it’s available for you to use! We can’t wait to see what you can do with this technology.

Here is an example of how to use Ink Analysis to recognize shapes.  For this snippet, we’ll use DirectInk to handle rendering the ink strokes.  Start by initializing an InkAnalyzer and connecting it with InkPresenter:

[code lang=”csharp”]

private void Initialize()
{
inkAnalyzer = new InkAnalyzer();
inkCanvas.InkPresenter.StrokesCollected += InkPresenter_StrokesCollected;
inkCanvas.InkPresenter.StrokesErased += InkPresenter_StrokesErased;
}

// Whenever the user draws a new stroke, you copy the stroke into Ink Analyzer’s stroke collection
private void InkPresenter_StrokesCollected(InkPresenter sender, InkStrokesCollectedEventArgs args)
{
inkAnalyzer.AddDataForStrokes(args.Strokes);
}

// When a stroke is erased in InkCanvas, remove the same stroke from Ink Analyzer’s collection.
private void InkPresenter_StrokesErased(InkPresenter sender, InkStrokesErasedEventArgs args)
{
foreach (var stroke in args.Strokes)
{
inkAnalyzer.RemoveDataForStroke(stroke.Id);
}
}

[/code]

Next you want to feed strokes to the analyzer. Commonly this is done via explicit user action (e.g. the user clicks a button) or after the user has been idle for a while.

[code lang=”csharp”]

inkAnalyzer.AnalyzeAsync();

[/code]

The result is a tree representation of the whole document with different kinds of nodes, such as paragraph, line, list, word, and drawing. If for instance you want to find all the shapes in the ink, you can with the code below:

[code lang=”csharp”]

IReadOnlyList<IInkAnalysisNode> drawings = inkAnalyzer.AnalysisRoot.FindNodes(InkAnalysisNodeKind.InkDrawing);
foreach (IInkAnalysisNode drawing in drawings)
{
var shape = (InkAnalysisInkDrawing)drawing;
switch (shape.DrawingKind)
{
case InkAnalysisDrawingKind.Circle:
// The user drew a circle. You can replace it with a perfect circle that goes through shape.Points.
break;
case InkAnalysisDrawingKind.Rectangle:
// The user drew a rectangle.
// You can find the full list of supported shapes here.
break;
}
}

[/code]

If you want to learn more about Ink Analysis, you can watch the BUILD 2017 recorded video Enable Natural Pen Interaction by Using Ink Analysis to Better Understand Users’ Ink, download the Ink Analysis sample on GitHub or check out the Ink Analysis API Reference.

An improved Ink Toolbar

In the Anniversary Update we created a customizable set of inking tools, Ink Toolbar and Ink Canvas, that any developer can add to their own application with only two lines of markup.

[code lang=”xml”]

<InkCanvas x:Name=“myInkCanvas”/>
<InkToolbar TargetInkCanvas=“{x:Bind myInkCanvas}”/>

[/code]

Many of Microsoft’s first party applications have incorporated the inking tools to create engaging user experiences. For example, Photos added a calligraphy pen and the ability to draw on any photo in the gallery. Maps added a feature that lets you measure the distance of a route drawn on the map. Edge browser added inking on webpages. It has never been easier to add Windows Ink to your applications.

In the Creators Update, we continue our commitment to improving these controls! If you already use them in your applications, these improvements will benefit you with no additional work!

In response to users, the Creators Update introduces a new stencil, the protractor. This new stencil makes it easy for you to draw circles and arcs of any size. When drawing an arc, the protractor displays a readout that tells you the precise angle of the arc. You can also resize the stencil with just a pinch/zoom gesture with your fingers.

We’ve also made the ruler stencil better! Like the protractor, it now provides an angle readout that shows the ruler’s angle with the horizontal line. The ruler also snaps to 0, 45 and 90 degrees for easy access to the most common angles being used by our users.

You asked for an improve stroke preview in the Ink Toolbar, and in the Creators Update, we have it! We’re also make changes in the Ink Toolbar to work better with High Contrast themes, by automatically showing only colors that meet visibility requirements for the current user profile.

New Exciting Inking Capabilities

Today we announced the new Surface Pro and the new Surface Pen. Together they enable the next generation of inking capabilities that truly make writing digitally as natural as pen on paper. Here are some of the highlights:

• Low latency Ink that virtually eliminates lag when you write
• Tilt support to capture an additional dimension in digital inking
• Ink that captures the entire spectrum of your expression with 4,096 levels of pressure sensitivity
• Effortless inking with half the activation force required to being inking

Our customers have asked us for these capabilities, and they are finally here! From a developer perspective, if you already use the Windows Ink platform, all these capabilities show up in your application automatically! There are no changes required, and you are ready for the new Surface Pro, with the new Surface Pen.

Low latency Inking is a unique addition to Windows Ink. It is the result of a close partnership between hardware and software. The Pixelsense Accelerator chip in the new Surface Pro, is the first device to run Windows Ink acceleration code natively on hardware. This is how we achieve a new milestone in inking, virtually eliminating lag between the pen tip and the ink that flows out of it, creating the most natural writing experience with Windows Ink.

Tilt is another great addition to the Inking experience. The great news is, in addition to the new Surface Pro/Pen supporting this new capability, Wacom Pens that feature tilt will also “just work”! Tilt allows Windows Ink to model natural pencil sketching that response to the tilt of the pen. This support is now built into the pencil brush on the Ink Toolbar. In the above diagrams, we demonstrate how the pencil brush can be used to shade lines (on the left) and to draw arcs of varying thickness depending on the degree of tilt (on the right).

As mentioned above, tilt integration happens automatically if you use the Ink Toolbar. However, if you are not using the Windows Ink platform to render ink, and want to build your own brush that responds to tilt, you still can! There are two properties, TiltX and TiltY (respective angle of tilt against each axis of the screen plane) which are included with pointer input messages. You can access the tilt values from the PointerPointProperties included with Pointer input events, or the POINTER_PEN_INFO struct from WM_POINTER input.

These improvements automatically show up on any application that uses the Windows Ink controls, and you can be confident that we’ll continue to evolve and improve them in each release of Windows.

What’s new with Surface Dial and RadialController?

The Surface Dial introduces a new input paradigm to computing. It was designed alongside the Windows Ink experience, allowing it to truly shine when used together with a Pen. We’ve seen many experiences built to harness the new capabilities the Surface Dial brings, and are also seeing new hardware emerging, and adopting the RadialController standard. In response to your feedback, we’ve added more capabilities to the RadialController experience in the Creators Update.

First off, are some new button events for RadialControllers. These new events, Pressed and Released, combined with existing events for rotation and screen contact, will allow you to track complex interactions such as press-and-rotate or press-and-move. The example below illustrates a simple way to capture a press-and-rotate action.

[code lang=”csharp”]

_radialController.ButtonPressed += OnButtonPressed;
_radialController.ButtonReleased += OnButtonReleased;

private void OnRotationChanged(RadialController sender,
RadialControllerRotationChangedEventArgs args)
{
if (args.IsButtonPressed)
{
/* When button is pressed, you can do modal interactions, fine-grained changes */
}
else
{
/* Otherwise, do the normal rotation behavior */
}
}
private void SendHaptics(SimpleHapticsController hapticController)
{
var feedbacks = hapticController.SupportedFeedback;
foreach (SimpleHapticsControllerFeedback feedback in feedbacks)
{
if (feedback.Waveform ==
KnownSimpleHapticsControllerWaveforms.Click)
{
hapticController.SendHapticFeedback(feedback);
return;
}
}
}

[/code]

You also now have access to the Haptics engine in the Surface Dial hardware. Using SimpleHapticsController—a new object that uses the HID Simple Haptics specification—you have the power to directly send feedback to the user. You can use this to customize the feel of your menu, adding a new dimension to the experience. This object is available in the arguments of all radial controller input events.

In cases where you may want to suppress the radial menu to prevent it from blocking UI, we now have new properties ActiveControllerWhenMenuIsSuppressed and IsMenuSuppressed to let you configure when the menu is available or suppressed. When a menu is suppressed, it will not appear on press-and-hold interactions for the foreground app. Your app can listen to a new event during menu suppression to give the user an indication the menu is blocked, or build an alternate experience. Here is a code sample for this functionality:

[code lang=”csharp”]

RadialControllerConfiguration config = RadialControllerConfiguration.GetForCurrentView();
config.ActiveControllerWhenMenuIsSuppressed = myController;
config.IsMenuSuppressed = true;

myController.ButtonHolding += MyController_ButtonHolding;

[/code]

User input running on a UI thread can sometimes lead to performance bottlenecks. With the Creator’s Update, radial controller interactions can now be handled on an off-UI thread using RadialControllerIndependentInputSource. Below is an example on how to get additional performance using this method.

[code lang=”csharp”]

RadialController controller;
Windows.UI.Input.Core.RadialControllerIndependentInputSource independentInput;
CoreApplicationView view;

view = CoreApplication.GetCurrentView();

var workItemHandler = new WorkItemHandler((IAsyncAction) =>
{
independentInput = Windows.UI.Input.Core.RadialControllerIndependentInputSource.CreateForView(view);

controller = independentInput.Controller;

controller.RotationResolutionInDegrees = 5;

controller.RotationChanged += Controller_RotationChanged;
controller.ScreenContactStarted += Controller_ScreenContactStarted;
controller.ScreenContactContinued += Controller_ScreenContactContinued;
controller.ScreenContactEnded += Controller_ScreenContactEnded;
controller.ControlLost += Controller_ControlLost;
controller.ButtonClicked += Controller_ButtonClicked;
controller.ButtonPressed += Controller_ButtonPressed;
controller.ButtonReleased += Controller_ButtonReleased;
controller.ButtonHolding += Controller_ButtonHolding;
controller.ControlAcquired += Controller_ControlAcquired;

// Begin processing input messages as they’re delivered.
independentInput.Dispatcher.ProcessEvents(CoreProcessEventsOption.ProcessUntilQuit);
});
action = ThreadPool.RunAsync(workItemHandler, WorkItemPriority.High, WorkItemOptions.TimeSliced);

[/code]

In addition to all the API additions above, you can now customize and easily add new menu items on the Radial Menu. Under “Wheel Settings” in the settings app, you can add application specific menu items that trigger keyboard combinations. Imagine customizing the controller to send your favorite shortcuts in Visual Studio, Photoshop or even when browsing the web!

The Surface Dial continues to excite users and developers alike, with these new enhancements, both developers and users have more control and flexibility in their experience. We invite you to join the numerous applications that have already delivered a great Surface Dial experience, like CorelDRAW, Autodesk’s SketchBook, Silicon Bender’s Sketchable and Algoriddim’s djay Pro. We can’t wait to see what you can do with this unique new form of input on Windows.

Join us in making Windows Ink better!

With Windows Ink and the Surface Dial additions in the Creators Update, we believe we’re just scratching the surface of what Windows Ink can do in people’s lives. Our commitment is to invest in areas that can help you innovate and remove all the barriers to our users using, loving and needing Windows Ink. This involves a spectrum of efforts, from the hardware we build by ourselves and with our partners, to the next SDK additions we make to power you app. As we continue this journey, we invite you to lend us your voice, your ideas and your feedback. Help us help you make the next great application and help us help you change the world. Tweet your ideas using #WindowsInk, email us at [email protected] or tweet us at @WindowsInk. We would love to hear from all of you.

Thank you!