Today’s global cybersecurity threats are both dynamic and sophisticated, and new vulnerabilities are discovered almost every day. We focus on protecting customers from these security threats by providing security updates on a timely basis and with high quality. We strive to help you keep your Windows devices, regardless of which version of Windows they are running, up to date with the latest monthly quality updates to help mitigate the evolving threat landscape.
That is why, today, as part of our series of blogs on the Windows approach to quality, I’ll share an overview of how we deliver these critical updates on a massive scale as a key component of our ongoing Windows as a service effort.
Quality and security at scale
The scale and diversity of the Windows ecosystem requires us to take a data-driven approach to quality and to leverage automation for testing, validation and distribution. To provide the best protection, our customers’ devices need updating before vulnerabilities are publicly disclosed, a complex effort that requires a high degree of cross-industry cooperation. To put this into perspective, each month, we update over one billion devices ranging from desktop PCs and IoT devices to servers. This includes numerous combinations of Windows versions and editions from the most current versions of Windows 10 to end-of-support versions such as Windows XP on custom support agreements. During peak times, we update over 1,000 devices per second, allowing us to deliver the latest security and quality improvements to hundreds of millions of customers that protect them from potential vulnerabilities. Windows monthly updates also include quality and reliability fixes based on user feedback and data we monitor, to improve the user’s overall experience.
Types of monthly updates
Windows 10 quality updates are cumulative and contain all previously released fixes to guard against fragmentation of the OS that can lead to reliability and vulnerability issues when only a subset of fixes are installed. Most users are familiar with what is commonly referred to as “Patch Tuesday” or Update Tuesday. These updates are published on the second Tuesday of each month, known as the “B” release (“B” refers to the second week in the month), and are the only regular monthly releases that include both new security fixes and previously released security and non-security fixes. We chose the second Tuesday at 10:00 a.m. Pacific time to give commercial customers plenty of time to test the updates and deploy them to devices.
We also release optional updates in the third and fourth weeks of the month, respectively known as “C” and “D” releases. These are validated, production-quality optional releases, primarily for commercial customers and advanced users “seeking” updates. These updates have only non-security fixes. The intent of these releases is to provide visibility into, and enable testing of, the non-security fixes that will be included in the next Update Tuesday release (we make these optional to avoid users being rebooted more than once per month). Advanced users can access the “C” and “D” releases by navigating to Settings > Update & Security > Windows Update and clicking the “Check for updates” box. The “D” release has proven popular for those “seeking” to validate the non-security content of the next “B” release.
We also provide updates that don’t follow a standard release schedule. We refer to these as on-demand releases. They are used in atypical cases where we detect an issue and cannot wait for the next monthly release because devices must be updated immediately either to fix security vulnerabilities or to solve a quality issue impacting multiple devices.
Update quality validation
Monthly update quality is critical given the importance of the security and other fixes we regularly release at scale. As I noted in my previous blog post on the “Windows 10 quality approach for a complex ecosystem,” we use a combination of testing procedures to build and validate both feature updates and the monthly updates. Every day we build and package the latest fixes, and our engineers test and validate the fixes through a combination of the following activities:
- Pre-release Validation Program (PVP) flights updates to validate fixes in the current release before they are made available to in-market customers. The goal is to catch problems early by testing what we will ship and shipping exactly what we test.
- Depth Test Pass (DTP) consists of automated and manual testing targeted at the specific areas where the code has been changed to ensure the reported issue is indeed fixed, no new issues have been introduced as a result of the code change, and there are no regressions.
- Monthly Test Pass (MTP) utilizes broad suites of regression tests and leverages internal and external testing labs with global coverage that include tens of thousands of diverse devices from PCs to servers to ensure application and hardware compatibility.
- Windows Insider Program (WIP) flighting of non-security fixes to the Windows Insider Release Preview Ring to obtain feedback and diagnostics at a scale and diversity that mirrors the real-world. We do not flight pre-release security fixes for the upcoming “B” release to prevent attackers from reverse engineering the security fixes and placing customers at risk. We use a different quality program to validate security fixes.
- Security Update Validation Program (SUVP) is an invitation-only program for large commercial customers and ISVs, that enables them to validate the impact of security fixes in their labs prior to the “B” release, so that any compatibility issues or regressions with their infrastructure and applications can be identified and remedied. This is a tightly controlled program due to the security issues we are addressing, and only targets the “B” release.
- Regular compatibility and validation testing with other Microsoft product teams including Azure, Office and SQL Server.
Release information and monitoring
An integral component of an update release is the documentation we provide to keep users informed. Each release is accompanied by a knowledge base (KB) support article that communicates key release elements and issues as part of our overall transparency approach. Once an update is released, our listening systems monitor how the update is performing across our in-market population. To ensure users are having a good update experience, we monitor a wide array of feedback signals including:
- Live Site Validation Testing (LSVT) within minutes of the update going live to validate that the release is available on Windows Update and is successfully downloading and installing on devices scanning for new updates.
- Active monitoring of customer support and feedback channels for any user-reported issues.
- Social media and forums monitoring augmented with machine learning (ML) to quickly detect problems and facilitate remediation of issues.
Communications and transparency – more to come…
We have and will continue to invest in new quality-focused features that protect Windows customers and keep them up-to-date. For the Windows 10, October 2018 Update we are providing regular updates for notable issues on the public Windows 10 update history page. We plan to improve this throughout 2019 to provide more information about our actions or partner actions to mitigate issues. I’ll have more to share on additional quality related topics in future blog posts in this series.
Editor’s note: Edited on 12/14 to clarify that the “C” and “D” monthly releases are validated, production-quality optional releases.