Announcing TraceProcessor Preview 0.1.0

Process ETW traces in .NET.

Background

Event Tracing for Windows (ETW) is a powerful trace collection system built-in to the Windows operating system. Windows has deep integration with ETW, including data on system behavior all the way down to the kernel for events like context switches, memory allocation, process create and exit, and many more. The system-wide data available from ETW makes it a good fit for end-to-end performance analysis or other questions that require looking at the interaction between by many components throughout the system.

Unlike text logging, ETW provides structured events designed for automated data processing. Microsoft has built powerful tools on top of these structured events, including Windows Performance Analyzer (WPA), which provides a graphical interface for visualizing and exploring the trace data captured in a ETW trace file (.etl).

Inside Microsoft, we heavily use ETW traces to measure the performance of new builds of Windows. Given the volume of data produced the Windows engineering system, automated analysis is essential. For our automated trace analysis, we heavily use C# and .NET, so we created a package that provides a .NET API for accessing many kinds of ETW trace data. This technology is also used inside Windows Performance Analyzer to power several of its tables. We are happy to release a preview of this package that we use to analyze Windows so that you can use it to analyze your own applications and systems as well.

Getting Started

This package is available from NuGet with the following package ID:

Microsoft.Windows.EventTracing.Processing.All

After installing this package, the following console application lists all the process command lines in a trace:

using Microsoft.Windows.EventTracing; 
using Microsoft.Windows.EventTracing.Processes; 
using System; 

class Program 
{ 
    static void Main(string[] args) 
    { 
        if (args.Length != 1) 
        { 
            Console.Error.WriteLine("Usage: <trace.etl>"); 
            return; 
        } 

        using (ITraceProcessor trace = TraceProcessor.Create(args[0])) 
        { 
            IPendingResult<IProcessDataSource> pendingProcessData = trace.UseProcesses(); 
            trace.Process(); 
            IProcessDataSource processData = pendingProcessData.Result; 

            foreach (IProcess process in processData.Processes) 
            { 
                Console.WriteLine(process.CommandLine); 
            } 
        } 
    } 
}

The core interface is ITraceProcessor, and using this interface follows a pattern of first, tell the processor what data you want to use from a trace; second, process the trace; and finally, access the results. Telling the processor what kinds of data you want up front means you do not need to spend time processing large volumes of all possible kinds of trace data. Instead, TraceProcessor just does the work needed to provide the specific kinds of data you request.

Data Sources

Process information is just one of many kinds of data that can be stored in an ETW trace. Note that which data is in an .etl file depends on what providers were enabled when the trace was captured. The following list shows the kinds of trace data available from TraceProcessor:

Extensibility

Many kinds of trace data have built-in support in TraceProcessor, but if you have your other providers that you would like to analyze (including your own custom providers), that data is also available from the trace live while processing occurs.

For example, here is a simple way to get the list of providers IDs in a trace:

HashSet<Guid> providerIds = new HashSet<Guid>(); 
trace.Use((e) => providerIds.Add(e.ProviderId)); 
trace.Process(); 

foreach (Guid providerId in providerIds) 
{ 
    Console.WriteLine(providerId); 
}

The following example shows a simplified custom data source:

CustomDataSource customDataSource = new CustomDataSource(); 
trace.Use(customDataSource); 
trace.Process(); 
Console.WriteLine(customDataSource.Count); 

… 

class CustomDataSource : IFilteredEventConsumer 
{ 
    public IReadOnlyList<Guid> ProviderIds { get; } = new Guid[] { new Guid("your provider ID") }; 

    public int Count { get; private set; } 


    public void Process(EventContext eventContext) 
    { 
        ++Count; 
    } 
}

Feedback

The documentation for TraceProcessor is currently limited to partial IntelliSense support, though we would like to add more reference documentation as well as samples. (Documentation is a work-in-progress).

This release is a preview to gauge interest, so if this package is useful to you, please let us know. We would also appreciate feedback on what areas we invest in further, such as:

• Expanding IntelliSense documentation
• Adding TraceProcessor API reference documentation to docs.microsoft.com
• Adding samples
• Removing the requirement for msis in the bin directory
• Adding additional data to TraceProcessor that is currently present in WPA
• Stabilizing on a GA/v1 release

For questions using this package, you can post on StackOverflow with the tag .net-traceprocessing, and feedback can also be sent via email to [email protected].