Compare browser type systems with the new API Catalog tool

Interoperability has long been a driving priority for the Microsoft Edge team. To make informed, data-driven decisions, we need the ability to find the similarities and differences between the type systems of all browsers, to compare how core global functions, namespaces, objects, and interfaces are exposed. We also need to know how these match or differ from existing web specifications. Finally, we need an easy way to review the data.

To achieve all this, we built the API Catalog, a new tool which we’ve now made public for the benefit of the web community. Learn more in Greg Whitworth’s talk introducing our new Platform Data site at Microsoft Edge Web Summit, or read on for a detailed overview!

https://channel9.msdn.com/events/WebPlatformSummit/edgesummit2016/ES1607

A quick tour of the API Catalog

The API Catalog has two views: a visualization and a table view. The former is a high level visualization of the browser and specification landscape in the form of an interactive Venn diagram. You can select and compare any three data sets (browsers or specifications) to see a graphical representation of both intersecting and unique APIs.

Screen capture showing the API Catalog "visualization" view, which displays the overlap of browsers' type systems as an interactive Venn Diagram. — Compare API surfaces of different browsers and specifications using API Catalog’s visualization view.

The table view allows you to browse the full list of known APIs that are either defined in a specification we have crawled or detected in the type system of a browser. You can search and filter to examine specific subsets of APIs. For example, you might filter to see only APIs from the Beacon specification that are detected in all browsers:

Screen capture showing the API Catalog table view tool, filtered to show only APIs in the Beacon specification which are detected in the type systems of all browsers. — Filtered view of APIs in Beacon specification that are detected in all browsers.

The raw CSV data is also available on Github for in-depth analysis – we welcome your feedback on this dataset on our GitHub issues page.

Compiling the data

In the weeks since announcing the API Catalog at Microsoft Edge Web Summit, we’ve gotten some questions about where the data comes from. To have the most objective view of browser APIs, we wanted the raw, unprocessed data from browser implementations and web specifications. The API Catalog combines data from two primary sources: To collect browser data, we extract the type system data from the most recent versions of the Microsoft Edge, Google Chrome, Mozilla Firefox and Apple Safari for comparison. We then coalesce the data with the Web IDL definitions inside the specifications.

Collecting data from each browser requires looping over the type system starting at the Window object (WorkerGlobalScope in case of workers) and collecting only the members defined on an interface. We then verify each member (using <object>.hasOwnProperty) collected to verify if it is actually a member belonging to that object. This is used to eliminate members from the prototype chain. Results of that filter provide a view of the object that is consistent with what is defined in the specifications allowing the catalog to compare the data in a consistent format.

	Object.getOwnPropertyNames(obj).forEach(function (memberName, index, array) {
	...
	if (!obj.hasOwnProperty(memberName)) { return; } // is inherited
	...
	}

view raw apicatalog.js hosted with ❤ by GitHub

To collect specification data, we use specref.org, a database of over 14,000 (and growing) specifications and documents that are helping define what should be implemented in each browser. We use this database to look up the URLs for the specifications in order to gather the necessary Web IDL data. We look up the latest published documents and the latest editors draft URLs from the specref.org database. Each URL is loaded and the document is then parsed for Web IDL sections. This builds the specification definitions for the catalog’s interface and members. Today, we are using a subset of notable specifications to populate API Catalog. We plan to expand the specification data collection to all specifications in the near future – we’d love your feedback on which specifications to prioritize that we may be missing!

Understanding and using the data

The API Catalog shows a view of the API members grouped by interfaces while not mixing in the prototype chain members. In looking at the data, a number of low level questions about the specifications and the types systems of the browsers arose:

When the specification definition does not match the reality of browser implementations, is it a specification issue?
Do the browser implementations define the member in the incorrect place on the prototype chain?
Is this a browser proprietary member?
Is this member just defined wrong or in the wrong place?
How are ES 2015 features exposed on the browser’s platform types?

Each specification and every implementation has many definitions for interfaces and members. With so many definitions, it is often the case where members are incorrectly defined or missed altogether. Using the API Catalog, we can quickly see how implementations vary, what each browser may have implemented, and where it may have implemented the members. Note that missing APIs are not necessarily indicative of whether a browser supports a given feature―it may simply be that they don’t expose it in the type system in a way that can be detected using the methods we’ve described.

With API Catalog, you can also find that not every specification is defined correctly. For example, if we look at the WebGL 2 Specification, we will find a Web IDL definition for WebGLRenderingContextBase interface where there are typedef members incorrectly defined (see Issue 1600). Specifications also have definitions that may be at odds with implementations. This may be the case where the browser vendors will want to drive changes into the specification based on real world implementation (see Issue 182). Alternatively, there may just be a lack of implementations (see Issue 151).

Finally, the API Catalog also helps expose bugs in each browser’s type system. For example, Microsoft Edge in EdgeHTML 13 shows that it doesn’t support the length property for many interfaces. Length is actually supported, but is unfortunately not in the correct location on the prototype chain, which potentially could cause interoperability issues (e.g., Issue 110711).

Screen capture showing the API Catalog Table view , filtered to illustrate that Microsoft Edge erroneously does not declare the length property on many interfaces where it is actually supported. — Filtered view showing length as not detected on a number of interfaces in Microsoft Edge.

The API catalog can be used to expose or illustrate bugs in other browser type systems as well – for example, Webkit bug 49739 and Chrome bug 43394. Our goal for API Catalog is to show an unbiased view of each browser’s type system. With this data we believe all browser vendors are armed to make type systems more interoperable.

A resource for the web community

Through the API Catalog, we try to make every browser’s type system easy to navigate. The views we expose are similar to what a developer might create herself by running scripts within the page, or from each browser’s developer console. Browser vendors and the web community can work together to expand the “interoperable intersection” of the web platform. We welcome collaboration from browser vendors and contributors from the web community to improve this tool, and look forward to your feedback!

– Justin Rogers, Principal Software Engineer
– Arron Eicholz, Program Manager
– Joseph Shum, Program Manager