Experimentation Lab

The Experimentation Lab (previously Metrics Platform) is a platform, a suite of tools that helps Wikimedia teams make data-driven decisions about product experiences. You can use the Experimentation Lab to create instrumentation that captures click rates, conversion rates, and other metrics. Get started by choosing a workflow below.


Workflows

Guides

Templates

API docs

Reference

Meta

Experimentation Lab maintainer documentation

Decision records

Support

If you have questions that are not covered by these pages, or additional guidance is needed, please don't hesitate to reach out using the Data Products intake process.

Contributing

Contributions to the Experimentation Lab client libraries are most welcome and appreciated. Learn more about contributing to Experimentation Lab development

About

Built on the Event Platform, the Experimentation Lab provides standardized algorithms, behaviors, and basic necessities for web and app instrumentation, including:

  • a specialized Event Platform client designed to require less work in creating instrumentation
  • a predefined core interaction schema covering the most common data fields
  • a library of event schemas designed for use across a wide range of projects
  • a library of already existing instruments to be reused directly or as a starting point
  • easy means to build new schemas to further enrich events with contextual data
  • the ability to mix in different schemas depending on your needs
  • standardized session ID generation, consistent across MediaWiki, Android, and iOS
  • standardized session expiry
  • determining which events are in-sample or out-of-sample based on a specific identifier (currently: pageview, session, or app install ID).

To learn more about the Experimentation Lab and which components are available to use, see the FAQ.

When to use Experimentation Lab

Experimentation Lab is a good fit for products within MediaWiki without pre-existing Event Platform instrumentation. Here are a few situations where the Event Platform would be a better option for creating instrumentation:

  • Browser extensions, backend services outside of MediaWiki, or programming languages not covered by the xLab API
  • Websites outside of MediaWiki (although these sites can still use the Experimentation Lab schema conventions)
  • Existing features instrumented using the Event Platform. In this case, it can be preferable to continue to use the Event Platform to keep everything consistent

Background

Previously, different teams implemented their own analytics solutions in isolation from one another. Those solutions were typically based on the Legacy EventLogging pipeline and, more recently, the Event Platform. The Experimentation Lab is an effort to unify that previous work and to establish uniformity and consistency across platforms. Its objectives include:

  • making it easier to implement and maintain instruments. xLab provides the APIs and protocols that MediaWiki developers need to create sophisticated instruments in as few lines of code as possible while maintaining quality, rigor, and safety.
  • making it easier for analysts to support teams which are not their primary teams. Previously, every instrument has its own quirks and conventions, which need to be remembered by analyst that works with the data produced by the instrument – including analysts that are providing temporary support.
  • making it easier to leverage data from multiple platforms to yield insights into how our users use our whole ecosystem of products in unison.

All pages