Wikimedia Search Platform

Category:Languages pages Category:WMF ProjectsCategory:WMF Projects 2017q4Category:WMF Projects 2018q1Category:WMF Projects 2018q2Category:WMF Projects 2018q3Category:WMF Projects 2018q4Category:WMF Projects 2019q1Category:WMF Projects 2019q2Category:WMF Projects 2019q3Category:WMF Projects 2019q4Category:WMF Projects 2020q1Category:WMF Projects 2020q2Category:WMF Projects 2020q3Category:WMF Projects 2020q4Category:WMF Projects 2021q1Category:WMF Projects 2021q2Category:WMF Projects 2021q3Category:WMF Projects 2021q4Category:WMF Projects 2022q1Category:WMF Projects 2022q2Category:WMF Projects 2022q3Category:WMF Projects 2022q4Category:WMF Projects 2023q1Category:WMF Projects 2023q2Category:WMF Projects 2023q3Category:WMF Projects 2023q4Category:WMF Projects 2024q1Category:WMF Projects 2024q2Category:WMF Projects 2024q3Category:WMF Projects 2024q4Category:WMF Projects 2025q1Category:WMF Projects 2025q2Category:WMF Projects 2025q2

The Search Platform team (part of Wikimedia Technology) is responsible for maintaining and enhancing the various Search features and APIs for MediaWiki. This includes the CirrusSearch extension which relies on OpenSearch (formerly Elasticsearch), the search backend used at the Wikimedia Foundation to support Wikimedia projects and Wikidata Query Service, the SPARQL endpoint used to query Wikidata.

Current work by this team is tracked on the Discovery-Search workboard in Phabricator (backlog board).

Mission

Our mission is to help people easily discover knowledge on Wikipedia and its sister projects by providing tools and infrastructure for casual readers and expert users with precise needs, while maintaining a strong emphasis on privacy.

Overview

  • We operate and maintain a disparate collection of production services related to content discovery, enabling the wiki community to find information that is not available through simply following links. We also provide a platform on which other people can create tools to support editing and other workflows.
  • We provide an open-source search engine, backed by an inverted index for non-structured on-wiki data. We work to develop more sophisticated searching with machine learning and natural language processing.
  • We provide a SPARQL-based query service for Wikidata, encouraging users to capitalize on this vast store of computer-readable structured data for use on-wiki and in knowledge discovery.
  • We endeavor to support underserved wiki communities, and we rely on those communities to help us understand their needs and evaluate potential solutions, especially with respect to underserved languages.
  • We prioritize privacy for logged-in users and anonymity for logged-out users over almost everything else, even when it slows down or complicates development or hinders our ability to collect or use data.

Goals

The Search Platform team's goals are part of the Product & Technology department's goals (e.g., P&T FY 24-25 OKRs and other Essential Work related to sustaining the platform). You can see links to the Search Platform team's current quarterly goals and some previous 2017-2021 quarterly goals. (Note that each fiscal year's Q1 is July–September.)

Other Projects

Wikidata Query Service (WDQS)

The Wikidata query service allows for searching structured data on Wikidata. It also provides an API through which tools can access Wikidata. Our current work is tracked on the Discovery-Search workboard (see also our WDQS backlog board) and weekly deployments of WDQS are documented on wikitech:Deployments. w:SPARQL

APIs

Application Programming Interfaces (APIs) provide developers ways to interact with the MediaWiki software.

API:Search and discovery lists the search APIs available and in development.

The Team

See the team page on Wikitech.

Communications

Moved to wikitech:Search Platform/Contact

Process

The Search Platform team uses a Scrumban process, which is a hybrid of Scrum and Kanban. It is described here: Search Platform Process.

Conferences, gatherings, and other events

Upcoming events

  • none scheduled due to COVID-19 global pandemic

Past events

Moved to wikitech:Search Platform/Documentation

Deployers

Useful reference for who can deploy code. It's nice to know whom to bug if you need something:

Person MediaWiki

Deployer

OpenSearch

Deployer

Maps

Deployer

Graphoid

Deployer

Portals Deployer
dcausse Yes Yes
ebernhardsen Yes Yes
jan_drewniak Yes
gehel Yes Yes Yes

Code

The Search Platform team supports the following code:

Repository Phabricator/Diffusion GitHub mirror
CirrusSearch extension https://phabricator.wikimedia.org/diffusion/ECIR/ mediawiki-extensions-CirrusSearch
Elastica extension https://phabricator.wikimedia.org/diffusion/EELA/ mediawiki-extensions-Elastica
GeoData extension https://phabricator.wikimedia.org/diffusion/EGDA/ mediawiki-extensions-GeoData
Wikidata Query Service https://phabricator.wikimedia.org/diffusion/WDQR/ wikidata-query-rdf
Wikidata Query Service GUI https://phabricator.wikimedia.org/diffusion/WDQG/ wikidata-query-gui
WDQS deployment https://phabricator.wikimedia.org/diffusion/WDQD/ wikidata-query-deploy
WDQS GUI deployment wikidata-query-gui-deploy
PHP textcat https://phabricator.wikimedia.org/diffusion/WTEX/ wikimedia-textcat
Relevance Forge wikimedia-discovery-relevanceForge
Discernatron wikimedia-discovery-discernatron
Discovery Analytics https://phabricator.wikimedia.org/diffusion/WDAN/ wikimedia-discovery-analytics
Lucene Explain Parser https://phabricator.wikimedia.org/diffusion/WLEP/ wikimedia-lucene-explain-parser
Category:Discovery
Category:Discovery Category:Languages pages Category:WMF Projects Category:WMF Projects 2017q4 Category:WMF Projects 2018q1 Category:WMF Projects 2018q2 Category:WMF Projects 2018q3 Category:WMF Projects 2018q4 Category:WMF Projects 2019q1 Category:WMF Projects 2019q2 Category:WMF Projects 2019q3 Category:WMF Projects 2019q4 Category:WMF Projects 2020q1 Category:WMF Projects 2020q2 Category:WMF Projects 2020q3 Category:WMF Projects 2020q4 Category:WMF Projects 2021q1 Category:WMF Projects 2021q2 Category:WMF Projects 2021q3 Category:WMF Projects 2021q4 Category:WMF Projects 2022q1 Category:WMF Projects 2022q2 Category:WMF Projects 2022q3 Category:WMF Projects 2022q4 Category:WMF Projects 2023q1 Category:WMF Projects 2023q2 Category:WMF Projects 2023q3 Category:WMF Projects 2023q4 Category:WMF Projects 2024q1 Category:WMF Projects 2024q2 Category:WMF Projects 2024q3 Category:WMF Projects 2024q4 Category:WMF Projects 2025q1 Category:WMF Projects 2025q2