Goal

This experiment sought to add suggestions to the empty state of our search bar, providing readers with recommendations on pages they could read next. We wanted to experiment with whether readers click on recommendations in the empty state of the search bar to see if this type of feature is helpful to readers.

In order to perform this experiment, we built a browser extension which injects the proposed recommendations into the search bar. Then we tracked clicks on the provided suggestions. We also considered using the extension itself to communicate to readers when the experiment is over.

Our hypothesis was that if we display recommendations in search, people will be interested in reading the recommended pages, as shown through a clickthrough rate similar to existing recommendation entry points (Related pages).

Experimental setup

A quicksurvey was displayed to a small percentage of readers on English and Spanish Wikipedias, inviting them to download a browser extension. Interested readers were prompted to download a browser extension which added the feature into the page. The feature provides search suggestions generated by the MoreLike API in Extension:CirrusSearch, the extension which provides search results. The suggestions are also currently available in the Related Pages feature and Extension:RelatedArticles.

Data was collected on how often readers clicked on the suggestions generated by the extension. No user or session data was collected.

Design

The search suggestions will appear once the user clicks into the search bar. The suggestions will clear once text is entered into the search bar, and will be replaced by the typeahead search suggestions relevant to the query.

Screenshot of search Suggestions feature on Italian Wikipedia

Results

We found that the readers interacted with the recommendations shown in the search bar at a rate higher than recommendations shown at the bottom of the page.

Report on the overall clickthrough rate (CTR) for search recommendations coming from the browser extension:

• On English wikipedia (enwiki) desktop, the overall CTR for search recommendation is 1.53%
• On Spanish wikipedia (eswiki) desktop, the overall CTR for search recommendation is 0.72%
• The overall CTR across both wikis is 1.44%

Comparison to clickthrough rate for related pages on desktop, mobile: we observed that the CTR for search recommendations is higher than that for related pages on desktop for both English Wikipedia and Spanish Wikipedia.

• English Wikipedia (enwiki): 1.53% vs 0.41% (CTR of related pages on desktop, enwikivoyage), diff 1.12 pp
• Spanish Wikipedia (eswiki): 0.72% vs 0.13% (CTR of related pages on desktop, eswikivoyage), diff 0.59 pp

Detailed results are available on T374965.

Next steps

• On mobile: Proceed with A/B test for the feature on one or more pilot wikis (pilot wiki list to be selected). Details on A/B test setup are available in phab:T378094.
• On desktop: Wait until the results of the article-level recommendations are ready and compare and proceed with the feature which shows better results.

Goal

After the successful browser extension experiment, we wanted to test the feature in a mobile environment, as well as gather data on its effects on higher order metrics such as session length. If the feature was successfully engaging people to learn more, we expected to see an increase in session length due to the feature in addition to clickthrough rates similar to the results we saw in the browser extension. To do this, we decided to test the feature on mobile in the context of an A/B test. Here, the control group was the current search bar, and the test group was the search bar with the search suggestions.

Our hypothesis was that users who interact with this feature will have sessions that are 5% longer than those who do not.

Experimental setup

The A/B test was run on anonymous mobile web users with the Minerva skin. It was enabled in tiers—starting with all wikis except enwiki on March 4 (UTC), and then deployed on enwiki on March 6 (UTC). The test was disabled on March 24, 2025. For this analysis, we reviewed A/B test data recorded from March 4 through March 23, 2025.

In this test, a randomly selected half of anonymous user sessions were assigned to the treatment group, where they saw empty search recommendations. The remaining half were assigned to the control group, where they experienced the current search setup and did not see any recommendations until they began typing in the search box.

Results

Click-through rate (CTR): 3.4% of searches resulted in clicks on the empty state recommendations. We estimate that 0.8% of these clicked searches would have engaged with autocompleted search recommendations even without the empty state feature. The remaining 2.6% of clicks represent boosted engagement driven by our new feature.

Session length:

• For all tested user sessions, being assigned to the treatment group increased session length by 0.1%, compared to sessions in the control group that focused on the search box but were not exposed to the empty search recommendations. The odds of a session having a length shorter than 30 seconds decreased by 0.68% when assigned to the treatment group. The overall impact is small, as only around 2.6% of sessions interacted with the empty-state search recommendations.
• For sessions that interacted with the empty state recommendations, engaging with the empty search recommendations increased session length by 15.4%, compared to sessions in the control group that focused on the search box but were not exposed to the empty search recommendations. Additionally, the odds of a session having a length shorter than 30 seconds decreased by 48% when users interacted with the empty search recommendations. However, the results may be influenced by selection bias, as sessions that interacted with the recommendations likely represent more active users.

Based on these results, we conclude that our hypothesis is confirmed, with our target of 5% being surpassed by the test results in which we saw a 15.4% increase. See the full results.