Previous research

Currently, much of the encyclopedic quality content is long-form and thus difficult to parse quickly. In addition, it is written at a reading level much higher than that of the average adult. Projects that simplify content, such as Simple English Wikipedia or Basque Txikipedia, are designed to address some of these issues. They do this by having editors manually create simpler versions of articles. However, these projects have so far had very limited success - they are only available in a few languages and have been difficult to scale. In addition, they ask editors to rewrite content that they have already written. This can feel very repetitive.

In our previous research (Content Simplification), we have identified two needs:

• The need for readers to quickly get an overview of a given article or page
• The need for this overview to be written in language the reader can understand

Wikimania 2024 session

At Wikimania 2024, Wikimedians discussed ways that AI/machine-generated remixing of the already created content can be used to make Wikipedia more accessible and easier to learn from. The session focused on ways to give editors control over the structure and content of these new features.

Userlytics experiment

We ran an experiment presenting article summaries to web readers. We wanted to determine whether article summaries have the potential to increase reader retention, as proxied by clickthrough rate and usage patterns. We were checking whether the proposed concept was clear to the users and whether the feature was usable.

We ran an unmoderated Userlytics study on a prototype of our Simple Summary feature with eight participants. The prototype included a summary of the introduction of the English Wikipedia article on Dopamine. We analyzed videos and screen recordings of the sessions.

Browser extension experiment

The second experiment added the new feature into the pre-existing browser extension. The goal was to determine whether summaries are worth future investment for more costly experimentation. This time, we gathered evidence for the following hypotheses:

1. Users are interested in the feature as presented (as measured through initial engagement with summary feature)

   Results: out of all pageviews where summaries were displayed, 8.09% of pageviews had a summary opened. The pages containing the summary received 10,198 impressions and 825 total clicks to open the summary.

2. Users report positive experiences with the user experience of the feature (as measured by the answers to the "was this useful?" question)

   Results: 101 responses were received in our survey. 75.2% of all clicks selected "yes", 24.8% of all clicks selected "no".

The above summary is based on data collected from the English Wikipedia (enwiki) desktop platform between December 4 and 17, 2024. Only users who had enabled the extension could view the summary, and not all pages included one. See phab:T381253 for more details

Design

For the browser extension experiment, the designs below were used. Summaries appear as collapsed when the page is loaded. If a user is interested in opening the summary, the can click the dropdown arrow, which allows the summary to appear.

The dropdown opens the summary, as well as additional information on how the summary was generated. It also contains the "unverified" tag to indicate this summary has not been verified by editors. (Once a future moderation workflow is in place, this tag will show as "verified" after editor verification)

• 
  Screenshot of collapsed state of article summary browser extension experiment
• 
  Screenshot of the open state of the simple article summary browser extension for the Dopamine article

Goals for further experimentation

Based on the results above, we believe this feature has the potential to significantly benefit readers. We have identified two hypotheses that will help guide and direct this work in the future that can focus both on the technical and moderation aspects of our next steps.

1. Evaluate impact of a proposed summary feature on key user metrics on a Wikimedia wiki
2. Evaluate community interest and sentiment towards the summary feature, and its range across different language communities
3. Explore community involvement in the feature, specifically focusing on the need and approach towards summary moderation
4. Explore requirements for improvements and applications of the model used for summary generation

Mobile summary pilot (Hypothesis 1)

If we introduce a pre-generated summary feature as an opt-in feature on a the mobile site of a production wiki, we will be able to measure a clickthrough rate greater than 4%, ensure no negative effects to session length, pageviews, or internal referrals, and use this data to decide how and if we will further scale the summary feature.

Moderation consultations (Hypothesis 2)

If we approach summary moderation design collaboratively with communities — through surveys and other on-wiki discussions, we will be able to determine the minimal viable moderation workflow required for initial scaling of the feature and clarify whether moderation should be community-led, automated (at the prompt level), or some combination of both.