20 May 2025

Peacock check becomes Tone Check, model test to start soon

As the term "Peacock" was specific to the English language, internationalizing it was challenging. The Editing team decided to change the name of the feature for a more universal term: Tone check.

Tone check will use a model, trained by users. This first test will be conducted for English, Spanish, Portuguese, French and Japanese languages. USers are invites to signup at Edit check/Tone Check/model test before May 23^rd.

12 May 2025

Peacock check model test to start soon

Peacock check will use a model, trained by users. We will soon start reaching at a few communties (listed at T388471) and ask for volunteers to test the model.

We selected the wikis based on several criteria:

• Technical reasons:
  • Wikis that use the “variants” feature (like Chinese) — because the model has to infer across different language varieties
  • Languages that don’t space-separate words (like Chinese, Japanese) — where the results will be very dependent on the tokenizer
  • Agglutinative languages (like Turkish, Indonesian) — where the model will be very dependent on the tokenizer
  • RTL (like Arabic, Hebrew) — because of potential user experience (UX) issues
• projects that see relatively high volumes of newcomers, specifically in Sub-Saharan Africa, and
• projects that have expressed a willingness to experiment with Peacock Check.

1 May 2025

A/B Test: Multiple Reference Checks

An analysis of leading indicators for the Multiple Reference Check A/B test is complete and the results are encouraging:

• New(er) volunteers are encountering Multiple Reference Checks in enough editing sessions to draw statistically significant conclusions from
• People shown multiple Reference Checks within an edit session are proceeding to publish edits that include ≥1 reference at relatively high rates
• Showing people multipole Reference Checks within an edit session is not leading to increases in revert rate or blocks

You can review these results in more detail below.

For context, leading indicator analyses of this sort are meant to uncover what – if any – adjustments we will consider prioritizing before evaluating the broader impact of the feature in question.

26 March 2025

Peacock Check

In collaboration with the Machine Learning team, the Editing team has started working on a new check: Peacock check (T368274) This check will detect the usage of puffery terms, and encourage the user to change them.

We are currently gathering on-wiki policies, templates used to tag non-neutral articles, and the terms (jargon) used in edit summaries for 20 wikis.

A/B Test: Multiple Reference Checks

Yesterday (25 March), an A/B test began at 12 Wikipedias that removes the constraint on how many Reference Checks people have the potential to see within a single edit.

Note: at present, a maximum of one Reference Check is shown per edit.

This experiment is an effort to learn: What – if any – changes in edit quality and completion do we observe when people have the potential to see multiple Reference Checks within a single edit?

The findings from this A/B test will be relevant for the near-term future where multiple Edit Checks of the same and/or different types (e.g. Peacock Check, Paste Check, etc.) have the potential to become activated within a single edit session.

5 March 2025

Curious to learn if and how the new desktop Edit Check experience might have impacted edit quality, we investigated how the number of reverted edits changed before and after the December 2024 release.

We learned:

1. The revert rate of new content edits where Reference Check was presented decreased by 15.7% (20.4% pre- to 17.2% post-change).
2. There was a 8% increase in the proportion of new content edits that included a reference following the change (34.8% pre-change to 37.9% post-change).

See full report for more details

7 February 2025

In December, we released a new design for the Edit Check desktop experience.

Last week, we finished an analysis that compared how – if at all – several key metrics shifted before and after this change.

The purpose of this analysis: decide whether there were any changes to the user experience we ought to prioritize making before beginning an A/B test that will evaluate the impact of it being possible for multiple Reference Checks to be shown within a single edit.

You can find a summary of the results below and the full report here.

Reference Detection

To start, the Editing Team is pursuing an approach with Edit Check that minimizes the likelihood of false positives and is implemented in ways^[7] that empower volunteers, on a per-project basis, to evolve the heuristic^[8] to become more robust over time.

This strategy amounts to the initial reference Edit Check becoming activated if/when all of the following conditions are met:

1. A minimum of one new paragraph of text is added to the article someone is editing
2. The "new paragraph(s) of text" someone has added does NOT include a reference
3. The changes described in "1." and "2." are happening on a page within the main namespace (NS:0)

The conditions above are implemented and maintained in code here: editcheck/init.js.

The Editing Team arrived at the decision to start with a relatively limited and straightforward set of rules in order to:

1. Increase the likelihood that newcomers and Junior Contributors find the guidance Edit Check is presenting them with, and the editing experience more broadly, to be intuitive and straightforward so that they feel encourage to return to edit again
2. Decrease the likelihood that Edit Check is creating more work for experienced volunteers by prompting newcomers and Junior Contributors to add sources when they are not needed

You can learn more about the assumptions that informed the thinking above in phab:T329988#8654867.

Other applications

Configurability

The Editing Team thinks it is crucial that moderators be empowered to configure when, and for whom, Edit Check becomes activated. This way, they can be confident the software is promoting behavior they deem to be productive and modify the software when it is not.

In line with the above, and drawing inspiration from how the Edit filter and Growth Team Community configuration systems afford volunteers the ability to audit and configure how they function on-wiki, Edit Check will enable volunteers, on a per project basis to:

• Audit and edit the logic that determines when the reference Edit Check becomes activated and
• Review the edits people who are shown Edit Check are making

Work to implement the above is ongoing in phab:T327959.

User Experience

Multi-Check (References) A/B Test

Multi-Check Phase 1 Impact Analysis

In December 2024, we released a new design for the Edit Check desktop experience. This change shifted Edit Checks from appearing within articles to appearing alongside them, in a new "siderail."

To decide how – if at all – this change impacted volunteer disruption and edit quality, we compared analyzed several key metrics shifted before and after this change.

Findings

• The revert rate of new content edits where Reference Check was activated decreased by 15.7%.
  • 20.4% ( pre) →  17.2% ( post).
• The proportion of new content edits that included a reference following the change increased 8%.
  • 34.8% ( pre) → 37.9% ( post)
• Excluding reverted edits, there was 3.2% increase [2 percentage points] in edit completion rate.
  • 68% of edits where Reference Check was presented were successfully saved and not reverted following the change in the Edit Check UX.
• The rate at which people declined to add a reference when Edit Check prompted them decreased 4.7%.

Conclusion(s)

The findings above suggest this design change has been net positive. As a result, we will continue depending on this new design paradigm as we introduce new types of Edit Checks and moments within the editing workflow when they are presented.

Reference Check A/B Test

To learn whether the Reference Edit Check is effective at causing newcomers to make edits they intended and experienced volunteers value , we conducted an A/B test with 15 Wikipedias.

Below you can read more about what this experiment demonstrated, what the Editing Team is planning in response, and more details about the test's design.

Conclusion and next step(s)

Reference Check caused an increase in the quality of edits newcomers publish and did not cause any significant disruption.

This combination is leading the Editing team to be confident that offering Reference Check as a default-on feature would have a net positive impact on all wikis and the people who contribute to them.

You can read the full A/B test report here.

Findings

New content edits *with* a reference

People shown the Reference Check are 2.2 times more likely to publish a new content edit that includes a reference and is constructive (not reverted within 48 hours).

• Increases were observed across all reviewed user types, wikis, and platforms.

• The highest observed increase was on mobile where contributors are 4.2 times more likely to publish a constructive new content edit with a reference when Reference Check was shown to eligible edits.

Revert rate

• New content edit revert rate decreased by 8.6% if Reference Check was available.
  • New content edits by contributors from Sub-Saharan Africa are 53% less likely to be reverted when Reference Check is shown to eligible edits.

While some non-constructive new content edits with a reference were introduced by this feature (5 percentage point increase), there was a higher proportion of constructive new content edits with a reference added (23.4 percentage point increase). As a result, we observed an overall increase in the quality of new content edits.

Constructive Retention Rate

• Contributors that are shown Reference Check and successfully save a non-reverted edit are 16 percent more likely to return to make a non-reverted edit in their second month (31-60 days after).
  • This increase was primarily observed for desktop edits. There was a non-statistically significant difference observed on mobile.