Wikimedia Engineering/Report/2013/October
Engineering metrics in October:
- 134 unique committers contributed patchsets of code to MediaWiki.
- The total number of unresolved commits went from around 1020 to about 1122.
- About 35 shell requests were processed.
Major news in October include:
- A report on the Open Access Media Importer, a tool that transfers multimedia files from scientific publications to Wikimedia Commons;
- A request for proposals for a new datacenter in the continental US, published by the Operations team;
- The creation of the Autonym Font, which allows language names to be displayed properly without degrading page loading time.
Note: We're also providing a shorter, simpler and translatable version of this report that does not assume specialized technical knowledge.
Upcoming events
There are many opportunities for you to get involved and contribute to MediaWiki and technical activities to improve Wikimedia sites, both for coders and contributors with other talents.
For a more complete and up-to-date list, check out the Project:Calendar.
[edit or add events] [] [view credits]
| Date | Type | Event | Contact |
|---|---|---|---|
| 6 November 2013 |  |
Requests for Comments review Architecture meeting | Tim Starling |
| 7 November 2013 | |
WMF Metrics and Activities meeting | Erik Möller |
| 12 November 2013 | |
Engineering Community Team Office Hour | Quim Gil |
| 26 November 2013 | |
Engineering Community Team Showcase | Quim Gil |
Personnel
Work with us
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.
- Software Engineer - Fundraising
- Software Engineer - Growth
- Software Engineer - Core Features
- Software Engineer - Language Engineering
- Software Engineer
- Senior Software Engineer - Team Lead
- Software Engineer Data Analytics (Back End)
- Dev-Ops Engineer - SRE
- Ops Engineer - Labs Contractor
- User Experience Designer
Announcements
- Ori Livneh transitioned to the Platform Engineering group as Senior Performance Engineer (announcement).
- Gergő Tisza joined the Platfom Engineering group as Software Engineer on the Multimedia team (announcement).
- Leslie Carr and Ryan Lane were both promoted to the position of Senior Operations Engineer (announcement).
- Rummana Yasmeen joined the Platfom Engineering group as Software Test Engineer on the QA team, working primarily on VisualEditor (announcement).
- Vibha Bamba was promoted to the position of Senior User Experience Designer (announcement).
Technical Operations
Site infrastructure
- The team continued to heavily refactor the Puppet configuration: manifests for MySQL, nginx, SSH, puppetmaster, and several others are now properly organized as modules. Alexandros Kosiaris has made considerable progress towards supporting multiple puppetmasters on our cluster, which will greatly improve Puppet performance.
- Andrew Bogott has been working with Yuvaraj Pandian to get the new proxy system properly deployed; this will greatly reduce our need to hand out public IPs to labs users. We're close to hiring a contractor to help with the upcoming migration of Labs from Tampa to Ashburn.
Features Engineering
Editor retention: Editing tools
In October, the VisualEditor team continued to improve the stability and performance of the system, and add new features. The deployed version of the code was updated five times (1.22-wmf20, 1.22-wmf21, 1.22-wmf22, 1.23-wmf1 and 1.23-wmf2). Beyond fixing bugs, the focus of the team's work this month was to make a number of large structural changes to make the system more dependable and extensible, and continue to make some usability improvements. For example, you now need to press the "delete" key twice to delete a template, reference or image; the first time, they only become selected, to avoid accidental deletion of infoboxes and similar content. A new feature, empowering users to switch from editing in VisualEditor to editing wikitext directly without having to save the page, was also implemented.
In October, the Parsoid team continued to refine the parser behavior in edge cases. Performance was improved by increasing the parallelism of API requests and separating page updates from dependency-triggered updates in the job queue. The round-trip testing server performance was improved so that we can now run round-trip tests on 160,000 pages over night. Support for private wikis was also added this month.
We also made additional progress on Rashomon, the revision storage service based on Apache Cassandra. Rashomon is initially going to be used for implementing HTML and metadata storage for Parsoid output. Rashomon was deployed on a test cluster and import/write tests were performed.
Core Features
In October, we released Notifications worldwide on about 800 different sites, including most of the remaining Wikipedia wikis and 'sister projects'. Fabrice Florin and Keegan Peterzell managed the community outreach for these final releases, while Benny Situ proceeded with the technical deployments. Community members we've spoken to generally find this tool helpful, across languages and world regions. Notifications are also available on mobile devices, and seem to be adding value on these platforms as well. We now plan a final release of Notifications on the German and Italian editions of Wikipedia at the end of November. To learn more, visit our project hub, read the help page and join the discussion on the talk page.
Flow Portal/Project information
In October, the Flow team implemented a new visual design treatment on the Flow prototype (currently hosted on WMF Labs), and we continued to work through the set of features needed for a minimum viable product (MVP) on Wikipedia. We held an in-office User Experience workshop, primarily with users new to Wikipedia, to get feedback on the usability of the new design. We're planning to demo the Flow MVP to interested WikiProjects in November to get more feedback on what's needed for a first on-Wikipedia trial.
Growth
In October, the Growth team completed its sixth and final major A/B test of the GettingStarted and GuidedTour extensions for the onboarding new Wikipedians project. Data analysis results for this test were also completed and published, making way for rolling out the winning version on non-English Wikipedias in November. This month, the team also completed background research and early designs for its upcoming work on anonymous editor acquisition and Wikipedia article creation.
Support
The fundraising team has been steadily ramping up to the big English fundraiser, and so the fundraising tech team has been transitioning into their usual year-end fundraising monitoring and support role. At the end of October, we went in to a feature freeze for CentralNotice, and have been concentrating on bugfixes with the intent of increasing stability on the payments cluster and in the rest of the donation pipeline.
We have also started planning for the international fundraising campaigns targeted for early 2014.
This month, we fixed several bugs in the Education Program extension and added one much-needed UI feature. More improvements are in the pipeline. For now, we're focusing on issues that are not linked to the extension's lack of integration with ContentHandler, and on features that might carry over to a new version of the extension. We've also begun the groundwork for redesigning the UX and broadening the extension's scope.
Mobile
Many changes were made to the iOS Commons app in October. Users can now select the license among the most common ones, and swipe to delete categories on yet-to-be-uploaded images. A number of visual and interface improvements have been made, including on the settings page and the image details slider, notably for iOS 7 compatibility. Icon consistency has been improved throughout the app. The app now also has better internationalization support (via Autolayout), as well as better landscape and iPad support.
This month, the team released a wide array of features and enhancements. Part of the work focused on reducing accidental data charges, for example by adding interstitials and suppressing UI chrome icons. The latter change caused a 30-minute outage of JavaScript and CSS assets on Wikidata.
We started redirecting ineligible access attempts against subdomains zero.wikipedia.org to a unified warning page; this will reduce residual content in search engine caches, and reduce charges for users accidentally visiting zero.wikipedia.org subdomains while off participating networks.
We added support for partner self-management of configuration for JSON-oriented users; this will be enhanced with stage gating before any actual push to production.
Last, we nearly finalized IP address zero-rating lists (with the support of the Operations team), and we've also started working on Wikipedia Zero support for all Wikimedia projects.
The mobile web team has been focusing on a variety of new features, as well as running tests and gathering metrics on new and new-ish mobile editors. The team has also been working on general design enhancements to improve the user experience and make the look and feel consistent across all aspects of the mobile experience.
Feature highlights from the past month include: a better AbuseFilter support for mobile editing, early experimentation around integrating VisualEditor with MobileFrontend for tablets, client-side performance enhancements, the ability to 'thank' from the watchlist, and Captcha support for mobile editing.
In beta, 'Near this page' is now available, as well as user profiles and 'Keep Going' (a series of calls to action for new editors).Language Engineering
- The Language Engineering team worked on building a new Main Page for translatewiki.net, the primary translation platform for MediaWiki projects. This includes a new sign-in screen and project displays. A special feature is also being introduced for new translators through which they can provide a limited set of translations for evaluation.
- As part of the enhancements towards Universal Language Selector (ULS), a new font has been developed for displaying Language Names in their own scripts. This is known as the Autonym font and contains only the characters needed for the language names. Significantly small in size, this font will soon be deployed on Wikimedia projects. The team also worked on fixing a font inheritance issue for the edit area (bug 53734).
- The monthly release of the MediaWiki Language Extension Bundle (MLEB) was also completed.
- The team prepared for the open-source language summit to be held in Pune, India on November 18−19 in collaboration with Red Hat and other open-source internationalization developers.
Platform Engineering
MediaWiki Core
The last deployment from the 1.22 MediaWiki release was on October 24. To see all the changes that occurred during the 1.22 development cycle, see the main 1.22 release page or the RELEASE NOTES file. Final release of MediaWiki 1.22 is expected to be November 29.
The first deployment using the newly-branched 1.23 version of MediaWiki was deployed to Wikimedia servers on October 24.
Site performance and architecture
We implemented logging, aggregation & graphing of the VisualEditor DOM load & save timing. We also rolled out
mw.inspect, a library for inspecting static asset metrics. We configured stable URLs and improved cache headers for font resources, and rolled out a localStorage module caching to test wikis and the beta cluster.This activity is still officially on hold. However, some development of the global rename user tool is under way at gerrit:92468.
In October, CirrusSearch was deployed as a secondary search engine to Wikidata, all Wikivoyage wikis, and Wikipedia in Bengali. It became the primary search engine on Wiktionary in Italian, Wikipedia in Catalan and Wikisource in English. In November, we plan to deploy many more wikis including some larger than the Catalan Wikipedia. To expand to those larger wikis, we've negotiated some new hardware that should be deployed mid month.
We continued to refine the OAuth UX with the design team, and completed all major development tasks for the initial OAuth product. The first third-party application approved to use OAuth, "Gerrit Patch Uploader", was successfully used by several end users. We plan to finish the OAuth deployment in November.
Security auditing and response
We responded to several issues reported in core and extensions. An emergency password reset was put into place to address a private data security issue.
Quality assurance
In October the Quality Assurance project brought on a new software tester for VisualEditor, Rummana Yasmeen, who became productive quickly, reporting a number of issues and validating a VE deployment. QA spent a significant amount of time with the WMF Language team, using automated browser tests to find issues with the UniversalLanguageSelector and other Language software. QA continues to expand the utility of the beta labs test environment, supporting the new Flow extension there, and increasing the role of beta for deployments. Membership in the QA mailing list continues to grow, and October saw more contributions to the automated tests from volunteers. QA will participate in the Google Code In project as well.
October has been dedicated to consolidating the Jenkins configuration to make it easier to edit. Most actions are now handled by shell scripts under
integration/jenkins.git:/bin; editing the scripts doesn't require updating Jenkins jobs. The second slave server has been added to production and is successfully running PHPUnit tests. The packaging of dependencies required to upgrade Zuul has been completed, and Antoine Musso now has a version working in Labs. Finally, we investigated the possibility of running the browser tests whenever a change is submitted in Gerrit; that work is still in progress.
Thanks to Carl Fürstenberg's work during the Summer, we are now able to build some Debian packages straight into Jenkins using a dedicated instance and the Jenkins Debian glue scripts. The jobs are listed in Jenkins under the Ops-DebGlue view.Quality Assurance/Browser testing
October saw a major change to the architecture of the browser tests. Since they are now in many different repositories (VisualEditor, UniversalLanguageSelector, MobileFrontend, Flow, etc.), we consolidated the code shared among the various tests in all the repositories into a single library. This allows all the tests, present and future, to use the best and most up-to-date infrastructure. As for the tests themselves, we expanded coverage and identified issues for VisualEditor, Flow, gadgets, Language features, MobileFrontend, and more.
Engineering community team
Bugzilla now offers a new guided bug entry form which will make creating good bug reports easier for newcomers. Bugzilla now also displays metadata changes of a bug report inline for all logged-in users, so they can see in the comments who changed a value of a field (without clicking on "History"). Daniel Zahn upgraded Wikimedia Bugzilla to latest version 4.2.7. Legoktm mass-imported about 400 Pywikibot tickets from Sourceforge to Bugzilla. On a related note, Amir ran a PyWikibot Bug Triage resulting in nearly 100 tickets receiving updates. Furthermore, Andre Klapper investigated Wikimedia Bugzilla's customizations in CSS and code in order to clean up and sync with the upstream code base, to simplify current maintenance and also make a potential future upgrade of Wikimedia Bugzilla from version 4.2 to 4.4 easier.
We presented the summary of GSoC 2013 and FOSS OPW round 6 at the Wikimedia Metrics Meeting. The document about lessons learned in mentorship programs was updated. Wikimedia applied to Google Code-In and was successfully accepted on November 1. We opened the call for candidates for FOSS Outreach Program for Women - Round 7.
Besides ongoing communications support for the engineering staff, Guillaume Paumier published three issues of the Tech newsletter in October, and started a list of open tasks related to technical communications, available for anyone to work on, primarily in preparation for Google Code-In.
Volunteer coordination and outreach
Our proposal for a Wiki devroom at FOSDEM was accepted and we launched the Call for participation. On tech community metrics, we obtained first results of the Community metrics: Who contributes code key progress indicator. We facilitated the MediaWiki 1.22 Release Plan tech talk and the Flow UX Workshop meet-up. The list of participants of the Architecture Summit 2014 was published. We also experimented with a monthly cycle of Engineering Community Team meetings.
Multimedia
In October, we continued to expand our multimedia team and hired Gergő Tisza as software engineer. Mark Holmquist worked with Gergő to develop a first beta version of the Media Viewer, which displays images in larger size or full screen; this work was based on designs by Pau Giner, May Tee-Galloway and Jared Zimmerman.
We also completed development on the Beta Features program, which invites users to try out new features before they are released for everyone. A first version of both products is now ready for testing by logged in users on MediaWiki.org (to try new features, click on the small 'Beta' link next to your 'Preferences'). We plan to release the Beta Features program in coming days to Wikimedia Commons and Meta-Wiki, then to all wikis worldwide at the end of November. Fabrice Florin managed the development of both projects, and updated our multimedia plans to prepare for roundtable discussions with community members next month.
Bryan Davis started work on improving the thumbnail pipeline and guided the development of an upcoming GLAM Toolset for batch uploads by museum curators. To discuss these features and keep up with our work, we invite you to join the new multimedia mailing list. We are also recruiting for a senior software engineer position on our team.Analytics
The team designed and implemented multiple data center configuration support for Kafka (message bus). A bug involving buffer space allocation was exposed and fixed in the Varnish module, and infrastructure work done on automated data ingestion and partitioning. Product Analyst Oliver Keyes did some exploratory work with Hive and provided feedback to the Development team on ease of use and use cases.
The Threshold metric was implemented, and numerous bugs were fixed.
We created dashboards for several Wikipedia Zero partners (Orange Madagascar, Bangalink, Umniah Jordan), and identified and fixed Wikipedia Zero data issues in collaboration with the Zero team.
This month, we continued to support Growth and Mobile as the team's focus areas for this quarter. We published the results of the latest GettingStarted test run by the Growth team, we completed the cohort analysis for Mobile user acquisition and we worked with the Mobile team to prepare the launch a new test for new user activation, currently underway.
We analyzed active editor trends to determine whether the Wikipedia total active editors data for September 2013 (~67k) represented an anomalous change from seasonality and the long-term trend, and concluded that this was not the case. The results of this analysis call for the need to apply time series analysis and forecasting methods to other key performance indicators that the Foundation publishes on a daily or monthly basis.
We continued to work with the analytics engineers to provide requirements for Wikimetrics (with a particular focus on UserMetrics feature parity), and to perform data QA and validate the output of the application for metrics that were recently implemented.
We completed a round of consultations with internal stakeholders to identify research needs of each team in the organization and determine their priority. We presented a review of our activities for Q1 and plans for Q2 at the Analytics Quarterly meeting. We identified "metric standardization" as one of the goals the team will focus on in this quarter.
We organized and announced the inaugural Wiki Research Hackathon, a global event hosted in 8 locations in 5 countries, bringing together Wikimedia researchers, academics and community members to work on wiki research projects. The hackathon — the first event organized in the context of the Labs2 initiative — will take place on November 9, 2013.Kiwix
The Kiwix project is funded and executed by Wikimedia CH.
- We have released a new version (1.4) of Kiwix for Android; besides other improvements, we have introduced a localized user interface. We continue our efforts to speed-up the ZIM throughput; Wikipedia is now available offline in more than 60 languages. The last flash drives from our Deaddrops & Geocaching pilot were sent; we hope to get them walled up before the end of the year. We have also submitted a few proposals about Kiwix for Android for the Google Code-In program.
Wikidata
The Wikidata project is funded and executed by Wikimedia Deutschland.
- At the beginning of the month, Lydia Pintscher took over the product management for Wikidata. In a blog entry, she elaborated her goals for the future. The developers continued their work on the numbers datatype, sorting of statements and simple queries. In addition, a prototype for a multilingual picture dictionary was published, as well as a tool to visualize time and location data from Wikidata and another one to build lists based on data in Wikidata.
- October was also the month to celebrate Wikidata's first birthday. Enjoy some essays, interviews and more from the community.
Future
- The engineering management team continues to update the Deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the annual goals, listing ongoing and future Wikimedia engineering efforts.