Review Optimization
Some figures
Start date: September 2023
End date: ∞
Participants: –
Project goal and tasks
The goal is to optimize the work of reviewers by automating and eliminating the need for them to do "extra" work.
Tasks:
Analysis of Wikipedia’s active members' contributions, search for those whose contributions meet the reviewing requirements, and assist in granting them the rights of the autoreviewers or reviewers.
Step-by-step analysis of individual aspects of reviewing, the search for certain actions that give a profound result in the automation of the review process of the Ukrainian Wikipedia.
Key organizers and project executors
NUAAR's Wikipedia Department
Project outcomes
Contributions of over 200 participants were analyzed, of which 3 eventually received reviewer rights and 4 received autoreviewer rights. This allowed reviewers to save time on checking about 900 edits per month (which is equivalent to 15-40 man-hours). Thanks to this, as well as the first iteration of the School of Reviewers, the number of edits that need to be reviewed has decreased by approximately 4.6%.
A series of several small analytical point studies on the automation of reviewing articles of certain types is being prepared.
Cool facts
–
Project conclusions
Some results from the first task have already been obtained, and this process will continue.
A separate committee consisting of 3-4 experienced reviewers and administrators will be created to assess the quality of contributions made by users who have the potential to become reviewers or autoreviewers.
Message from the CEO of NUAAR
It has long been evident to me that the autoreview function on Ukrainian Wikipedia is not being fully utilized. Firstly, not all users whose contributions can be considered almost exclusively positive and meet specific clear criteria for reviewing are granted autoreviewer rights. Secondly, some of these users are given reviewer rights instead of autoreviewer, and they only review their own articles, meaning they do not fully utilize their reviewer rights. As a result, some reviewers are not reviewing at all, while others are forced to review contributions that do not require manual verification. This is happening even though there are nearly 300,000 pages with unreviewed edits, and this number increases by an average of ~1500-1800 each month.
If the solution to the second problem lies rather in the "legislative" realm, the first one can be addressed simply by allocating a specific time to review user's contributions. In practical terms, achieving the result requires only about 2-3 hours per person. This time investment will be compensated in the first month of an autoreviewer's work, as others will not need to review dozens or even hundreds of edits.
Also, what can be "reviewed" automatically must necessarily be reviewed in such a way. For example, categories in an article either exist or do not. Similarly, with sources in an article – which are either present or absent. It does not require creating any artificial intelligence or conducting manual checks (consuming the reviewer's scarce time). A relatively simple script that checks for the presence of keywords or tags is sufficient. Such automatic checks for all Wikipedia articles will save thousands of hours of review work, leaving them only to press "Approve version".
It is necessary, first and foremost, to analyze such possible automatic actions and the expected effect. Subsequently, implement those that will be effective enough in terms of results compared to efforts.
Related projects
Wikipedia Project: Improving Good and Featured Articles
Wikipedia Project: Addition of Notability Criteria
School of Reviewers