Wikipedia:Pending changes/Metrics/Preliminary Analysis
Analysis by Howief
[edit]Note: Per Ocaasi's request, here is some basic analysis on the Pending Changes data. This analysis is meant to serve as a starting point to help focus the data for community discussion. Since individual reviewers have the deepest experience with the feature and how it works with specific types of articles, I've kept most of the commentary on the quantitative aspects. The exception here is the discussion of most frequently edited articles, where I pointed out some obvious commonalities. Hopefully the cuts of data in this analysis will be useful. If not, there are the full tables which people may use to look at the data in detail.
Throughout this analysis, I've focused on metrics that we've frequently used to evaluate the success of Pending Changes:
- Participation: approximated by the number of anonymous edits an article under Pending Changes receives
- Quality: the quality of the anonymous edits. This is approximated by the % of anonymous edits that are not reverted
- Work: the amount of work required from reviewers to manage the incoming anonymous edits. This is approximated by the number of anonymous edits an article receives, the idea being that anonymous edits require review, which requires work from reviewers.
Howief (talk) 23:52, 20 August 2010 (UTC)
First, I made a simplifying assumption. The amount of time that an article is under pending changes for varies and this variation results in outliers later in the analysis. For example, slag was under pending changes for only 38 seconds. I decided to exclude articles that have been under pending changes for less than one week (152 out of 1647, or 9.2% of total). The total number of articles that have been under pending changes for at least one week is 1495.
By Average Daily Anon Edits
[edit]Comparing articles by the total number of edits would not take into account the fact that some articles have been under pending changes longer than others. So normalizing by the number of days under pending changes can help us compare articles that have been under Pending Changes for differing amounts of time. Here is a histogram of articles by the number of days the articles were under Pending Changes:
| Ave Edits/Day | # of Articles | % of total |
| >10 | 3 | 0.2% |
| 5-10 | 14 | 0.9% |
| 2-5 | 70 | 4.7% |
| 1-2 | 150 | 10% |
| .5-1 | 206 | 14% |
| .25-.5 | 238 | 16% |
| .1-.25 | 226 | 15% |
| 0-.1 | 303 | 20% |
| 0 | 285 | 19% |
| Total | 1,495 | 100% |
One of the goals of Pending Changes is to open up semi-protected articles to anonymous editors. Yet from the data, we see that 84% of the articles under pending changes receive, on average, less that one edit per day. Much of this may be due to the natural edit velocity of the articles that have been placed under Pending Changes. Note that this is not a judgement on the value of the edits as a single edit may prove very valuable to the quality of an article. All this analysis tells us is that the frequency of anonymous edits is relatively low for a large percentage of the articles that were under Pending Changes.
Of articles that were under Pending Changes for at least one week, here are the 10 articles that received the most number of edits per day from anonymous users:
| Page title | Time under Pending (days) | Total anon edits while under Pending | Anon edits/day | % nonreverted/total anon edits |
| Total Drama World Tour | 20.77 | 493 | 23.7 | 71.8% |
| Spain national football team | 8.00 | 158 | 19.8 | 65.8% |
| Toy Story 3 | 12.47 | 146 | 11.7 | 51.4% |
| Australia's Next Top Model, Cycle 6 | 14.21 | 140 | 9.9 | 65.7% |
| Juan Pablo Pino | 14.00 | 132 | 9.4 | 17.4% |
| Wayne Rooney | 8.05 | 70 | 8.7 | 12.9% |
| SummerSlam (2010) | 14.33 | 114 | 8.0 | 27.2% |
| The Twilight Saga: Eclipse | 30.00 | 224 | 7.5 | 42.0% |
| Jason Leopold | 7.63 | 56 | 7.3 | 17.9% |
| Manchester City F.C. | 10.81 | 77 | 7.1 | 26.0% |
It's interesting to note that almost all of these articles are about current events. Toy Story 3 and The Twilight Saga: Eclipse were released around the time of the trial. Total Drama World Tour and Australia's Next Top Model, Cycle 6 are TV shows that were active during the time of the trial. The World Cup gives rise to Spain national football team, Juan Pablo Pino, Wayne Rooney, and Manchester City F.C.. And SummerSlam (2010) is a WWE Wrestling event that took place on Aug. 15, 2010.
Here are examples of articles that received between 1-2 edits per day:
| Page title | Time under Pending (days) | Total anon edits while under Pending | Anon edits/day | % nonreverted/total anon edits |
| Thomas Edison | 7.16 | 12 | 1.7 | 0.0% |
| Club Penguin | 42.20 | 70 | 1.7 | 14.3% |
| The Simpsons | 33.35 | 55 | 1.6 | 12.7% |
| Sum 41 | 44.67 | 73 | 1.6 | 49.3% |
| 2010 Central Canada earthquake | 42.30 | 69 | 1.6 | 34.8% |
| Nicolas Mahut | 41.54 | 67 | 1.6 | 67.2% |
| Republic of Macedonia | 27.28 | 44 | 1.6 | 15.9% |
| Oksana Grigorieva | 22.40 | 36 | 1.6 | 50.0% |
| Baseball | 44.28 | 71 | 1.6 | 4.2% |
| World War II | 10.64 | 17 | 1.6 | 0.0% |
And the following are examples of articles that received no edits during the time they were under pending changes:
| Page title | Time under Pending (days) | Total anon edits while under Pending | Anon edits/day | % nonreverted/total anon edits |
| Sonny Fai | 45.29 | 0 | - | |
| Richard Goldberg | 45.29 | 0 | - | |
| Wildcliff | 45.29 | 0 | - | |
| Steven Geller | 45.29 | 0 | - | |
| Robert Garside | 45.28 | 0 | - | |
| Tran Van Huong | 45.28 | 0 | - | |
| William B. Black (Illinois politician) | 45.28 | 0 | - | |
| Suryoyo Sat | 45.23 | 0 | - | |
| Tig Notaro | 45.23 | 0 | - | |
| Steve Hoffman | 45.23 | 0 | - |
If we take the articles that have at least one anonymous edit per day while under pending changes (237 total articles), we can then look at the quality of the edits. Articles which have at least 50% of anonymous edits unreverted:
| Page title | Time under Pending (days) | Total anon edits while under Pending | Anon edits/day | % nonreverted/total anon edits |
| Alvin Greene | 17.00 | 25 | 1.5 | 88.0% |
| Uruguay national football team | 33.14 | 111 | 3.3 | 82.0% |
| Total Drama World Tour | 20.77 | 493 | 23.7 | 71.8% |
| France | 34.43 | 150 | 4.4 | 68.0% |
| Nicolas Mahut | 41.54 | 67 | 1.6 | 67.2% |
| Sonisphere Festival | 46.39 | 66 | 1.4 | 66.7% |
| Angel Locsin | 17.65 | 18 | 1.0 | 66.7% |
| Spain national football team | 8.00 | 158 | 19.8 | 65.8% |
| Australia's Next Top Model, Cycle 6 | 14.21 | 140 | 9.9 | 65.7% |
| John Isner | 41.54 | 89 | 2.1 | 65.2% |
| Australia | 44.28 | 90 | 2.0 | 64.4% |
| The Hunchback of Notre Dame (1996 film) | 45.23 | 51 | 1.1 | 62.7% |
| Gary Glitter | 43.30 | 45 | 1.0 | 62.2% |
| Madonna (entertainer) | 42.23 | 118 | 2.8 | 60.2% |
| This Is It (Michael Jackson concerts) | 45.44 | 62 | 1.4 | 58.1% |
| Big Brother 2010 (UK) | 14.00 | 56 | 4.0 | 55.4% |
| 2010–11 Arsenal F.C. season | 28.61 | 34 | 1.2 | 52.9% |
| 2010 Formula One season | 25.29 | 55 | 2.2 | 52.7% |
| Toy Story 3 | 12.47 | 146 | 11.7 | 51.4% |
| Oksana Grigorieva | 22.40 | 36 | 1.6 | 50.0% |
| List of programs broadcast by Cartoon Network | 21.39 | 30 | 1.4 | 50.0% |
On the opposite end of the spectrum, the following articles that have at least one anonymous edit per day, none of which were unreverted:
| Page title | Time under Pending (days) | Total anon edits while under Pending | Anon edits/day | % nonreverted/total anon edits |
| List of serial killers by country | 9.20 | 32 | 3.5 | 0.0% |
| Osama bin Laden | 9.16 | 31 | 3.4 | 0.0% |
| David Henrie | 7.26 | 19 | 2.6 | 0.0% |
| Batman | 11.21 | 25 | 2.2 | 0.0% |
| Chicken | 32.90 | 64 | 1.9 | 0.0% |
| Emile Heskey | 15.32 | 29 | 1.9 | 0.0% |
| Robert Kotick | 8.58 | 15 | 1.7 | 0.0% |
| Brad Pitt | 10.92 | 19 | 1.7 | 0.0% |
| Thomas Edison | 7.16 | 12 | 1.7 | 0.0% |
| World War II | 10.64 | 17 | 1.6 | 0.0% |
| Westboro Baptist Church | 34.14 | 49 | 1.4 | 0.0% |
| Dinosaur | 43.55 | 60 | 1.4 | 0.0% |
| Akshay Kumar | 12.33 | 16 | 1.3 | 0.0% |
| Apple | 34.38 | 42 | 1.2 | 0.0% |
| Danielle Staub | 18.91 | 23 | 1.2 | 0.0% |
| South Slavic languages | 16.54 | 20 | 1.2 | 0.0% |
| Christopher Columbus | 29.12 | 34 | 1.2 | 0.0% |
| Penguin | 11.48 | 13 | 1.1 | 0.0% |
| Victoria of the United Kingdom | 13.58 | 15 | 1.1 | 0.0% |
| Turtle | 19.36 | 21 | 1.1 | 0.0% |
| Moon | 20.08 | 21 | 1.0 | 0.0% |
| Amy Winehouse | 11.48 | 12 | 1.0 | 0.0% |
| Cheetah | 11.62 | 12 | 1.0 | 0.0% |
| Obesity | 45.79 | 46 | 1.0 | 0.0% |
By Total Anon Edit Count
[edit]Another way of cutting the data is by anonymous edit count (rather than average anonymous edits per day). The following is a histogram based on the total number of edits during the trial:
| Total Anon Edits | # of Articles | # of Total |
| >200 | 3 | 0.2% |
| 100-199 | 32 | 2.1% |
| 50-99 | 108 | 7.2% |
| 25-49 | 190 | 13% |
| 10-24 | 302 | 20% |
| 5-9 | 192 | 13% |
| 1-4 | 383 | 26% |
| 0 | 285 | 19% |
| Total | 1495 | 100% |
Of articles that received at least 50 edits during the period, the following articles had over 50% of the anonymous edits unreverted:
| Page title | Time under Pending (days) | Total anon edits while under Pending | Anon edits/day | % nonreverted/total anon edits |
| Uruguay national football team | 33.14 | 111 | 3.3 | 82.0% |
| Total Drama World Tour | 20.77 | 493 | 23.7 | 71.8% |
| France | 34.43 | 150 | 4.4 | 68.0% |
| Nicolas Mahut | 41.54 | 67 | 1.6 | 67.2% |
| Sonisphere Festival | 46.39 | 66 | 1.4 | 66.7% |
| Spain national football team | 8.00 | 158 | 19.8 | 65.8% |
| Australia's Next Top Model, Cycle 6 | 14.21 | 140 | 9.9 | 65.7% |
| John Isner | 41.54 | 89 | 2.1 | 65.2% |
| Australia | 44.28 | 90 | 2.0 | 64.4% |
| The Hunchback of Notre Dame (1996 film) | 45.23 | 51 | 1.1 | 62.7% |
| Madonna (entertainer) | 42.23 | 118 | 2.8 | 60.2% |
| This Is It (Michael Jackson concerts) | 45.44 | 62 | 1.4 | 58.1% |
| Big Brother 2010 (UK) | 14.00 | 56 | 4.0 | 55.4% |
| 2010 Formula One season | 25.29 | 55 | 2.2 | 52.7% |
| Toy Story 3 | 12.47 | 146 | 11.7 | 51.4% |
And of the articles that received at least 50 edits during the trial, the 20 articles with the lowest percentage of unreverted anonymous edits:
| Page title | Time under Pending (days) | Total anon edits while under Pending | Anon edits/day | % nonreverted/total anon edits |
| Will Smith | 18.81 | 58 | 3.1 | 6.9% |
| Frank Lampard | 46.46 | 58 | 1.2 | 6.9% |
| Star Wars | 46.27 | 59 | 1.3 | 6.8% |
| Cat | 46.43 | 153 | 3.3 | 6.5% |
| John Terry | 46.44 | 51 | 1.1 | 5.9% |
| Ranbir Kapoor | 49.86 | 53 | 1.1 | 5.7% |
| The Beatles | 44.26 | 129 | 2.9 | 4.7% |
| Miloš Krasić | 46.46 | 67 | 1.4 | 4.5% |
| Baseball | 44.28 | 71 | 1.6 | 4.2% |
| World of Warcraft | 39.65 | 98 | 2.5 | 4.1% |
| Jessica Alba | 43.32 | 50 | 1.2 | 4.0% |
| Down syndrome | 43.55 | 53 | 1.2 | 3.8% |
| Pig | 43.33 | 58 | 1.3 | 3.4% |
| RuneScape | 37.03 | 121 | 3.3 | 3.3% |
| Salman Khan | 46.40 | 93 | 2.0 | 2.2% |
| Atheism | 39.64 | 51 | 1.3 | 2.0% |
| Wonders of the World | 45.78 | 51 | 1.1 | 2.0% |
| William Shakespeare | 22.89 | 64 | 2.8 | 1.6% |
| Chicken | 32.90 | 64 | 1.9 | 0.0% |
| Dinosaur | 43.55 | 60 | 1.4 | 0.0% |
Hopefully these cuts of data will help focus the discussion of Pending Changes. Please feel free to add additional commentary. Howief (talk) 23:52, 20 August 2010 (UTC)
Addition by Chris.urs-o
[edit]I believe that the main goal of Pending Changes is that destructive edits of IPs are not going live. So that the frustrated IPs will do less vandalism. I do not understand why an IP does vandalism, if it does not get visible after all. The articles on the table were current events, the IP behaviour changes too much over time :( Of course, IPs could have tried to falsify the results of the Pending Changes trial to dump the feature :( --Chris.urs-o (talk) 09:57, 13 September 2010 (UTC)
| Page title | Pending Changes after | Total IP edits before Pending | Rev of these IP edits | nonreverted/total IP edits | Pending Changes: nonreverted/total IP edits |
| Spain national football team | "21:16, 11 July 2010" | 143 (10 days) | 15 | 89.5% | 65.8% |
| Toy Story 3 | 14:49, 23 July 2010 | 156 (10 days) | 30 | 80.8% | 51.4% |
| Australia's Next Top Model, Cycle 6 | 21:01, 21 July 2010 | 200 (10 days) | 68 | 66.0% | 65.7% |
| SummerSlam (2010) | 18:11, 21 July 2010 | 167 (10 days) | 45 | 73.1% | 27.2% |
| The Twilight Saga: Eclipse | 21:24, 25 April 2010 | 32 (10 days) | 17 | 46.9% | 42.0% |
| Jason Leopold | 10:57, 28 July 2010 | 7 (10 days) | 2 | 71.4% | 17.9% |
- 2010; nonreverted/total IP edits
- Apple:
- Pending Changes trial after 16:47, 1 July 2010
- 0/0 (Jan), 0/0 (Feb), 0/0 (March), 0/0 (April), 0/0 (May), 0/0 (June)
- Down syndrome:
- Pending Changes trial after 12:50, 22 June 2010
- 0/0 (Jan), 0/0 (Feb), 0/0 (March), 0/0 (April), 0/0 (May), 0/5 (June)
- The Hunchback of Notre Dame (1996 film):
- Pending Changes trial after 20:32, 20 June 2010
- 0/0 (Jan), 0/0 (Feb), 0/0 (March), 0/0 (April), 0/0 (May), 8/13 (June)
- Apple:
- --Chris.urs-o (talk) 18:13, 13 September 2010 (UTC)