I developed two pilot machine-learning models of how articles and their discussion pages develop over time. The first is a hidden Markov model (HMM) trained on weekly activity for 20,000 Wikipedia articles (500 in each of 40 languages) to detect states such as deliberation and high churn or reverting. The second is a Gaussian Mixture Model (GMM) that uses the output of the first to cluster whole article histories, revealing patterns like contested but undiscussed, abandoned and bot-built articles.

This approach to finding signals of content quality and integrity can be applied to other platforms with user-generated content.

One motivation for this research is to support knowledge integrity. My starting premise: community deliberation is key to keeping Wikipedia accurate, and deliberation is a dynamic process. This means we should study how editors build and discuss articles.

This work is part of a larger project to explore and model topologies across Wikimedia readership, communities and content.

Sample model output

Here’s a sample of the HMM output:

Timeline of an article on French Wikipedia, showing states detected by the pilot hidden Markov model. The colored bar at the top shows the states the model detected; the rows below it are weekly observations about the article and the talk page. The model detected a deliberation state (1) associated with a talk page discussion (2), and a possibly contentious state (3) associated with reverted edits and multiple editors adding and removing content (4).

Here’s a sample of the GMM output, showing the percentage of articles in each of the clusters the model identified, across the largest Wikipedias in the sample:

Clusters detected by a Gaussian mixture model, based on the pilot hidden Markov model. The plot shows the percentage of articles in each of the clusters the model identified, across the largest Wikipedias in the sample. Weeks active (1) is the percentage of weeks in each article’s timeline that had at least one human edit on the article or talk page. Both (1) and (2) aggregate all 40 languages in the sample, with equal weight given to each language. Even though half of the clusters (3) were not stable across model variants, the model provides meaningful evidence of structure across languages.

Dataset and code

Code and documentation are on Wikimedia GitLab, and the sample (with user names and IDs redacted) is on Zenodo.

Next steps

Major updates to the model are planned. The next step is to test the assumption that article and talk page edits are coupled via an underlying community process. To do so, I’ll use a metric called talk-to-article nearest-neighbor distance (TA-NND), defined as the distance in time between each talk edit and the nearest article edit. TA-NNDs can be calculated for a talk page with its corresponding article, or for a talk page re-paired with a different article. I’ll compare TA-NNDs with and without re-pairing, within calendar-aligned time windows of multiple lengths. This should separate talk-to-article coupling from trends and seasonality, and show at which time scales coupling occurs. A preregistration with full details is forthcoming.

This will lay the groundwork for an improved state model that accounts for both the coupling and the partial independence of articles and their talk pages. I’m also considering dropping the hidden Markov model’s state transition matrix and instead simply clustering time-windowed states, since state transitions are likely caused by complex social processes that aren’t represented in available data.