AI Overviews and Wikipedia traffic: the estimates explained

English Wikipedia’s search referrals fell by about 15% in each of three samples. Yet the estimated decline associated with Google’s AI-overview rollout ranges from about 5% to 17%. The difference starts with what happened to the comparison pages.

Mehrzad Khosravi and Hema Yoganarasimhan’s September 2026 working paper compares English Wikipedia articles with their German, French or Japanese counterparts. Its main German and French comparisons estimate 5.45% and 4.82% fewer monthly external-search referrals, conditional on the design’s counterfactual assumptions. The Japanese comparison gives 16.53%, over a shorter period and a different set of articles. These are three estimates from one research project, not three independent replications.

The checks strengthen a conditional traffic-loss interpretation. They also show why “the tests passed” is an inadequate account: different checks ask different questions, and allowing larger departures from parallel trends can make the sensitivity interval include no effect.

First, keep the three comparisons separate

Each panel below sets its own December 2023–April 2024 average to 100. The points show the average monthly referrals per article-language after that period, calculated from the paper’s published means. The bold result below each pair is the paper’s regression estimate; the points are descriptive averages, not regression estimates or confidence intervals.

In the German panel, English referrals fall 15.70%, while German referrals fall 10.84%. Comparing the two proportional changes gives 84.2963 ÷ 89.1585 − 1 = −5.45%. Subtracting the two indices instead gives −4.86 index points. Those operations answer different questions.

The ratios of the published means closely match the reported regression results here. That arithmetic is a useful way to see the contrast; it does not reproduce the fitted model, its covariance or its uncertainty from article-level data. In particular, the Japanese panel changes both the articles and the post-period length. Its larger result cannot be attributed to the choice of control language alone.

What a “referral” and a “rollout effect” mean here

The outcome is Wikimedia’s other-search count: requests arriving at an article from external search engines, aggregated by article, language and month. It combines engines. It does not identify Google alone, an AI overview’s display, a citation inside one, a unique person or the subsequent articles someone reads after landing. Wikimedia describes bot exclusion as an attempt. Clickstream documentation, dump README.

The authors match articles by Wikidata ID. A pair must have existed in November 2023 and have published counts in both languages in each of the five pre-period months. This selects a consistently observable cohort. Means are much higher than medians: for example, the German panel’s English pre-period mean is 4,222.13, while its median is 628. The figure therefore does not describe the experience of a typical article. Sample and summary statistics, pp 5–8.

The model is Poisson pseudo-maximum likelihood, with a fixed effect for each article-language and a common fixed effect for each calendar month. Persistent differences in an article’s baseline traffic and common monthly shocks are absorbed. The English-by-post coefficient estimates a proportional departure associated with the rollout; its percentage transformation is 100 × [exp(coefficient) − 1]. The authors report all three coefficients as significant at p < .001, with two-way clustering by article and month. There are thirteen months in the main panels and eight in the Japanese panel, despite millions of repeated article-language-month records. Those records are not independent people. Model and inference, pp 6–9.

For a causal reading, the untreated English and comparison series need appropriate parallel proportional trends, no anticipation and sufficiently clean exposure and spillover assumptions. A contemporaneous English-specific search or demand shock can survive the fixed effects. We have not rerun the regression, verified its covariance or independently reproduced inference with so few month clusters.

Language is also an imperfect exposure proxy. English Wikipedia reaches readers outside the United States; its referrals include non-Google searches and searches without an overview. Conversely, a language edition is not a country boundary. The paper’s roughly 40% US readership figure is not a joined measure of treatment exposure. Dividing the estimated loss by 40% would invent an exposed-reader effect.

Google announced US default availability on May 14, 2024; the paper names May 15. Both imply a partially exposed May, which the model codes as a whole post month. Rollout restrictions and August expansion and link changes further change the regime during observation. The estimate is about that mixed rollout environment, not the effect of displaying one overview. Exposure dilution can imply attenuation under additional sign and comparability conditions. It does not automatically make every estimate a causal lower bound.

What happens when the counterfactual gets more room to move?

A control can look reassuring in one test and questionable in another. The paper’s main fitted linear pre-period differences are not statistically significant. But its monthly event-study plots contain pre-period pointwise 95% intervals that exclude zero. The French monthly estimates in particular rise toward the omitted April reference. “No significant fitted slope” is therefore narrower than “no pre-period differences.” Table A1 and Figure A3, pp 19–20.

The more useful question is how much departure the negative conclusion survives.

Bounded departures: The authors separate a monthly event-study coefficient into a treatment effect and an untreated English-versus-control difference. April is the reference. Their sensitivity analysis limits each adjacent post-period change in that untreated difference to a multiple, M̄, of the largest absolute adjacent pre-period change. It targets the equally weighted May–December average of event-study treatment effects, not Table 2’s single post coefficient. Equation A4 and Figure A4, p 21.

Within the displayed grid, the 95% sensitivity interval stays below zero through M̄ = 0.40 for German and 0.35 for French. Larger displayed allowances admit zero. Those cutoffs do not mean “40% of the estimated loss is explained away,” or “40% confidence.” They specify a permitted change in the counterfactual at every step.

For scale, suppose—purely hypothetically—the largest pre-period step were 0.02 log units. M̄ = 0.40 would permit a 0.008-log-unit change at each post-period step. Eight same-direction steps could accumulate to 0.064 in month eight and average 0.036 across those eight months. These are illustrative calculations, not observed steps, fitted effects or a recalculated confidence interval. They explain why a small permitted monthly drift can matter across a longer window.

Continued fitted trends: A different check fits a line through the non-reference pre-period event coefficients, anchored at April’s zero, and extends it forward. In the German panel, subtracting that assumed continuation changes the factual post-period mean of the event-study coefficients, −0.0552, to a gap of −0.0510 under ordinary least squares, or −0.0505 under inverse-variance weighting. In the French panel, the continued positive pre-trend makes the gap more negative: a factual post-period event-study mean of −0.0748 becomes −0.1163 or −0.1100. Figures A5–A6 and Table A2, pp 22–23.

These are log-coefficient gaps, not directly observed percentage losses. The French −0.0748 is also a different target from the headline −0.0494. The authors report significant gaps; significance does not establish that the extended line is the true untreated path. The bounded-departure check and the continued-line check are two conditional answers, not interchangeable validations.

The large traffic number and the dollar number take extra steps

The paper’s about 100.27 million monthly foregone referrals is a model-based scaling exercise. Appendix F averages the two main cohorts’ post-period traffic bases and coefficients. These are separately constructed cohorts; their exact intersection is not verified here. The calculation is neither a deduplicated union of the cohorts nor two independent losses to add together. It is not a census of all English Wikipedia articles. Appendix F, pp 31–32.

Using published rounded inputs, the averaged observed post base is about 1.852 billion monthly referrals. Because that base is already after the estimated decline, the calculation reverses the decline: base × [exp(−average coefficient) − 1]. Multiplying the post base by 5.14% uses a different denominator. Rounded coefficient choices give roughly 100.21–100.31 million; that brackets the paper’s number but does not exactly reproduce its unrounded calculation. 1.20 billion annually assumes twelve months of persistence, beyond the main panels’ eight observed post months.

The accompanying $10.82 million–$37.08 million annual advertising scenario is hypothetical. Wikipedia does not run these ads. The range is not observed Wikimedia revenue loss, a confidence interval, money transferred to Google or a measure of social welfare.

How the other evidence changes the claim

The Wikipedia paper supports a particular publisher-and-rollout comparison. Other evidence makes a universal claim about AI search and website loss harder to sustain, but it does not supply a substitute estimate with the same denominator.

Participation can rise: A separate Reddit working paper, v3, compares 70,683 baseline-eligible safe-for-work subreddits with 34,329 not-safe-for-work subreddits, January 2024–July 2025, around the August 15, 2024 expansion. It estimates +12.621 daily comments and +6.667 commenting authors, about 12.0% and 12.4% of safe-for-work pre-period means. Those are participation gains, not referral gains. Its pre-period includes May US exposure; pre-period departures and a later trigger audit do not establish clean historical treatment traces. The finding is contrary evidence to universal loss, not an independent reproduction or numerical rebuttal of Wikipedia’s referral result.

Assignment differs from successful removal: In Wang and colleagues’ US Chrome experiment, v1, the source’s intention-to-treat comparison of removing AI features versus current Google is among retained active-search participants. It estimates +2.7 external clicks per 100 counted searches (95% interval −0.7 to +6.2; 742 participants). The Wikipedia-any-click outcome is instead a binary user outcome: +2.4 percentage points of users under assignment (−2.9 to +7.6). Both assignment estimates are nonsignificant. The click numerator can include multiple external clicks; its denominator excludes vertical searches and counts each AI Mode turn. It is not the probability that a search produces any click. These short-run assignment outcomes do not settle the monthly Wikipedia regime comparison. The compliance-adjusted comparison below uses a different input and a selected population.

A displayed-overview experiment is a different target: Agarwal and Sen’s primary abstract reports 39.8% fewer outbound organic clicks and 34.5% more zero-click searches conditional on appearing AI overviews. This article relies only on that abstract. The study’s complete design, exclusions and uncertainty have not been independently audited here; the abstract is not enough to reconcile that conditional effect with the Wikipedia rollout estimate.

Broader traffic and aggregate clicks can tell different stories: The Wikimedia Foundation’s October 2025 account reports roughly 8% lower human pageviews across languages and routes than corresponding 2024 months, after changed bot classification. It attributes decline to AI and social platforms. That account has no AIO control group; it cannot be added to the selected 2024 search-referral decline. Its text places reclassification in March–August, while its chart caption describes revised pageviews since April. Google’s August 2025 account describes relatively stable aggregate organic clicks, higher clicked-visit quality and redistribution, without an inspectable comparable numerical series. Aggregate stability can coexist with publisher-specific losses, and higher quality among clicked visits can coexist with fewer referrals. Neither account validates or numerically refutes the narrower panel by itself.

Return to the number with its conditions attached

For the main matched cohorts and May–December 2024, the paper estimates about 5% fewer direct external-search referrals to English Wikipedia relative to German and French counterparts. Its Japanese panel points in the same negative direction but changes the sample and window. Calling the observed 15% English decline the rollout effect drops the controls; calling 16.53% a more precise version of 5% drops the panel differences.

The diagnostics preserve negative conclusions under specified bounded departures or fitted continuations. They neither prove parallel trends nor exclude every English-specific concurrent shock. Larger permitted departures can admit zero. The exposure proxy, suppression and selection, unreproduced inference, and mixed rollout remain reasons to qualify the causal claim. The revenue scenario adds further assumptions rather than evidence of actual revenue loss.

When reusing an estimate, carry the outcome, matched cohort, post window, comparator and counterfactual condition with it. The comparison download keeps those fields beside the values; the diagnostic links return to the actual target and allowance. A claim about displayed overviews, all Wikipedia traffic, a typical commercial publisher or advertising returns needs evidence those fields do not provide.