The list that turned out to be a list of noisy sites
This started as an operational question, not a research one. We keep a record of how each audited site scores before and after our proxy, and one cut of it produces a list nobody wants to have: sites where our own numbers say we made things worse. The list had 76 names on it and a filter for exactly this problem, because each of those sites had come out negative in two separate measurements rather than one.
Re-measuring the top of it, carefully, dissolved most of it. Several of those sites now came out ahead. One of them produced 49 and 89 on the untouched page, in alternation, with both values reproducing across repeated attempts. “Negative twice in a row” does not filter out a site like that; it selects for it. A site whose score swings 40 points lands on the wrong side of zero about half the time, and twice in a row is a coin landing heads twice.
That is a bad way to find out that your measurements move. So we went and measured the measurement.
Three rounds, one sitting
Every slice we measure is measured three times and we keep the median. The rounds run seconds to minutes apart, against the same URL, with nothing changed in between. Any disagreement between them is the instrument, not the page.
Across 1,669 slices carrying two or more recorded rounds:
| Slice | Measurements | Rounds identical | Median gap when they differed | Rounds in different bands |
|---|---|---|---|---|
| Site as it is, mobile | 443 | 46% | 9 | 18% |
| Site as it is, desktop | 440 | 52% | 7 | 13% |
| Through our proxy, mobile | 398 | 52% | 7 | 11% |
| Through our proxy, desktop | 394 | 58% | 7 | 13% |
The last column is the one worth sitting with. A Lighthouse band is the traffic light everybody actually reads: red below 50, orange to 89, green from 90. On the mobile measurements of pages we had not touched, close to one in five produced two rounds that would be reported in two different colours, in the same sitting, on the same URL.
Across all 1,669 slices, a gap of 10 points or more appears on 21% and a gap of 20 or more on 8%. Those shares are computed over every slice, including the ones where all rounds agreed.
Half of the rounds never ran
The “rounds identical” column reads like stability, and it is not. Splitting the slices by how long they took gives the same shape every time:
| Slices | Median time for the whole slice | Finished under 25 s | |
|---|---|---|---|
| All rounds returned the same score | 869 | 18.3 s | 78% |
| Rounds returned different scores | 798 | 58.2 s | 6% |
Three analyses of a real page do not finish in eighteen seconds. One does.
We checked directly. Six calls to the PageSpeed API from our server, three on a plain URL and
three with a unique parameter appended, reading the analysisUTCTimestamp each response carries
— the field the API documents as
the UTC timestamp of this analysis:
| Call | Time to answer | analysisUTCTimestamp | Score |
|---|---|---|---|
| Same URL, 1st | 27.7 s | 10:25.117Z | 41 |
| Same URL, 2nd | 0.4 s | 10:25.117Z | 41 |
| Same URL, 3rd | 0.3 s | 10:25.117Z | 41 |
| Unique URL, 1st | 27.4 s | 10:25.033Z | 42 |
| Unique URL, 2nd | 18.0 s | 10:25.645Z | 41 |
| Unique URL, 3rd | 20.0 s | 10:26.643Z | 41 |
The three requests to the same address are one analysis, returned three times. Nothing about the response says so: the status is 200, the payload is complete, the score is a number. Only the timestamp and the clock give it away.
What moves between two sittings
386 of these audits also carry an archived earlier measurement, days older. Comparing the two gives a different question: not “do two rounds agree” but “does the conclusion survive”.
| Sites | Share | |
|---|---|---|
| Proxy behind the original, this measurement | 100 | 26% |
| Proxy behind the original, previous measurement | 150 | 39% |
| Behind in both | 70 | 18% |
| Ahead in both | 184 | 48% |
| Changed sign between the two | 92 | 24% |
The median before/after difference shifted by 7 points between measurements. And the page we never touch — the site as it is — moved by 5 points or more on half the sample (192 of 386), by 10 or more on 28%, by 20 or more on 8%. One site in five changed Lighthouse band.
Some of that is the sites themselves changing, and we cannot separate the two: a site is free to deploy something between our visits. That is exactly why the three-rounds-in-one-sitting number matters more. It has no such escape hatch.
What this does to a before/after comparison
Both numbers in a comparison carry this. If a single measurement of a page can land anywhere in a 9-point window, a difference of a few points between two of them establishes nothing at all, regardless of which direction it points.
We have published a study of when acceleration helps and when it does not, and it reports 67% of sites improving and 28% getting worse. Those figures were computed from one measurement per site, which is the standard way to do it and is what our own pipeline produced. Read against what is on this page, the 28% is not a group of sites we harmed; it is that group plus everything the instrument put on the wrong side of zero that day. The share of sites where the sign held across two separate measurements is 18%, not 28%. Neither number is wrong, and they answer different questions — but only one of them supports the sentence people will write from it.
That is the general form of the problem, and it does not belong to us. Any before/after claim about page speed, from any tool, is a difference between two draws from distributions this wide, usually with one draw each. Lighthouse’s own documentation is unambiguous about it: the median Lighthouse score of 5 runs is twice as stable as 1 run. The interesting part is not that the advice exists. It is how much a tool that follows the advice can still be fooled.
If you are measuring your own site
Four things, in order of how much they change the answer.
Check that your repeat runs are repeat runs. If you are calling the API in a loop, compare
analysisUTCTimestamp across responses, or watch the clock: a reply in under a second is a
copy of the previous one. In the web interface, the equivalent is that a re-test moments later
tells you nothing new. Change the URL with a parameter your server ignores, or wait. The three
ways to run the test, and where each one gets this wrong, are in our
guide to running a Lighthouse speed test.
Compare bands, not points. A move from 62 to 68 is inside what we see the same page produce twice in a row. A move from 48 to 71 is not. If the decision you are making needs the six points, the measurement you have does not support it.
Measure the thing you can hold still. Metric values carry the same noise as the score, but some of them are far steadier than others: the score is a weighted blend, so it inherits variance from every metric at once. Watching the specific metric you set out to change is a narrower question and a firmer answer.
Be suspicious of your own filtered lists. Ours was two measurements deep and it still filled up with the wrong sites, because a filter that asks for the same outcome twice selects for whatever is widest, not for whatever is worst. If a list is going to drive work, the entry requirement is repeated measurement with the cache defeated, not repetition of the conclusion.
Method and limits
Every audit we run measures four slices — the site as it is and the site through our proxy, each on mobile and desktop — and every slice is measured in three rounds, seconds to minutes apart, of which we keep the median. We kept the individual round scores too, and on 2026-08-24 we read them back: 1,669 slices across 447 audited sites carried two or more recorded rounds. For each slice we took the highest and lowest round and asked whether they agree, by how much, and whether they fall in the same Lighthouse band (0-49, 50-89, 90-100). Separately, 386 of those audits also held an archived earlier measurement of the same site, which lets us compare two sittings days apart rather than two rounds minutes apart. The timings quoted are the wall-clock seconds each slice took, recorded by our worker. The live probe in the last section was six PageSpeed API calls on one URL, three plain and three with a unique query parameter, run from our server on 2026-08-24.
Measured on 24 Aug 2026 across 447 sites. These figures are frozen at that date — we don't quietly restate a published study when the audit base grows.
What this doesn't show
- Not a random sample of the web. These are 447 sites we chose to audit, and the mix leans toward sites someone already suspected were slow. Nothing here says how variable the web is; it says how variable this instrument is on these pages.
- The spread figures are computed on the slices whose rounds actually differed, which is a selected subset. Roughly half of all slices returned identical scores in every round, and the timings say most of those are one analysis reported three times rather than three agreeing measurements. So the honest reading of "8 points at the median" is: among rounds that genuinely ran separately. The true per-round spread over all slices is somewhere between that figure and zero, and our data cannot narrow it further.
- Three rounds is a small number to estimate a spread from. Taking the highest and lowest of three understates the range a site can produce, and the more rounds collapsed into a cache echo, the more it understates.
- The comparison between two sittings mixes two things we cannot separate: the instrument moving and the site changing. A site is free to deploy, swap an ad, or have a slow afternoon between our measurements. We do not store the date of the archived measurement (the archive is written at the moment of the re-measurement, not of the original), so we cannot even bound the gap precisely; it is days rather than weeks, because we re-measure the base on a rolling cycle.
- We measure through the PageSpeed Insights API, which runs on Google's machines, not ours. That is deliberate — it is the same instrument our readers use — but it means we cannot attribute any of this variance to a cause. We can only show its size.
- A performance score is not a linear reading of a page. It is a weighted blend of metric curves, and near the steep part of a curve a difference of a few hundred milliseconds moves the score more than the same difference elsewhere. Two runs 9 points apart are not necessarily two loads that differed by much.
Sources
- Lighthouse Variability — GoogleChrome/lighthouse
The median Lighthouse score of 5 runs is twice as stable as 1 run.
- About PageSpeed Insights — Google for Developers
Variability in performance measurement is introduced via a number of channels with different levels of impact.
- PageSpeed Insights API — runPagespeed reference — Google for Developers
The UTC timestamp of this analysis.