← Cases

Analysing with data · Telecom

How much can one home move a national ranking?

By Juancar Mayorga

Every week I help explain a number.

It's a score out of 1,000 that ranks fixed broadband operators on customer experience. Hundreds of physical probes sit in real homes, on real fibre lines. Around the clock, they download, upload, ping, load web pages, open social networks, stream video, play online games, even ask an LLM a question. Millions of measurements become one weekly score per operator, split across 10 services and around 36 KPIs.

That score turns into a ranking. The ranking ends up in board reports, in network decisions and, sometimes, in a marketing claim: "the best fibre experience".

Here's what makes it interesting: the gap between positions is often a few tenths of a point. Out of 1,000.

So every week the real question isn't "who won?". It's "why did each tenth move?". Was it the operator's network? A server on the other side of the internet? One broken probe in one home? One bad day?

Two stories from the last few months show why that question matters more than the score itself.

One mislabelled probe weighed about as much as the gap between first and second

One house, one ranking

While reviewing the data, one probe stood out. Its line was provisioned as 1 Gbps. It delivered around 640 Mbps. Every test, every day, all month. In the data it looked like a customer getting 64% of what they paid for.

The automation flagged it: a probe that was always below its contracted speed. That's exactly what it's good at. It crosses hundreds of probes, every test, every day, without getting tired. But "this probe is slow" is not a diagnosis. Knowing how lines are provisioned is what turned it into one: the line was capped at roughly 640 Mbps but labelled with a 1 Gbps profile. Not a network problem. A cataloguing problem.

Then we measured what it was worth. Removing that probe moved the operator by about 0.20 points per week. In a ranking where positions are separated by a few tenths, that isn't noise. One home, out of hundreds, weighed about as much as the whole gap between operators.

That's the uncomfortable part. A ranking is a claim about millions of customers, built from a sample of homes. If one home can move it that much, the first question isn't who's on top. It's how reliable the number is, and whether the sample really represents the customers it speaks for: the right speed tiers, the right connection type, the right profile on every line.

And it means data cleaning is governance. Which probe is excluded, when, and why has to be recorded and reproducible. Otherwise you're not measuring networks. You're measuring whoever cleaned the data that week.

It's not your network, it's TikTok

A few weeks later, time to first byte for TikTok got worse. For one operator, that would look like a network problem. Someone would open a ticket. Someone would get a difficult phone call.

But it got worse for all six operators. At the same time.

When everyone moves together, it isn't anyone's network. It's the destination: the service, its servers, its CDN. Weeks later it went the other way. A common step of around 12 ms, on the same day, improved TikTok for everyone at once.

A useful benchmark has to separate the common cause (the destination) from the operator's own cause (its network). Without that split, the report blames whoever happens to be lowest that week. And because the margins are so small, a common movement can still reshuffle positions, simply because it hits each operator slightly differently.

Why the human still matters

None of this came from staring at dashboards. Each week there's roughly 3.5 GB of raw test data. Nobody reviews that by hand.

I work with AI as a co-developer. It writes the scripts, crosses the data exhaustively and proposes explanations. I bring the domain knowledge and make the calls. The probe story is that loop in one example: the AI found the anomaly; experience explained it. Without the first, I'd never have spotted it among hundreds of probes. Without the second, we'd have a perfectly computed, wrong conclusion ("this operator under-delivers speed") instead of the right one.

That clinical eye, knowing what looks wrong and why, is what consultants and analysts bring. AI doesn't replace it. It gives it far more to look at.

For each KPI change, we ask one question three ways: if I remove this group (a destination, a day, a set of probes) and recompute the KPI, how much of the weekly change disappears? If removing TikTok's servers makes the change vanish for every operator, it's the destination. If removing three probes makes it vanish for one operator, it's those homes. If removing one day does it, it's that day.

It's a simple test. It stops us comparing milliseconds with percentage points, and it puts a number on every explanation.

The lesson outside telecom

You don't need probes in living rooms for this to apply.

The score is what everyone looks at. The work is explaining each tenth of it.

A big hug and have a great day,
Juancar Mayorga

Working on something similar? I help teams turn messy measurement data into explanations they can act on. Let's talk.

Message me on LinkedIn