Access to speedy, reliable broadband matters a lot to our economy, whether we're at work, on our smartphone, or sitting on the couch. But it’s not always easy to measure how well we’re doing as a nation in terms of deploying speedy broadband. One sticking point is the gulf between speeds reported by Internet service providers (ISPs) and users, who sometimes complain that ISPs promise higher speeds than they actually deliver. Although these claims are sometimes legitimate, end user hardware, network configuration, and speed test design can also play a large role. And some data sets don't really offer a true picture of broadband in America.
Recently, the Open Technology Institute at New America published a report that seeks to “map… the gulf between what ISPs report and actual Internet speeds.” The report explains the limits of data the FCC collects from ISPs through its Form 477, which doesn’t always provide a reasonable representation of the throughput that ISPs actually offer in a given geographic area. OTI’s stated aim is to improve on FCC-collected data in its new report, which contains a detailed map of speed test results accumulated by M-Lab, a testing lab that New America spun off earlier in 2019.
The map does a great job visualizing a large, complicated data set. And, in general, the more data is publicly available about the real-world experience of US broadband subscribers, the better. But OTI’s “The United States of Broadband Map” is deficient in ways that undercut its ability to accurately represent the broadband throughput that Americans actually experience.
First, as TPI’s Scott Wallsten explains, one data set that OTI’s map displays is a comparison of the download throughput reported to the FCC to the actual throughput from subscribers using M-Lab’s speed test. The problem, Wallsten notes, is that whereas “ISPs report the maximum bandwidth available in a given geographic area,” many consumers who take the speed test don’t pay for the maximum amount of throughput their ISP is willing to sell them. Wallsten, for instance, could buy a 1 Gbps downstream connection from RCN, his home ISP—but he only pays for a plan offering up to 110 Mbps.
Second, as High Tech Forum’s Richard Bennett explains, M-Lab’s speed test tends to understate broadband throughput, especially for particularly fast connections. The reasons why M-Lab isn’t as accurate as other speed tests are pretty technical, but if you want to understand why M-Lab’s speed test isn’t the best, take the test and compare it against other popular tests.
That’s what I did, and here are the results. I have a Comcast residential broadband subscription in Washington, DC, that offers up to 1 Gbps downstream and up to 35 Mbps upstream. The PC I used has a Realtek RTL8111E Gigabit Ethernet NIC connected via a Cat 6 ethernet cable to an Archer AC1750 Gigabit router, which is turn connected to a Comcast XB6 DOCSIS 3.1 modem.
Here’s the result from M-Lab, using the default M-Lab server in Chrome 75. 540.78 Mb/s downstream.
Not bad, but nowhere near the theoretical 1 Gbps (before overhead) downstream throughput that I pay for.
Then I tried the Speedtest.net Windows 10 app. I first tried the Comcast/Xfinity server in Baltimore, about 35 miles from my home. Because this test involves a server within my ISP’s network, it might not represent my real-world experience. But it’s still a useful data point. 947.47 Mbps downstream. That’s a solid result—very close to the theoretical maximum throughput I should be realizing.
But what about a server outside the Comcast network? Again using the Speedtest.net app, I tried the AT&T server in Washington, DC. It’s a bit slower, at 755.76 Mbps, than my last result. But it’s still much better than my M-Lab result.
Finally, I tried a third local non-Comcast server, Xiber LLC in Washington, DC. Another drop in throughput—655.33 Mbps—but still over 20 percent faster than the M-Lab speed test.
There’s another problem with speed tests, especially when gigabit connections are involved: it’s often unclear what percentage of test results involve devices connected to the Internet via Wi-Fi (typically slower) versus wireline ethernet (typically faster). Although many home wireless access points offer a theoretical throughput of 1 Gbps or more, real-world wireless connectivity often falls far short of what the technology can offer on paper. Interference between Wi-Fi access points is especially likely in urban areas, where a larger portion of broadband subscribers live in multi-dwelling units with lots of wireless access points competing for scarce spectrum. Practical speeds may vary based on a number of other factors including the number of devices connected, the types of ethernet cables, the type of router, the number of other devices on the network, whether they’re using the 2.4 Ghz vs. 5 Ghz connection, etc.
In the end, OTI’s broadband map doesn’t use a data set that’s sufficiently robust from which to draw hard conclusions about broadband policy -- in this case it appears to be building a case to go after ISPs for their advertising practices. Other data sets, such as Ookla’s 2018 Fixed Broadband Speed Test data, probably come closer to portraying the real-world broadband experience of US subscribers, at least on an aggregate basis.