Is ComScore Accurate? Online Audience Measurement Challenges Revealed In Vox Media Deal

It’s 2014. Computers and mobile devices are tracking our every last interaction, yet the seemingly basic task of measuring how many people visit a website is still a flashpoint for debate in media. That debate came to a head once again last week with news that Vox Media, the rapidly growing digital news outlet, received a cash infusion of $46.5 million from the investment firm General Atlantic. If nothing else, the investment marks a vote of confidence in Vox Media’s reach, but just how big is that reach?

That depends on whom you ask. Following news of the Vox deal, some commentators were quick to point out that while Vox was touting an audience of 150 million unique visitors for its portfolio of popular websites, numbers from comScore -- the granddaddy of online analytics companies -- were considerably lower. According to comScore, Vox had a total of 50.7 million unique visitors in October.

Why the discrepancy? After all, how hard can it be to insert some code, track the clicks and tally up the results at the end of the month?

If only online audience measurement were that simple.

“There are many technical, technological and methodological complexities associated with measuring digital media consumption,” Sherrill Mane, senior vice president of research, analytics and measurement at the Interactive Advertising Bureau, said. “One example is the desire of the ad industry to have both standard demographics -- which require rigorous sampling methodologies -- while also wanting granular data that can be best provided by census not sampling methodologies.”

In fact, the only consensus in online analytics is that none of the major firms -- whether it’s comScore, Nielsen, Quantcast or Google -- measure online audiences particularly well.

“All measurement is bad,” Brian Wieser, a senior research analyst at Pivotal Research Group and a former adman, said. “It’s about making a less-bad product wherever possible.”

Which brings us back to Vox Media. In an interview with Fortune magazine last week, Zach Kaplan, a vice president for General Atlantic, was asked about the discrepancy between Vox’s internal analytics and comScore’s much lower numbers. His answer was dismissive, to say the least.

“I don’t want to be harsh, but comScore is increasingly a statistic that doesn’t matter because of what it misses in terms of mobile and global,” he told the magazine.

Not surprisingly, Andrew Lipsman, comScore’s vice president of marketing and insights, disagrees with that assessment. In a phone interview, he said the notion that comScore doesn’t track mobile or global audiences is a misconception. “ComScore was the first measurement company to provide global reporting of online audiences, and the first company to provide unduplicated reporting of desktop and mobile audiences,” Lipsman said.

The “unduplicated” part, Lipsman added, is key to understanding why comScore’s figures are generally lower than internal analytics provided by a publisher. He said the company takes great pains to assure that visitors to a website aren’t counted more than once across multiple devices. For instance, a single user may visit Vox.com while at work, on her smartphone during her commute, and then again on her laptop when she gets home.

ComScore says it uses “proprietary multi-platform audience de-duplication methodology” to control for variables that its competitors do not. “It is important to remember that Web analytics software measures the incidence of unique cookies, which does not accurately reflect the number of people, due to factors such as cookie deletion, non-human traffic, multiple browser usage and multiple device usage,” Lipsman said.

Lipsman conceded, however, that comScore’s global figures don't currently include data from mobile devices, part of the reason its figures are lower than those put out by publishers. He said a plan to measure that data is in the pipeline, but he didn't have a timeline for when that would be rolled out.

Numbers are money

Any company that attempts to measure audiences is going to face criticism. For more on that, look up Nielsen, which has been in the audience measurement game since the days of radio, and has probably been called “irrelevant” by its critics for almost as long.

But Nielsen isn’t irrelevant. Love it or hate it, its ratings data are used to set ad rates for multi-billion-dollar media conglomerates. Advertisers pay $4 million for Super Bowl spots because they have some degree of confidence that the event attracts the audience Nielsen says it attracts. More important than whether Nielsen’s numbers are accurate is having a standardized metric on which advertisers and the media industry can use as a form of currency.

ComScore, too, is used as a measuring tool for ad buyers, and like Nielsen it has no shortage of critics. It’s been accused of extorting money from publishers by charging thousands of dollars for tracking pixels that increase sites’ traffic counts. (It later made those available for free.) And its figures are often balked at by online publishers who prefer the higher tallies calculated by their own internal analytics methods.

Wieser said such disagreements highlight the importance of using third-party data to begin with. “No matter what the publisher says, a buyer of advertising will rarely want to rely on that,” he said. “For example, I have 12 browser windows open. I don’t happen to see the video that’s running in the background right now, and yet that’s going to get counted by someone. So that speaks to one of the reasons why just taking the publisher’s data at face value is rarely what a buyer wants.”

As if the challenges of measuring online audiences wasn't tough enough, it’s getting increasingly complicated as Internet consumption shifts from desktops to mobile devices, whose app-based infrastructure has proven difficult to measure. “Mobile presents additional technical challenges,”Mane said. “Importantly, as digital media consumption grows, we must develop new standards, continue to have oversight and formulate the right research questions so that products delivered to the industry answer our needs best.”

In the meantime, Wieser said, advertisers and media-industry types will have to rely on imperfect metrics, comparing bad apples to bad apples -- at least until better ones come along. “Is the comScore number any good? I’m not optimistic,” he said. “Is it better than the alternative? Quite likely. Should you put much weight in the numbers? Not really.”

Christopher Zara is a senior writer who covers media and culture. Got a news tip? Email me here. Follow me on Twitter @christopherzara.