This article from NYTimes.com has been sent to you by frankthomasft@netscape.net. this sounds interesting: web audience mesurement agencies are not interested having independant auditors as "the internet is so different" /Frank Thomas frankthomasft@netscape.net Ratings Agency Says It Erred in Measuring Web Site Use February 24, 2003 By SAUL HANSELL ComScore Media Metrix, one of the leading companies that measure Web site audiences, has discovered flaws in the methodology that it introduced in October, and it has restated its measurements for the last three months of 2002. Web ratings services have faced questions about accuracy since they began trying to estimate audiences by projecting the behavior of a panel of presumably representative users. Despite their limits, ComScore's ratings, and those of its main competitor, Nielsen/NetRatings, are widely used by advertisers, investors, journalists and the Web sites themselves. The biggest differences in ComScore's ratings, announced last week, come in its estimates of Web use at the workplace, always the most difficult to measure. Big companies in particular do not want employees to install the software that the ratings companies use to track Web site usage. When ComScore adjusted its formulas to account for the underrepresentation at big companies, its audience projections increased, in some cases sharply. For instance, ComScore now estimates that Primedia's sites, mainly those run by about.com, had 56 million users in the United States in December, rather than the 44 million counted under its previous formula, a 25 percent increase. EBay's audience was increased 19 percent, to 54 million. The estimate for Terra Lycos was increased 21 percent, to 51.6 million users. That change moved Terra Lycos to the sixth-most-visited site in December, from seventh under the previous numbers, putting it in front of Google. As with any ranking, being placed at No. 1 is an invitation to gloat. WeatherBug, a fast-growing service of AWS Convergence Technology, has noted in news releases that it is "the No. 1 source for weather information on the Web, according to ComScore Media Metrix." ComScore's restated data, however, shows WeatherBug second to Weather.com, the site run by the Weather channel, owned by Landmark Communications. "There was some bragging rights associated with being No. 1," said Andy Jedynak, the general manager of WeatherBug. "To the extent we have lost those, we are disappointed." WeatherBug's bragging, of course, did not go down so easily at Weather- .com, which was still listed as the No. 1 weather site by Nielsen. "It was annoying," the chief executive of Weather.com, Deborah Wilson, said, " I can't say that it undermined our sales efforts, but to have obviously bad data floating around was a nuisance." The restatements result from the financial collapse last June of Jupiter Media Metrix, a company that combined Web audience measurement with Internet prognostication. As Jupiter was disassembled, it sold the Media Metrix business and its panel of Web users to ComScore, an upstart with a different technology that was created mainly to monitor products that Web users bought online. However, the software program that Media Metrix had used to track Web audiences was not sold to ComScore but to Nielsen, ending a patent dispute between the two companies. ComScore very quickly had to start using its own monitoring technology. It also expanded the size of the Media Metrix panel of users at work, introducing in October what it called Media Metrix 2.0. In response to complaints by its clients, the company has discovered some flaws in its new system. It decided to publish restated data for last fall rather than change its formulas from now on. "We could have made these changes on a forward basis," Peter Daboll, president of ComScore's Media Metrix division, said. "We felt it was the right thing to do to make the changes to provide an accurate trend." Marketers and adverting experts agreed that ComScore's approach ultimately improves the quality of the data available. "It's kind of embarrassing to have to admit this," said Rex Briggs, the principal of Marketing Evolution, an online advertising research firm. "I applaud them for trying to make their measurements better." The impact of a restatement, even of this magnitude, Mr. Briggs added, is less now than it would have been a few years ago, when billion-dollar deals were made on the basis of Media Metrix ratings. "Stock prices are no longer tied to the number of unique visitors you have," Mr. Briggs said. "Now investors have this little idea of being profitable." Furthermore, these audience measures are not used to set Internet advertising rates, as they are in radio and television. Since broadcasters have no way to tell how many people are watching a given program, they must use Nielsen's television ratings to determine how much to charge advertisers. On the Internet, Web sites charge to display an advertisement a certain number of times. Still, some industry executives worry that the restatement will add further doubt about the online medium, which is still trying to establish its credibility with marketers. "They have some explaining to do," said Geoffrey Ramsey, an analyst with eMarketer, a research firm that uses data from ComScore, Nielsen and other sources. This uncertainty and potential undercounting of daytime audiences is especially problematic for news and information sites, which are used mostly by people at work. "Daytime is prime time on the Internet," said Michael Zimbalist, executive director of the Online Publishers Association, a trade group. "The at-work numbers of the ratings companies are notoriously bad." The biggest changes that ComScore made involved how it estimated work audiences. But there were other issues as well. For example, after the acquisition, ComScore changed its method for calculating how much time users spend on each site, in some cases cutting the numbers by far more than half. Some publishers that boasted that their users spent an hour a month on their site, an unusually high amount of time, found the new system giving them credit for only 15 minutes. ComScore discovered that it had been allocating some of the time that users spend on some Web pages to the sites of advertisers rather than the Web publisher. Its revised system changes that. Not surprisingly ComScore and Nielsen each insist that their methods are more statistically rigorous. ComScore says that its sample of people at work has 30,000 people, far more than the 7,000 claimed by Nielsen. Nielsen, by contrast, says its work panel is chosen by placing phone calls to random telephone numbers, an expensive but methodologically superior method. ComScore uses various other methods to recruit its work panel, like sending e-mail messages to random names on lists it buys. It does not help that Nielsen and ComScore are vastly different in their estimates of most sites' audiences. ComScore, for example, now says Yahoo had 107 million users in the United States in December. Nielsen counted only 81 million. Some of that difference is because ComScore measures usage on college campuses and Nielsen does not. But the difference in the estimates of Yahoo's audience, 26 million users, far exceeds the total American dormitory population. ComScore claims that its numbers are far closer to the internal counts generated by Web site operators. But William Pulver, chief executive of NetRatings Inc., the company that publishes the Nielsen/NetRatings measurements, argues that there are good reasons that a measurement service will count fewer people than a site's own computers. "Just because numbers are higher doesn't mean they are more accurate," he said. Many advertisers and publishers, however, say they are tired of such claims and counterclaims and want both rating services to submit to the sort of independent audits that are regularly conducted of the services that measure traditional media. Both ratings services have so far refused, saying the audits are expensive and apply standards that are not realistic for the Internet. "There is no consistency over time or common guidelines applied to either of these services," Mr. Zimbalist said. "They need to be audited just like radio and television ratings." http://www.nytimes.com/2003/02/24/technology/24NET.html?ex=1047086118&ei=1&e... HOW TO ADVERTISE --------------------------------- For information on advertising in e-mail newsletters or other creative advertising opportunities with The New York Times on the Web, please contact onlinesales@nytimes.com or visit our online media kit at http://www.nytimes.com/adinfo For general information about NYTimes.com, write to help@nytimes.com. Copyright 2002 The New York Times Company