Researcher Brown & Wilson has had a meteoric rise, but some tech services firms question its methods.

The tech industry is crawling with market-research firms that size up products, services, and companies. One firm that’s had a meteoric rise in recent years is Brown & Wilson Group of Clearwater, Fla., which began publishing rankings of global outsourcing service providers in 2005 and publishes The Black Book of Outsourcing. When the firm issues its annual lists, many companies rush to publicize their rankings. A flurry of such press releases came out after Brown & Wilson’s June report on the world’s 50 best-managed outsourcing companies.

But while Brown & Wilson has become something of a kingmaker, key players in the outsourcing industry question whether it merits such status. They say its methodology is not consistent with most accepted business research techniques. As a result, companies often rise or fall dramatically from one year to the next—an unusual pattern for a customer satisfaction ranking. “We looked at their methodology, and we couldn’t find any rhyme or reason to it,” says David Parker, vice-president of marketing for IBM’s (IBM) strategic outsourcing business.

Brown & Wilson Group’s principals defend their way of doing business. Scott Wilson and Doug Brown say their survey, which results in multiple rankings, is the world’s most independent and objective rating of outsourcers. “We bring an unbiased view. It comes from the clients,” says Wilson. What they publish matters because companies consider their data when choosing outsourcing providers. A growing slice of the $750 billion tech services industry goes to outsourcing firms.

Among a dozen outsourcing companies contacted by BusinessWeek, three were positive about Brown & Wilson. Marcelo Costa, marketing manager for Miami-based outsourcer Neoris, says, “We feel like it’s a credible firm.” The other companies were skeptical. Peter McLaughlin, manager of public relations for Infosys Technologies (INFY), says the firm peppered Infosys executives with e-mails pressuring them to participate in the survey process. “Beyond the methodology, which we question, we also have concerns about their aggressive promotional behavior,” he says.

Infosys and IBM (IBM) were among the companies whose rankings dropped precipitously this year. But even some companies that scored well were skeptical of Brown & Wilson. Jessie Paul, chief marketing officer for Wipro Technologies, ranked sixth, says, “I don’t use this research to validate [our customer service] because the rankings are so erratic across years.”

Brown & Wilson relies on the Internet to conduct its research. It sends out e-mails to 400,000 people at companies large and small who it believes are involved in outsourcing relationships. The e-mails invite people to go to the Brown & Wilson Web site and complete a survey. About 24,000 people responded this year, Brown and Wilson say. The firm also encourages outsourcers to contact customers and urge them to participate.

TILTED RESULTS?

Claes G. Fornell, a professor at the University of Michigan Ross School of Business who specializes in customer satisfaction surveying, says Brown & Wilson’s methods aren’t sound. First, he says, the firm can’t be sure all the people who respond are qualified. Second, the results could be tilted in favor of companies that urge their customers to participate. “You’d be better off not doing anything than doing a survey like this,” says Fornell.

Brown and Wilson argue that their results are fair and valid because they check the credentials of many of the participants. Concerning the volatility, they point out that all of the companies in the top 50 are outstanding performers and they can move on the list significantly if their raw score changes just a fraction of a point.

Brown & Wilson’s approach is different from most established tech researchers. Firms such as Gartner (IT) and Forrester Research (FORR) employ dozens of analysts who use detailed phone surveys of high-ranking tech purchasing executives as one of their research methods. Gartner’s so-called Magic Quadrant reports typically show only gradual movement in companies’ positions from year to year.

The way Brown and Wilson promote their business has also raised questions about their reliability. Last month, an article showed up on their Web site that appeared to have been published by CIO Magazine. The story compared Brown & Wilson favorably to two of its rivals. But CIO Magazine says it published no such article. Brown and Wilson say they had no hand in putting the story on their site; the article was provided by a service they hired to supply stories about them.

The two also claimed on their Web site to be “adjunct professors” at Syracuse University. But a school spokesman says they hold no such position. When questioned, the pair said they never taught at Syracuse but tried to develop online classes.

There is other information on the Brown & Wilson Web site that is not quite what it appears. It shows a photo of a sleek office building alongside a listing of an office address. But Brown & Wilson doesn’t have such a building. The “office address” they list on their Web site is actually the address of a UPS Store (UPS) where they have a mailbox. They work from home, using shared office space when needed. “It’s not a big operation,” explains Brown. “It’s a database.”