AmCham Business Surveys: Not Exactly Scientific, but Undeniably Useful

April 22, 2013

The American Chamber of Commerce in China regularly publishes surveys of its members, which routinely include criticism of the government/legal system/business climate and complaints about anything from IP infringement to protectionism. So I’m not too surprised to see this pushback from the Ministry of Commerce put out by Xinhua:

An official with the Chinese Ministry of Commerce said Wednesday that a recent survey conducted by the American Chamber of Commerce in China, or AmCham-China, has under-represented foreign businesses in China, making the survey results debatable.

According to the group’s annual business climate survey, which was released on March 29, 28 percent of respondents said they saw China’s investment environment improving, down from 43 percent the previous year. The report also said more than a quarter of respondents said they had experienced data breaches or theft in their China operations.

The MOC official, who works with the ministry’s Department of American and Oceanian Affairs, said AmCham-China’s report was based on answers from 325 respondents among its 1,100 members, but the number of foreign-funded enterprises in China has exceeded 285,000, with over 20,000 funded by American firms.

While no one should be surprised by this, the more interesting issue is whether or not this guy from MOC has a reasonable point. He went on to say that if indeed the results are misleading, then potential foreign investors might get a skewed view of conditions in China, leading to poor investment decisions. He also criticized the media for essentially treating these surveys as gospel, without questioning how representative the results are.

I think these are fair points. Obviously the AmCham surveys are not scientific, and as is the case with surveys like this, I have a feeling that certain folks are over-represented, plus the complainers probably respond in greater numbers than companies that are more satisfied with how things are going.

So yes, it’s possible that potential investors are getting misleading information. Who can we blame for that? AmCham itself? The media?

I don’t think I’m ready to throw AmCham under the bus for the way it handles member surveys. When AmCham publishes these things, the surveys are upfront about methodology, and although I haven’t checked recently, I dimly recall that accompanying press releases make clear how the data was collected.

I’m gonna let AmCham off the hook for this.

How about the media? Well, I’m slightly more open to a bit of criticism, but only a little. When the media reports on a new survey, the articles written generally contain the basics on methodology that you find in the AmCham press release. Anyone reading one of these articles carefully will understand the limitations of the survey.

However, and this is the only area where I’ll let Mr. MOC have his moment in the sun, a cursory glance at these articles, with their screaming headlines, might mislead folks who are skimming their way through the news. Headlines tend to be sensational, and they are usually written in such a way that focuses on one particular hot-button issue.

Shocking, I know. Unfortunately, the editors who write these headlines know what they’re doing. Which of these two headlines would catch your attention?

Seventy-eight percent of U.S. companies in China say they’ve been hacked.

AmCham: 78% of respondents to a survey with a 24% response rate say they’ve been hacked.

Right then. So a small amount of criticism for headline writers, with a fair amount of contributory negligence on behalf of lazy news readers. Aside from that, I’m giving AmCham and those who report on these surveys a pass.

One final point. I read these surveys on a regular basis. Even though they are not scientific and only represent a fraction of the companies over here, they do help to identify problem issues. As a private lawyer, this was useful in giving me a heads-up about matters that I would sooner or later see from my clients. As an in-house guy, it tells me what some other companies are dealing with. I think a lot more people out there benefit from this intelligence than are misled by it.