AmCham Business Surveys: Not Exactly Scientific, but Undeniably Useful

April 22, 2013

The American Chamber of Commerce in China regularly publishes surveys of its members, which routinely include criticism of the government/legal system/business climate and complaints about anything from IP infringement to protectionism. So I’m not too surprised to see this pushback from the Ministry of Commerce put out by Xinhua:

An official with the Chinese Ministry of Commerce said Wednesday that a recent survey conducted by the American Chamber of Commerce in China, or AmCham-China, has under-represented foreign businesses in China, making the survey results debatable.

According to the group’s annual business climate survey, which was released on March 29, 28 percent of respondents said they saw China’s investment environment improving, down from 43 percent the previous year. The report also said more than a quarter of respondents said they had experienced data breaches or theft in their China operations.

The MOC official, who works with the ministry’s Department of American and Oceanian Affairs, said AmCham-China’s report was based on answers from 325 respondents among its 1,100 members, but the number of foreign-funded enterprises in China has exceeded 285,000, with over 20,000 funded by American firms.

While no one should be surprised by this, the more interesting issue is whether or not this guy from MOC has a reasonable point. He went on to say that if indeed the results are misleading, then potential foreign investors might get a skewed view of conditions in China, leading to poor investment decisions. He also criticized the media for essentially treating these surveys as gospel, without questioning how representative the results are.

I think these are fair points. Obviously the AmCham surveys are not scientific, and as is the case with surveys like this, I have a feeling that certain folks are over-represented, plus the complainers probably respond in greater numbers than companies that are more satisfied with how things are going.

So yes, it’s possible that potential investors are getting misleading information. Who can we blame for that? AmCham itself? The media?

I don’t think I’m ready to throw AmCham under the bus for the way it handles member surveys. When AmCham publishes these things, the surveys are upfront about methodology, and although I haven’t checked recently, I dimly recall that accompanying press releases make clear how the data was collected.

I’m gonna let AmCham off the hook for this.

How about the media? Well, I’m slightly more open to a bit of criticism, but only a little. When the media reports on a new survey, the articles written generally contain the basics on methodology that you find in the AmCham press release. Anyone reading one of these articles carefully will understand the limitations of the survey.

However, and this is the only area where I’ll let Mr. MOC have his moment in the sun, a cursory glance at these articles, with their screaming headlines, might mislead folks who are skimming their way through the news. Headlines tend to be sensational, and they are usually written in such a way that focuses on one particular hot-button issue.

Shocking, I know. Unfortunately, the editors who write these headlines know what they’re doing. Which of these two headlines would catch your attention?

Seventy-eight percent of U.S. companies in China say they’ve been hacked.

AmCham: 78% of respondents to a survey with a 24% response rate say they’ve been hacked.

Right then. So a small amount of criticism for headline writers, with a fair amount of contributory negligence on behalf of lazy news readers. Aside from that, I’m giving AmCham and those who report on these surveys a pass.

One final point. I read these surveys on a regular basis. Even though they are not scientific and only represent a fraction of the companies over here, they do help to identify problem issues. As a private lawyer, this was useful in giving me a heads-up about matters that I would sooner or later see from my clients. As an in-house guy, it tells me what some other companies are dealing with. I think a lot more people out there benefit from this intelligence than are misled by it.

4 thoughts on “AmCham Business Surveys: Not Exactly Scientific, but Undeniably Useful

  1. colin

    The truth about hacking and cyber espionage was the first thing thrown under the bus by the recent hyping of the issue in the West, and China being painted by special interests as the great bogeyman, It is quite ridiculous. Even more so when there seems to be a real agenda by some to set up the next great digital industrial complex in the US. I’m sure if you survey 100 companies in the US, or anywhere in the world, a similar percentage would report being hacked. What’s important is are the hack attempts actually attributable to China (and even if so, which parties within China)? There is no proof of this, despite some high winded attempts like Mandiant and the NYTimes, The former of which is salivating at billings generating by hyping the false narrative.

  2. Christopher Devonshire-Ellis

    They are what they are Stan, and I should also point out many of Amchams surveys are regional – Amcham South China is seperate from Shanghai and Beijing and Hong Kong. Amcham, along with Eurocham, are the only two chambers that now produce white papers accepted by the Government. And the Givernment have to respond. (I resisted the British Chambers attempts to ditch their own some years ago, and failed, much to their eternal shame and loss of influence).
    I don’t think many American businesses in China take a definitive stragetic view over what these reports contain. Amcham do need to incentivise their members more to participate, that much is obvious. However these are honestly researched, and presented. They provide an anecdote, and statistics are not all that matters. Kudos to Amcham for making a report that the Chinese Government felt they had to respond too. They may not agree, but they are listening, and that’s a good thing. – Chris

  3. bystander

    The AmCham survey data doesn’t seem to jump wildly from survey to survey; instead, it shows a picture of pretty steady trends. American businesses have been reporting a slow but steady increase in the difficulty of doing business in China, steady worsening of the IP situation, steady worsening of China as an investment environment. Nothing radical that would be expected from a survey that suffers from under-sampling.

    325 out of 1100 members is a *HUGE* fraction by an reasonable standard of sampling. 325 out of 20000 is still a very healthy sample size. So I think the argument for sampling error is pretty tough to make here.

    There’s not much doubt that the AmCham survey represents what its members think. Whether they are right to think so is a different matter, but that argument that the survey paints a false picture of their beliefs doesn’t hold much water IMO.

  4. Tim

    I’ve perused AmCham SHanghai’s business report and frustratingly could not find a section on methodology. I find these reports interesting but given how they ask respondents on their China-wide business, I have found them somewhat misleading as the pool of respondents are going to be heavily biased towards Shanghai in an AmCham Shanghai survey.