China CSR Report: A New Low in Statistical Analysis

November 9, 2011

I’m not sure whether it’s a coincidence that the news about Johnson & Johnson, which I wrote about last night, also had to do with Corporate Social Responsibility (CSR), but today’s news had plenty more of the same. Today it was all about the new report by the China Academy of Social Science (CASS), the “Corporate Social Responsibility Blue Book.”

Disclaimer: I do not have a copy of the report, so my criticisms here reflect reports in the (State) media only. However, the media reports seem to agree on the basics, and I’ll stick to those.

What’s the big takeaway here? That companies in China suck when it comes to CSR. One wonders why we needed an academic report to tell us that. Anyway, here’s how the report was put together:

The report investigated the top 100 State-owned, private and foreign companies in China and scored them on a 100 points scale by comparing them to international CSR indices, the Domestic CSR Initiative and the CSR Evaluation Package of the World’s Top 500. (China Daily)

So far, so good. The benchmarks were based on bribery, quality control, environmental protection and philanthropy.

The suspense is probably killing you. Here are the results: the average score of all companies came out at an abysmal 19.7. This is atrocious, and surprisingly low, even though I certainly didn’t expect high scores.

But here’s where it gets interesting:

The report revealed that SOEs are the best CSR performers among the three groups, with an average score of 31.7, much higher than private businesses at 13.3 and foreign companies at 12.6.

A total of 26 companies scored zero, including foreign giants such as Daimler Chrysler and Coca-Cola. Adidas (China) brought up the rear of list with minus 4 points. (Global Times)

At this point, you might be thinking WTF? Good instinct. Note that this story is prominent in today’s local news, with headlines that scream about how horrible Adidas is (among others). Some of the accompanying commentary has been hilarious, claiming that SOEs are better supervised and are more committed to CSR than foreign multinationals. I’m sure that’s true for some (e.g. I’m sure that SOEs scored very high in the money-given-to-earthquake-relief category), but we are talking about aggregate numbers on a variety of issues here, so I’m at best amused by these statements.

Anyway, as usual, there is a reasonable explanation for this weirdness, and it lies in the bizarre methodology of this report that is unfortunately being taken very seriously by the press.

Basically, the researchers compiled data on these companies, which included requests of their CSR policies/activities:

“We mainly collected CSR data from the companies’ own coverage of their social activities, as well as their websites and annual reports,” said Zhai, an employee of the CASS’ CSR Research Center, told the Global Times.

“To balance their own CSR information, we also collect negative reports about their activities,” Zhai said.

Fair enough, but what happened if CASS didn’t receive any information from a company? Naturally, these companies were not included in the statistics, right? Uh, unfortunately that’s not the case.

Believe it or not, if there was no information on the CSR activities of a company, they scored that firm zero! And if their search of negative reports uncovered anything, then the company ended up with a negative score. That’s how Adidas wound up with a minus-four rating.

Gee, you think that scores of zero and lower might have something to do with those low averages?

I’m not the only critic:

Liu Baocheng, director of the Center for International Business Ethics at the University of International Business and Economics, said that the CASS report was encouraging as it tried to promote CSR in China, but called its rating method questionable.

“Their research oversimplifies quantitative data analysis, is not very representative in its sample selections and its scientific approach to data is questionable. From an academic standpoint, they still need to improve their systematic analysis,” Liu said.

I think Liu is being charitable here. I could end here with a comment on CSR in China, but I think most of you who read this blog are already aware of the challenges. Alternatively, I could say something about lies, damn lies, and statistics, but it’s probably not necessary at this point.

13 thoughts on “China CSR Report: A New Low in Statistical Analysis

  1. S.K. Cheung

    LOL. A study is only as good as its methodology, and this one reeks. So the data and conclusions are basically useless, unless the purpose was to promote an agenda or further a certain narrative. Gee, the China Academy of Social Science wouldn’t be engaged in anything like that now, would they?

    Self-reporting. And the presumption that lack of self-reporting information received equals the company having done nothing for social good. This is great stuff. This sort of thing makes Weibo internet surveys look like precision science.

  2. Robust Engine

    Want to create a fast report? Easy. The search engine does it all for you — and it costs you nothing. Whether the information you get just from the Internet are true or false, it doesn’t matter. Reaching out to companies to verify or get missing information is too much trouble. The general rule of getting first-hand information from the first party doesn’t apply here. We are in an Internet age, and CASS is keeping up with the times — fast, cheap and sensational.

  3. perspectivehere

    Stan, I think this is a good post. Anything that highlights CSR is a good thing. And I applaud your applying critical thinking to the press reports and questioning the premise of the survey. And anyway 98% of all statistics are made up anyway.

    But I have a couple of questions regarding what you call “the bizarre methodology of this report that is unfortunately being taken very seriously by the press” which is that CASS awarded zero points for companies that did not publish any data about CSR. You suggest that the logical step would be for CASS to exclude those companies from the survey entirely.

    But let’s think about that along 2 lines:

    1. Transparency: Companies which publicly report more data about CSR activities (and for example, break out specific reporting for China region) – should they be given a higher CSR rank for reporting on their activities? In my view, a company that publishes detailed reports of its activities (especially if they are reviewed or audited by an independent CSR best-practices body) should be given points for that. The fact that a report is being disclosed would generally mean that a company (a) has given the subject matter of the report some priority, (b) has invested resources to that function, including funding, staff, vendors, etc. and (c) has given management responsibility to achieving results in that area. Yes, many companies will create CSR reporting functions as merely another form of PR but the very fact that a company is institutionalizing a disclosure and reporting function means that it is exposing itself to the risk of being caught out by discrepancies between its disclosures and its practices. So I would think that transparency in-and-of-itself ought to be rewarded. Conversely, a company that reports nothing about its CSR activities or metrics should be penalized in the rankings, on the assumption (perhaps unfounded as you suggest) that non-reporting means that CSR is not a company priority. I think it is in the interest of CSR movement to have more good-quality information about a company’s CSR activities than no information at all.

    2. Incentives. If you’re trying to design a study that would incentivize companies to report on their CSR activities, what would be a bigger incentive? Exclude them from the study entirely (as you seem to suggest)? Or include them, but give an asterisk or N/A for “no data available”? (could be confusing) Or include them, give them a zero ranking, which will incentivize them to invest resources (i.e., hire a consultant or set up a team) to publish reports? This last approach seems to be the most harsh, and could unjustly tar a company which has good CSR activities but does not publicize them.

    Given the balancing of objectives, my personal feeling is, if I were trying to create a survey with the strongest incentive for public disclosure (or the most disincentives for non-disclosure), I would take a hard approach and give non-reporters zero points. The message to the companies is, if you want better rank, then you must put resources into making CSR a priority, including CSR disclosure and transparency.

    I note that Fortune China recently published a CSR 100 Ranking for 2011, and they also give lower rankings to companies that do not publish information, although it is unclear exactly how much of a reduction there is and how it affects the rank:

    See:
    http://www.fortunechina.com/rankings/c/2011-03/15/content_51895.htm

    “The Fortune China CSR 100 Ranking offers a standardized assessment of CSR practices across the leading companies in China. Scoring relies upon corporate public disclosures (annual reports, CSR reports, website disclosures), as well third party sources which we deem to be reliable. Unfortunately, some companies, especially in China, are less accustomed to public reporting. If relevant information is not found, it may result in a poorer ranking of the company.”

    So Fortune CSR also penalizes the lack of transparency, and they do not exclude the company – this is essentially the same approach CASS takes. So CASS’s method may not be as bizarre as you think.

    Note that Fortune China sets out their agenda:

    “By putting the spotlight on company CSR performances, the Fortune China CSR 100 Ranking delivers both ‘carrots and sticks’ to companies in China; giving recognition on the most socially responsible companies and a needed wake-up call to the laggards.”

    It seems to me that the CASS ranking is less carrots and more sticks compared to the Fortune China ranking.

    1. Stan Post author

      OK, that is a reasonable argument. However, either way has problems. If CASS scores zero for non-transparency, perhaps that is an incentive, but I also wonder whether companies that have decent records and just chose not to cooperate with CASS (for whatever reason) are not being treated fairly. The suggestion was that those companies have no CSR programs, which may/may not be true.

      If CASS simply excluded companies that didn’t publish CSR records, then yes, that might leave some of them off the hook. But the only danger here is that some companies are shunted off into the “we don’t know” category. Consumer that care a lot about CSR would perhaps use that information in their purchasing decisions. But at least with the “we don’t know” approach it is not suggestive of something that isn’t necessarily true.

      1. S.k. Cheung

        “Firms were then screened for the presence of a fiscal 2009 CSR report, and the highest ranking firms are then selected for inclusion in the ranking population.”

        “Data relating solely to activities conducted prior to this period were not recognized and those without a CSR report for fiscal year 2009 could not be ranked.”

        The fortune china ranking has some more of the hallmarks of decent and legitimate social science, unlike the cass business. I’ve highlighted the two statements in its methodology that already distinguish the relative value of the two reports. Only companies with csr reports were ranked, so there was no arbitrary zero score for those companies that did not issue 2009 reports, as with cass. In fact, those without 2009 reports were excluded, as fortune stated. They do say that those reports lacking relevant info May result in a poorer ranking, without explicitly explaining how so. Given that they were screening for 180 different metrics, I presume certain individual metrics may be given zero score for non-reporting, but I could be wrong. In principle, that is the same as “zero for no report”, but it is far less punitive and much more defensible when applied in the context of fortune’s methodology than cass. While penalizing for non-transparency may not be bizarre on its face, doing it cass’s way certainly is. Fortune’s methods are certainly not “essentially the same” as cass.

        Fortune’s report might serve as a basis for incentivizing better transparency. The cass report is nothing more than a political sideshow for her political masters.

  4. ChinaCSR

    @Stan.

    What I find so ridiculous about this report is that they claim they looked for publicly available information, yet some of those firms not only have publicly available reports, but those reports are in CHINESE.

    As to why some firms did not respond, here are some thoughts:
    1) If the “researcher” was contacting firms by going to the CSR page and emailing CSR@coke.com, with an email (Chinese or English), then it likely went unanswered as firms are not regularly monitoring these bulk boxes

    2) for one reason or another, the firm felt there was no point to answering the questions. Not all that likely given this is coming from CASS, but perhaps the email wasn’t written in a way that communicated the fact that this was as survey that should be taken seriously.

    @Perspectivehere

    There are groups who regularly rank firms, and some of those firms are given a zero for not having enough/ appropriate content online. However, that does not comprise their entire score. Looking at the firms that they list as zero, and with my background in CSR, I can tell you that there are few firms that invest more into CSR than Coke and Adidas on CSR/ Sustainability. Both have TEAMS of people at the China and Asia level. Both have Chinese reports. Both are seen as leaders in areas that fall under CSR.

    R

    1. Stan Post author

      Thanks. We shouldn’t give some of these guys too much credit. Definitely wouldn’t surprise me if their “research” was less than thorough. My thinking when I first saw this was of all the emails I have received over the years (at different jobs) asking for information that I completely blew off and ignored.

    2. perspectivehere

      ChinaCSR / Stan,

      Thanks for the responses. I’m no expert on CSR reporting methodology, but it seemed to me that the “zero score for non-reporting” is not so much “bizarre” as one extreme in a continuum of how much to penalize non-transparency.

      For example, the following blog-post is illustrative of a CSR analyst who prefers a strong emphasis on transparency:

      “Opaque Transparency”
      http://csr-reporting.blogspot.com/2010/01/opaque-transparency.html

      Commenting on the Global 100 Most Sustainable Corporations in the World list published by The Global Sustainability Research Alliance, led by the Canadian Corporate Knights, the blogger observed:

      “The Transparency Levels of the 10 most sustainable Companies are all lower than the most transparent Companies, ranging between 25% and 81% transparency. The number 2 most sustainable Company has a Transparency score of only 25%. The corollary of this is that the most transparent companies are not rewarded for being transparent in this ranking. This is disappointing, especially given the presence of Ernst Ligteringen, the CEO of the GRI, champion of transparency, on the Global Council of Experts who oversaw the ranking. Whilst I understand that the methodology aggregates a number of factors, and that not everything has the same degree of weighting in calculating an overall sustainability quotient, I fail to understand how Companies can be sustainable without being transparent.

      For me, and perhaps I am a little extreme, or obsessive, or passionate, or stupid, transparency is an essential core element in sustainability which serves to leverage and drive performance in many different ways. Where there is low transparency, there is low trust, low dialogue, low openness to innovation, and low attention to stakeholder needs. When a Company is not transparent about its performance, I wonder how committed that Company is to improving performance. Whilst I do not expect such a ranking to be based on transparency alone, I believe that some degree of overlap would make this ranking more credible.”

      In response, one of the researchers connected with the report wrote:

      “We struggled with how to deal with companies that chose not to report certain key metrics. We had three broad choices:
      A. Apply a zero score (penalize harshly)
      B. Apply a null score (neutral/no penalty)
      C. Apply a null score and include a separate score for transparency, which is weighted in inverse proportion to the extent of ESG/CSR disclosure (ie. the less a company reports, the more the transparency score is worth).

      We chose C, and it is interesting to note that 9 of the top 10 Global 100 are GRI reporters. We look forward to refining the art of presenting a new balance sheet for corporate social/environmental performance–with input from insightful observers like yourself.”

      **************************************
      The question of the weighting of transparency appears to be an old problem; a German sustainability research firm, Oekom, came under criticism in the mid-2000′s for giving low ratings to companies with poor disclosure, even if the companies are otherwise known (or have the public image of being) companies strong performers. This became known as “Oekom’s Dilemma”. See:

      http://www.csrwire.com/csrlive/commentary_detail/1667-Corporate-Sustainability-Ranking-Gets-a-Face-Lift-at-Davos-

      **************************************

      Regarding Adidas and Coca Cola, It’s interesting to note that neither company made the list of top 50 foreign companies in China in the Fortune China 100 list either. Do you think that is because the Fortune China list applied the same criteria as CASS (i.e., penalizing non-transparency?). The Fortune China list was done by InnoCSR Co Ltd.

      *************************************

      For those that want to attempt to read the CASS CSR Bluebook, the results may be accessed here at the link at the foot of the article:

      http://www.cass-csr.org/en/index.php?optionid=148&auto_id=642

      1. S.k. Cheung

        As the researcher concluded, “option c” is the most scientifically valid approach. The scientific principle here is that an absence of information on csr activity is NOT the same as information on/ confirmation of absence of csr activity. A null score with a separate transparency index accurately conveys the former without erroneously inferring or implying the latter.

      2. S.k. Cheung

        Since Adidas and coke weren’t in the fortune rank, based on its methodology, one can presume they didn’t issue 2009 csr reports. The “penalty”, if one could call it that, is being left out of the report. That is much different than cass, who, on the basis of the lack of a csr report, simply concluded that Adidas and coke did nothing on the csr front.

  5. perspectivehere

    Stan / SKC / China CSR

    Thanks for the discussion. I’ve learned a lot from this.

    When I read Stan’s posting, my first thought was “US News and World Report” rankings of Colleges.

    http://en.wikipedia.org/wiki/Rankings_of_universities_in_the_United_States#U.S._News_.26_World_Report_College_and_University_rankings

    These rankings started in 1983 and have come to influence and even dominate priorities of US higher education, and not necessarily in a good way. Schools focus so much on the rankings (for example, some link college president pay to the rankings) that they may allocate more resources to getting a better ranking than on education.

    And yet, the methodologies behind the rankings have been accused by The National Opinion Research Center (a social research organization at the U of Chicago) of “lack[ing] any defensible empirical or theoretical basis”.

    There has been somewhat of a revolt by certain universities, who chose not to participate and thereby receive a lower ranking (the survey accounts for 25% of the overall total points). So there is no way to “exclude” oneself from the results without hurting one’s rank. See:

    http://en.wikipedia.org/wiki/Criticism_of_college_and_university_rankings_(2007_United_States)

    And this poignant plea:

    The Cost of Bucking College Rankings
    http://www.washingtonpost.com/wp-dyn/content/article/2007/03/09/AR2007030901836.html

    And yet despite the criticisms the rankings continue, and appear to be stronger than ever. The strongest argument in favor of producing the rankings is that consumers need information and a way, albeit imperfect, to compare like-vs-like.

    While there is a huge measure of subjectivity inherent in any ranking system that reduces a complex set of incomparables into a single number, it seems like having a system of collecting information so that it is available side by side so comparisons may be done is valuable. It’s just that the reduction of all that information to a simple ranking that makes it easy to compare, also makes it easy to object to if something appears out of whack.

    ********************************

    What I find interesting about the CASS results is the uniformly bad scores they give everyone (even the giants) who wind up doing not-so-bad in the Fortune China survey. CASS sets a higher bar – is that not a good thing (in the abstract)? If Stan is right that he assumes companies in China are not doing a good CSR job, isn’t it a good thing that they are getting bad publicity and beaten up in the press? Isn’t it a good thing that CASS (described in Wikipedia as “the premier and highest academic research organization in the fields of philosophy and social sciences as well as a national center for comprehensive studies in the People’s Republic of China. It was described by Foreign Policy magazine as the top think tank in Asia.”) is getting into the act and coming up with this annual bluebook ranking? Isn’t it a good thing that the government is getting behind CSR and promoting awareness of the issue? Would it be better if the government ignored the issue or made more regulations instead?

    As much as people like Coca Cola’s CSR efforts and are speaking up for how unfairly Coca Cola is treated by the rankings, remember its main product is fizzy sugar water that causes all kinds of health problems and banned by some school districts (like Los Angeles), so their CSR efforts must be weighed against that. They also have been under criticism for their pesticide and water use in India, antiunion activities and other harmful practices in other developing countries. See http://www.guardian.co.uk/media/2006/aug/18/marketingandpr.business

    Do their CSR efforts make up for all the health problems their products cause? Remember that more than anything, Coca Cola is a branding and marketing machine whose sales rely heavily on image, so CSR for Coke could be seen as merely a cynical attempt at burnishing their image. See this:

    “A ban on soda in schools in the nation’s second largest district will likely spark similar movements nationwide, which is why Coca- Cola has threatened to end its sponsorship of the LAUSD’s Academic Decathlon events and why many schools whose extra-curricular activities are funded by revenue from soda sales on campus are worried.”

    http://www.mindbodyhealth.com/sodaban.htm

    I feel so cynical about their CSR efforts when I read that. I note all this as a Coca Cola probable addict, and likely indirect shareholder through mutual fund holdings.

    I like this the most:

    Nuns Who Won’t Stop Nudging
    http://www.nytimes.com/2011/11/13/business/sisters-of-st-francis-the-quiet-shareholder-activists.html?_r=1&ref=general&src=me&pagewanted=all

  6. S.K. Cheung

    I agree that school rankings, being a semi-quantitative attempt at assessing a wide range of qualitative variables, need at the best of times to be taken with a large grain of salt. People do put disproportionate amounts of emphasis in such rankings for reasons that are unclear to me, and that can certainly lead to misplaced priorities of improving a school’s ranking rather than improving its actual educational environment. That said, there is nothing wrong with compiling a ranking, or making use of such a ranking, as long as the consumer understands what the ranking information tells them, and perhaps more importantly, what the ranking information doesn’t tell them.

    I couldn’t access the CASS methodology because the site needed password access. The overall CASS average and group averages seem low in comparison to the Fortune scores. However, that in itself is insufficient to conclude that the CASS criteria were more strict or that they gave uniformly bad scores. The problem is they reported the “average”. I assume this to be the mean, or numerical average. In a data set, the mean is a good measure of a mid-point assuming a normal (or Bell) distribution. But given that 26 out of 300 companies were arbitrarily given zero scores, it seems far more likely that the scores are in a skewed distribution. In such data sets, the median value is a far more applicable assessment of the mid-point of a group, for comparative purposes. I would be interested to see the median scores in the various subsets CASS delineated.

    That said, it is still possible CASS was simply more strict. It would be useful to see their scoring metrics, to see if their criteria were reasonable, or attainable. If everyone gets a bad score, it could mean that everyone is doing poorly and needs to do better, or it could mean that the scoring system is simply unfair. While bad scores on a good test should spur better performance and effort, bad scores on a bad test should simply spur a better test. In that regard, a standard deviation might provide some information (where a small SD means everyone did equally poorly; but a large SD would mean that some companies were in fact able to score very highly, which makes it less likely to be a bad or unfair test).

    I agree that raising awareness of this issue is laudable. Individual scores for individual companies would be useful for direct comparison, and perhaps even inform consumer choices. Grouping SOE’s vs foreign MNC’s, however, seems less geared towards altering behaviour.

    A company’s CSR ranking should reflect its CSR efforts. Coke is a legal product, after all. If the fact that coke may promote obesity and poor health is to be factored in to its CSR score, then China’s tobacco companies will be in for a rough ride, since no matter how bad Coke is for you, smoking is most certainly much worse. While holistically, it may be reasonable for a company’s CSR efforts to be weighed against the potential harm of their product, that is not the purview of a ranking that purports to assess CSR efforts.