Using the MSI Evaluation Tool

The MSI Evaluation Tool is available for download and public use. We encourage researchers, MSI practitioners and other organizations to use the Tool to conduct comprehensive evaluations of MSIs, to engage in comparative analyses of different initiatives, or to facilitate dialogue and debate.

The MSI Evaluation Tool includes a detailed assessment methodology, developed following extensive discussions with practitioners, academic review, global consultation and the pilot testing of five MSIs. This Evaluation Methodology includes a standardized terms of reference for evaluating MSIs and step by step guidance on proper use of the MSI Evaluation Tool, including:

  • Selection of the MSIs to evaluate (see our Selection Criteria for those looking to evaluate a range of MSIs)
  • Outreach to the MSI and exploration of whether the terms of reference can be developed
  • Use of publicly available information and material to allow an evaluation of the transparency and accessibility of an MSI
  • Engagement with the MSI staff, governing members, and other core stakeholders directly to understand their perspectives on the MSI
  • Independent expert review of a draft evaluation report
  • Final compilation of an evaluation report, incorporating feedback from stakeholders and identifying areas where such engagement provided context or information that was not available in the initial research of public information
  • Presentation of the report to the MSI’s governing body and membership
ADDITIONAL GUIDANCE ON THE METHODOLOGY

While the Evaluation Methodology provides answers to many questions about the proper use of the MSI Evaluation Tool, we strongly encourage potential evaluators to contact us for guidance related to specific aspects of the evaluation process.

At present, MSI Integrity can offer guidance clarifying the methodology regarding:

Selection criteria used by MSI Integrity to identify and engage a set of MSIs to evaluate

MSI Integrity seeks to evaluate voluntary initiatives that, at a minimum:

  1. Bring together various stakeholders, such as corporations, civil society, governments, and affected communities;
  2. Set standards for some or all members to follow, adhere to, or implement; and
  3. Directly or indirectly address human rights impacts with their standards.

Moreover, in selecting particular MSIs for evaluation, MSI Integrity looks for:

  1. Diversity among the MSIs we evaluate. This diversity contributes to the legitimacy of evaluations by ensuring a balance of focus and increases the breadth of data and knowledge on different MSIs. MSI Integrity focuses on common aspects of MSI design across sectors in order to distill broader lessons for dialogue between different MSIs, industries, and stakeholder groups. The group of MSIs selected for evaluation should display:

(a) Geographical diversity;
(b) Different industry concentrations;
(c) Varying structure, size, age, and level of development; and
(d) A mix of global- and local-level focus.

  1. Availability of information on the MSI. In order to provide detailed evaluation and analysis of an MSI’s design, MSI Integrity requires information on a variety of aspects of the MSI. This information may be publicly available, or the MSI may indicate a willingness to provide it through an engagement process with MSI Integrity. Where possible, this cooperative engagement process will be specified through a Terms of Reference.

Our pilot round of MSI evaluations includes five MSIs that covered a diverse range of industries and geography, stages of development, human rights concerns, and global- or local-levels of focus. The MSI evaluated are the: Common Code for the Coffee Community (4C Association); Extractive Industries Transparency Initiative (EITI); Fair Labor Association (FLA); Global Network Initiative (GNI); and Kimberley Process Certification Scheme (KPCS).

 

Independent expert review of draft evaluation reports
MSI Integrity has found it to be critically important that MSI evaluators select experts with a sound understanding of the industry and the associated human rights concerns, and that experts should not conduct their own subjective evaluation of the MSI. This is because introducing subjective perspectives undermine the objectivity and comparability of reports, which are based on the reviewed indicators contained in the MSI Evaluation Tool. MSI Integrity is able to elaborate on appropriate conflict of interest provisions for expert reviewers, as well as relevant expertise.

 

Communicating evaluation results

MSI Evaluation Tool results are intended to show the extent that an MSI incorporates the design features necessary, but not sufficient, to protect and promote human rights.  MSI Evaluation Tool results are communicated in the number of essential elements met by the MSI in each evaluation category:

  • Scope and Mandate for Human Rights
  • Standards
  • Internal Governance
  • Implementation
  • Development of the MSI
  • Affected Community Involvement
  • Transparency

No overall score for the evaluation an MSI will be calculated or expressed.  Instead, results will be communicated as a simple fraction showing the number of essential elements the MSI meets, as a proportion of the total number of essential elements in that category or subcategory.

For example, the category “Standards” includes three essential elements; therefore, an MSI Evaluation Report will identify the number of essential elements out of three that an MSI incorporates in the form of an “x/3” fraction. There will be no weighting assigned to the Standards essential elements as compared to the other categories. A similar approach would follow for each of the categories listed above.

 

The MSI Evaluation Tool is available for download here. The earlier draft MSI Evaluation Tool (2013) can also be downloaded here.

Contact us for guidance and advisory assistance to ensure that the evaluation benefits from our experience and learning.