Press Release: WAPOR finds no orchestration of data fabrication by IISEPS
Lincoln, Nebraska, 20 December 2016
After its investigation into allegations of data fabrication by the Independent Institute of Socio- Economic and Political Studies (IISEPS), the World Association for Public Opinion Research (WAPOR) has found no evidence of such orchestrated activity.
The allegations were brought to WAPOR’s attention this summer by IISEPS president Dr. Oleg Manaev, after the airing of a program on Belarus Television on August 1, 2016 (http://www.belta.by/society/view/eksperty-o-nisepi-fabrika-podloga-203967-2016). The charges pertained to data fabrication by IISEPS interviewers and missing or inadequate information regarding fieldwork (e.g., interviewers’ identification, routes, and response rates).
WAPOR’s Executive Council charged an ad-hoc committee to investigate the allegations. The four-person committee – comprising WAPOR members Tamas Bodor (University of Wisconsin-Stevens Point), Claire Durand (University of Montreal), Timothy Johnson (University of Illinois at Chicago), and Eugene Kritski (GlobeScan Incorporated) – provided expertise on survey research and polling as well as native fluency in Russian.
To examine the possibility of interviewer fraud, the committee secured from IISEPS a data file of surveys conducted during the period when data fabrication allegedly occurred and which included interviewers’ IDs. A review of the data file revealed that three to four percent of the data were duplicates, a level considered tolerable and not abnormal in face-to-face surveys. In addition, tabulations of key variables showed that a tolerable proportion (five to eight percent) of respondents gave inconsistent answers.
The committee also examined the questionnaire used and concluded that the questions were well-worded and neutral. Finally, the trend results published on IISEPS’s website appear to be supported by the data at hand and do not seem to be outliers compared to what is known or expected in the Belarus context.
The committee also contacted two scholars who were interviewed in the Belarus TV program. These two researchers — Dr. David Rotman, vice president of the Belarusian Sociological Association and director of the Center for Sociological and Political Research of Belarusian State University, and Dr. Christian Haerpfer, research professor of political science at the University of Vienna and president of the World Values Survey Association — were interviewed about survey methodology in general; their appearance on the television program may have led viewers to believe that they were criticizing IISEPS’ work. In their communication with the WAPOR committee, the two said they did not know that their interview would be used in such a context and linked to allegations related to IISEPS’s work.
Finally, the committee asked Dr. Manaev to provide information on the procedures in place to control for data quality. According to Dr. Manaev, IISEPS engages in the following measures of quality control: control of interviewers’ work, i.e., control of filters in the questionnaires; control of the adequate execution of instructions, of route sheets and of closing statements by regional supervisors and network managers; spot-checks (up to 10%) of the conducted interviews, involving telephone calls or second visits by supervisors or third parties; logical control of the aggregated results; and data reweighting. These means did not allow for the detection of these small levels of possible data fabrication (less than 5% of duplicates).
In examining the available information, the committee uncovered no evidence of data fabrication that would have been orchestrated by IISEPS. However, allegations such as those against IISEPS highlight the need to document how opinions polls are conducted. The more sensitive the situation, the more important the documentation becomes. As highlighted in WAPOR’s Code of Ethics, all researchers and pollsters need to safeguard their work and ensure that all operations have been conducted properly and double-checked. They should keep detailed information on the work of supervisors and interviewers, including information on sample members, contacts that are made and their results, refusal rates and response rates, etc.
WAPOR remains committed to the freedom to conduct and publish opinion polls. Criticisms lodged against opinion polling as such and against polling methodology should not prevent the conduct and publication of public opinion polls. On the contrary, it is through open transparency of methods and procedures that polling and public opinion research can provide their greatest service to the public.
For full details, the committee’s report can be accessed at: http://wapor.org/wp-content/uploads/2010/10/Report-of-the-WAPOR-adhoc-committee_IISEPS_final2.pdf.