General information
Code: J5-3100
Period: 1.10.2021 - 30.9.2024
Range on year: 0.34 FTE | 2021
Project leader at FDV: prof.dr. Vasja Vehovar
Research activity: Social sciences
Abstract
Due to technological developments and increased survey data collection costs, social science researchers have been gradually moving from the use of traditional surveys (i.e. face-to-face, telephone and mail surveys) to web surveys. This transition has also highlighted the importance of nonprobability online panels, sometimes called ‘access panels’, which contain large pools of respondents who willingly and regularly participate in surveys in exchange for an incentive. Most market and public opinion surveys and a certain section of social science research surveys have already switched to using these panels, which are inexpensive but might not represent the target population well. This can endanger scientific rigour and lead to dubious outcomes. To avoid these problems, some government and academic surveys now use more expensive probability online panels because probability sampling supports standard statistical inference. The literature suggests that probability online panels perform relatively well compared to traditional probability surveys, while nonprobability online panels underperform when compared to any probability survey. However, these comparisons typically consider survey estimates as the only evaluation criteria and ignore costs, which makes these evaluations incomplete and misrepresentative of nonprobability online panels. Thus, there exists a major resear In an environment of limited resources, researchers and decision-makers increasingly face the dilemma of whether the better data quality of probability online panels truly outweighs the substantially higher costs. Recent COVID-19-related issues
have made this issue even more critical, as face-to-face recruitments for probability online panels are now difficult.
The project aims to develop an integrated evaluation approach for comparing probability and nonprobability online panels. The project objectives are as follows:
1. To perform a review and synthesis of past comparisons that will focus on the evaluation criteria used and the survey characteristics (e.g. topic, respondent characteristics and other panel specificities) related to these criteria.
2. To develop a new integrated evaluation approach that simultaneously encompasses the three evaluation criteria when comparing the performances of probability and nonprobability online panels: (i) estimates, (ii) costs and (iii) response quality.
3. To analyse the relationships between the evaluation criteria and the survey characteristics with a mixed method perceptual mapping study on the importance of various cost-error aspects amongst researchers.
4. To apply the integrated evaluation approach in an experimental study that will simultaneously implement the same questions in probability and nonprobability online panels.
The proposed project addresses a very complex and highly relevant problem in contemporary social science research. The project promises a breakthrough in this area by developing an integrated evaluation approach, as this will provide the research community and other stakeholders with new knowledge to facilitate the decision between the two panel types. Crucially, it will also contribute to general unresolved
problems on comprehensive comparisons between alternative survey designs. The proposed research group has conducted pioneering research in web survey methodology and contributed to highly cited papers, a comprehensive monograph on this topic and an open-source web survey tool (1KA). This experience promises that the ambitious goals of the project will be achieved. Moreover, the capabilities of the group are strengthened by its associations with the partnering organisation, which stands at the very frontier of the corresponding research, the international Scientific Advisory Board, which is composed of the most prominent researchers worldwide, and the National Expert Panel, which includes the leading Slovenian experts.ch gap in the comprehensive evaluation of these two panel types.
The phases of the project and their realization
WP1: PROJECT MANAGEMENT; WP 1.1: Project administration and reporting; WP 1.2: Risk management and quality assurance; WP 1.3: Data management and archiving; WP2: CONCEPTUAL DEVELOPMENT; WP 2.1: Systematic literature review; WP 2.2: Conceptual synthesis; WP 2.3: Development of an integrated evaluation approach; WP3: MIXED-METHOD PERCEPTUAL MAPPING STUDY; WP 3.1: Preparation of the research design; WP 3.2: Qualitative study; WP 3.3: Stakeholder survey; WP 3.4: Analysis and synthesis of the results; WP4: PREPARATION OF EXPERIMENTAL STUDY; WP 4.1: Identification and communication with national stakeholders; WP 4.2: Questionnaire preparation and testing; WP 4.3: Administrative and technical preparation; WP 4.4: Pilot study; WP5: EXPERIMENTAL STUDY IMPLEMENTATION; WP 5.1: Probability online panel; WP 5.2: Nonprobability online panel; WP 5.3: Traditional benchmark surveys; WP 5.4: Data processing and preparation; WP6: ANALYSIS AND INTEGRATION OF FINDINGS; WP 6.1: Data analysis; WP 6.2: Integration, synthesis and reporting; WP7: DISSEMINATION; WP 7.1: Development of dissemination strategy; WP 7.2: Pilot online calculator; WP 7.3: Execution of dissemination strategy.
Research Organisation
http://www.sicris.si/public/jqm/prj.aspx?lang=eng&opt=2&subopt=403&hits=1&id=18755&search_term=J5-3100
Researchers
http://www.sicris.si/public/jqm/prj.aspx?lang=eng&opt=2&subopt=402&hits=1&id=18755&search_term=J5-3100
Citations for bibliographic records
http://www.sicris.si/public/jqm/prj.aspx?lang=eng&opt=2&subopt=400&hits=1&id=18755&search_term=J5-3100
Key words
online panels, social science research, survey costs, survey errors, web surveys
Back to list of projects