Project “Evaluating probability and nonprobability online panels”

The project entitled “Evaluating probability and nonprobability online panels” was launched by Centre for Social Informatics (CSI) in September 2021. The project is financed by the Slovenian Research Agency (ARRS) for the period of three years and is classified as a basic project. The leader of the project is prof. Vasja Vehovar, PhD.

Due to technological developments and the increased cost of collecting survey data, social science researchers have gradually transitioned from using traditional surveys (i.e. face-to-face, telephone, and mail surveys) to web surveys. This transition has also highlighted the importance of nonprobability online panels, which contain large pools of respondents who willingly and regularly participate in surveys in exchange for an incentive. Nonprobability online panels, while inexpensive, may not represent the target population well, which can lead to questionable results. To avoid these problems, some government and academic surveys now use more expensive probability online panels because probability sampling supports standard statistical inference.

The scientific literature suggests that probability online panels perform relatively well compared to traditional probability surveys, while nonprobability online panels underperform compared to any probability survey. However, these comparisons typically use survey estimates as the only evaluation criterion and ignore costs, making these evaluations incomplete and misrepresentative of nonprobability online panels. To address this research gap, the project will develop an integrated evaluation approach for comparing probability and nonprobability online panels.

Main objectives of the project:

1. To perform a review and synthesis of past comparisons that will focus on the evaluation criteria used and the survey characteristics (e.g. topic, respondent characteristics and other panel specificities) related to these criteria.

2. To develop a new integrated evaluation approach that simultaneously encompasses the three evaluation criteria when comparing the performances of probability and nonprobability online panels: (i) estimates, (ii) costs and (iii) response quality.

3. To analyze the relationships between the evaluation criteria and the survey characteristics with a mixed method perceptual mapping study on the importance of various cost-error aspects amongst researchers.

4. To apply the integrated evaluation approach in an experimental study that will simultaneously implement the same questions in probability and nonprobability online panels.

The project will provide new insights to the research community and other stakeholders to help decide between probability and nonprobability online panels. Most importantly, it will help address unresolved issues in comprehensive comparisons between alternative survey designs.