General information
Code: TAP J5-50159
Period: 1.10.2023 - 30.9.2026
Range on year: 0.45 FTE | 2023
Project leader at FDV: asist.dr. Gregor Čehovin
Research activity: Sociology
Abstract
BACKGROUND Web surveys have become the predominant survey mode, especially when combined with nonprobability sampling. However, this proposal addresses academic, government and other social science research that relies on probability surveys using traditional survey modes (mail, telephone, face-to-face), which are now transitioning to probability-based web surveys (PBWS). In the context of total survey error, web surveys are already known for their strong measurement properties, while noncoverage is a diminishing issue, but nonresponse remains a major problem for PBWS. Recruitment into PBWS relies on traditional survey modes, as there are no email lists for general population surveys, which are the primary domain of this proposal. However, to save costs, recruitment into PBWS is mostly carried out by mail (or email where possible), which significantly reduces response rates, which are often even below 20%. This is very low compared to traditional surveys, where response rates of 70% or 80% were once the standards that have been gradually abandoned (with little justification). Such low response rates increase the danger of nonresponse bias, which means that the estimates from PBWS can differ notably from the true values of the parameters. This presents a serious threat to the quality of empirical social science research. RESEARCH GAP Previous evaluations of PBWS have focused on sample representativeness in terms of control variables (e.g., R-indicator) or measurement equivalence. No systematic question-level evaluations have examined PBWS-specific nonresponse biases that may occur for some topics (e.g., poverty) but not for others (e.g., elections). We lack a thorough insight into the moderators associated with this bias. OBJECTIVES The main objective is to comprehensively elaborate and model the PBWS-specific nonresponse bias. The overarching research question is: Under what circumstances can PBWS be used instead of traditional surveys, given the impact of nonresponse bias? The sub-objectives are: (1) Analyse and model the moderators of PBWS-specific nonresponse bias, particularly response rate but also research topic, respondent characteristics and methodological features; (2) Predict PBWS-specific nonresponse bias for any question; (3) Evaluate PBWS-specific nonresponse bias in the contexts of accuracy and cost. METHOD Objective (1) relies on a systematic literature review and secondary data analysis. PBWS-specific nonresponse bias will be investigated at the question level for selected academic and official surveys in Slovenia and for the CRONOS European Social Survey (ESS). A separate experimental survey will be conducted in parallel with ESS 2025. Objective (2) will draw on the results of the first objective to develop a predictive model using advanced statistical and knowledge discovery methods. In addition, the model will be integrated as a prototype module in an open-source web survey tool. Objective (3) will simultaneously observe the costs and accuracy (which is a measure that integrates bias and variance). IMPACT The results will provide: • A comprehensive elaboration of nonresponse bias in PBWS; • Decision-making guidance on the suitability of PBWS for different objectives in social science research; • A feasibility study for a stand-alone tool to predict bias in PBWS, complementing the Survey Quality Predictor (SQP), which predicts measurement error; • Contributions to the transition of ESS 2025 to PBWS; • Facilitation of the implementation of PBWS in Slovenian official and academic surveys. TEAM The principal investigator has demonstrated excellence by publishing in high-impact journals. The research team has pioneered web survey research, authored highly cited articles, written a seminal textbook on web survey methodology and developed an open-source web survey tool (1KA). The advisory board consists of leading international scholars in the field. This promises the successful achievement of the project’s ambitious objectives.
The phases of the project and their realization
WORK PACKAGES AND ACTIVITIES WP1: Project management 1.1: Project administration and reporting 1.2: Quality assurance and risk management 1.3: Data management and archiving WP2: Research synthesis 2.1: Systematic literature review 2.2: Meta-analysis of selected data WP3: Reanalysis of the available probability-based web surveys (PBWS) microdata 3.1: Identification and retrieval of the suitable PBWS data 3.2: Preparation of the data 3.3: Modelling of the PBWS-specific nonresponse bias WP4: Development of a predictive model 4.1: Identification of the methodological approach 4.2: Development of the predictive model 4.3: Integration into open-source web survey software 4.4: Feasibility study for stand-alone prototype WP5: Experimental study 5.1: Research design 5.2: Preparation of the fieldwork 5.3: Execution of the fieldwork 5.4: Data analysis WP6: integration of costs and errors 6.1: Developing the model and identifying the inputs 6.2: Integration, synthesis and reporting WP7: Dissemination 7.1: Developing dissemination strategy
Research Organisation
https://cris.cobiss.net/ecris/si/en/project/20836
Researchers
https://cris.cobiss.net/ecris/si/en/project/20836
Citations for bibliographic records
https://cris.cobiss.net/ecris/si/en/project/20836
Results / Key findings
The project is ongoing.
Key words
costs, mean square error, nonresponse bias, probability sampling, response rates, social science research, web survey methodology, web surveys
Sustainable Development Goals
SDG9 | Industry, innovation and infrastructure
Back to list of projects