Quality, confidence, and participation in surveys
Abstract
This paper studies the quality and reliability of surveys, how this reliability is perceived by the population that responds to surveys, and whether this affects their participation. For this, the opinions and perceptions of survey respondents are analysed, to see if they are consistent with the results of the surveys, or diverge from the survey methodology. The explanatory models of survey reliability, obtained through discriminant analysis and binary logistic regression in the 2017 survey of the social perception of surveys conducted by the Centro de Investigaciones Sociológicas (CIS) [Centre for Sociological Research], show that reliability mainly depends on the usefulness attributed to the survey. This usefulness is linked to the representativeness of the sample and the validity of the data it provides, in addition to whether it is considered to be beneficial for people, though not companies. Quality is also found in the agreement to participate in surveys, although the subject of the survey and the availability of time to respond to it are more significant. Both factors are beyond the control of the designer of the survey, unlike whether it meets quality requirements, which may determine whether or not to participate in a survey, depending on the reliability granted to the data provided. Total survey error is still an ideal theoretical benchmark for assessing quality, which should not be limited to sampling and non-response errors.
Keywords
quality, confidence, participation, survey errorsReferences
Álvarez, M. y Font, J. (2007). «Las encuestas a encuesta. Estudio CIS n.º 2.676». Boletín CIS, 3, 1-6.
Bates, N.; Dahlhamer, J. y Singer, E. (2008). «Privacy concerns, too busy or just not interested: Using doorstep concerns to predict survey nonresponse». Journal of Official Statistics, 24 (4), 591-612.
Beatty, P. y Herrmann, D. (2002). «To answer or not to answer: Decision processes related to survey item nonresponse». En: Groves, R. M. et al. Survey nonresponse. Nueva York: John Wiley & Sons, Inc, 71-86.
Becker, R. y Mehlkop, G. (2011). «Effects of prepaid monetary incentives on mail survey response rates and on self-reporting about delinquency». Bulletin de Methodologie Sociologique, 111, 5-25. https://doi.org/10.1177/0759106311408870
Berk, M. L.; Mathiowetz, N. A.; Ward, E. P. y White, A. A. (1987). «The effect of prepaid and promised incentives: Results of a controlled experiment». Journal of Official Statistics, 3 (4), 449-457.
Beullens, K.; Loosveldt, G.; Vandenplas, C. y Stoop, I. (2018). «Response rates in the European Social Survey: Increasing, decreasing, or a matter of fieldwork efforts?”. Survey Methods: Insights from the Field, 1-12. https://doi.org/10.13094/SMIF-2018-00003
Biemer, P. P. (2010) «Total survey error design, implementation, and evaluation». Public Opinion Quarterly, 74 (5), 817-848. https://doi.org/10.1093/poq/nfq058
Biemer, P. P. y Lyberg, L. E. (2003). Introduction to survey quality. Nueva Jersey: John Wiley & Sons.
Bollen, K. A. (1989). Structural equations with latent variables. Nueva York: John Wiley & Sons.
Boulianne, S.; Klofstad, C. A. y Basson, D. (2011). «Sponsor prominence and responses patterns to an online survey». International Journal of Public Opinion Research, 23, 79-87. https://doi.org/10.1093/ijpor/edq026
Boyle, T. (2012). «The effect of lottery scratch tickets and donation offers on response fraction: A study and meta-analysis». Field Methods, 24 (1), 112-132. https://doi.org/10.1177/1525822x11424549
Camacho, C.; Prado, C.; Romero, B. y Valera, S. (2000). «¿Cómo hacen encuestas los encuestadores?». Metodología de Encuestas, 2 (1), 117-133.
Cantril, H. (1944). Gauging public opinion. Princeton University Press.
Cea D’Ancona, M.ª Á. (2017). «Measuring multiple discrimination through survey methodology». Social Science Research, 67, 239-251. https://doi.org/10.1016/j.ssresearch.2017.04.006
Cea D’Ancona, M.ª Á. (2002). Análisis multivariable. Teoría y práctica en la investigación social. Madrid: Síntesis.
Cea D’Ancona, M.ª Á. (2004). Métodos de encuesta. Teoría y práctica, errores y mejora. Madrid: Síntesis.
Chang, L. y Krosnick, J. A. (2009). «National surveys via RDD telephone interviewing versus the internet». Public Opinion Quarterly, 73 (4), 641-678. https://doi.org/10.1093/poq/nfp075
Citro C. F. (2014). «From multiple modes for surveys to multiple data sources for estimates». Survey Methodology, 40 (2),137-161.
Couper, M. P. (2017). «New developments in survey data collection». Annual Review of Sociology, 43, 121-145. https://doi.org/10.1146/annurev-soc-060116-053613
Couper, M. P.; Singer, E.; Conrad, F. G. y Groves, R. M. (2010). «Experimental studies of disclosure risk, disclosure harm, topic sensitivity, and survey participation». Journal of Official Statistics, 26 (2), 287-300.
Curtin, R.; Presser, S. y Singer, E. (2000). «The effects of response rate changes on the index of consumer sentiment». Public Opinion Quartely, 64, 413-428. https://doi.org/10.1086/318638
De Leeuw, E.; Hox, J. y Luiten, A. (2018). «International nonresponse trends across countries and years: An analysis of 36 years of Labour Force Survey data». Survey Insights: Methods from the Field. https://doi.org/10.13094/SMIF-2018-00008
Díaz de Rada, V. y Núñez Villuendas, A. (2008). Estudio de las incidencias en la investigación mediante encuesta. Madrid: CIS. Colección Monografías, n.º 251.
Dillman, D. A. (1978). Mail and telephone surveys: The total design method. Nueva York: John Wiley & Sons.
Dillman, D. A. (2000). Mail and internet surveys: The tailored design method. Nueva York: John Wiley and Sons, Inc.
Dillman, D. A.; Smyth, J. D. y Christian, L. M. (2014). «Reducing people’s reluctance to respond to surveys». En: Dillman, D. A. et al. Internet, Phone, Mail and Mixed-Mode Surveys. Nueva Jersey: John Wiley & Sons, 19-55.
Dykema, J.; Jaques, K.; Cyffka, K.; Assad, N.; Hammers, R. G.; Elver, K.; Malecki, K. C. y Stevenson, J. (2015). «Effects of sequential prepaid incentives and envelope messaging in mail surveys». Public Opinion Quarterly, 79 (4), 906-931. https://doi.org/10.1093/poq/nfv041
Eckman, S. y Koch, A. (2019). «Interviewer involvement in sample selection shapes the relationship between response rates and data quality». Public Opinion Quarterly, 83 (2), 313-337. https://doi.org/10.1093/poq/nfz012
Edwards, M.; Dillman, D. A. y Smyth, J. D. (2014). «An experimental test of the effects of survey sponsorship on internet and mail survey response». Public Opinion Quarterly, 78 (3), 734-750. https://doi.org/10.1093/poq/nfu027
Gillin, J. L. (1915). «The social survey and its further development». Journal of the American Statistical Association, 14, 603-610. https://doi.org/10.2307/2964894
Goyder, J. (1986). «Survey on surveys. Limitations and potentialities». Public Opinion Quarterly, 50, 27-41. https://doi.org/10.1086/268957
Grauenhorst, T.; Blohm, M. y Koch, A. (2016). «Respondent incentives in a national face-to-face survey: Do they affect response quality?». Field Methods, 28 (3), 266-283. https://doi.org/10.1177/1525822x15612710
Groves, R. M. (1989). Survey errors and survey costs. Nueva York: John Wiley
Groves, R. M. (2015). «Improving government, academic and industry data-sharing opportunities». En: Krosnick, J. A. et al. The future of survey research: Challenges and opportunities. Arlington, VA: National Science Foundation, 130-132.
Groves, R. M. y Couper, M. P. (1998). Nonresponse in household interview surveys. Nueva York: John Wiley & Sons.
Groves, R. M. y Peytcheva, E. (2008). «The impact of nonresponse rates on nonresponse bias: A meta-analysis». Public Opinion Quarterly, 72, 167-189. https://doi.org/10.1093/poq/nfn011
Groves, R. M.; Cialdini, R. B. y Couper, M. P. (1992). «Understanding the decision to participate in a survey». Public Opinion Quarterly, 56 (4), 475-495. https://doi.org/10.1086/269338
Groves, R. M.; Fowler Jr., F. J.; Couper, M. P.; Lepkowski, J. M.; Singer, E. y Tourangeau, R. (2009). Survey methodology. Hoboken, NJ: John Wiley and Sons.
Groves, R. M.; Presser, S.; Tourangeau, R.; West, B. T.; Couper, M. P.; Singer, E. y Toppe, C. (2012). «Support for the survey sponsor and nonresponse bias». Public Opinion Quarterly 76 (3), 512-524. https://doi.org/10.1093/poq/nfs034
Groves, R. M.; Singer, E. y Corning, A. (2000). «Leverage-saliency theory of survey participation: Description and an illustration». Public Opinion Quarterly, 64 (3), 299-308. https://doi.org/10.1086/317990
Hansen, M. H.; Hurwitz, W. N. y Madow, W. (1953). Sample survey methods and theory. Nueva York: Wiley.
Heerwegh, D. y Loosveldt, G. (2008). «Face-to-face versus web surveying in a high-internet-coverage population differences in response quality». Public Opinion Quarterly, 72 (5), 836-846. https://doi.org/10.1093/poq/nfn045
Hobson, A. (1916). «The use of the correspondence method in original research». Journal of the American Statistical Association, 15, 210-218. https://doi.org/10.2307/2965113
Hox, J.; Leeuw, E. de y Vorst, H. (1995). «Survey participation as reasoned action: A behavioral paradigm for survey nonresponse?». Bulletin of Sociological Methodology, 48, 52-67. https://doi.org/10.1177/075910639504800109
James, J. M. y Bolstein, R. (1990). «The effect of monetary incentives and follow-up mailings on the response rate and response quality in mail surveys». Public Opinion Quartely, 54, 346-361. https://doi.org/10.1086/269211
Juran, J. y Gryna, F. (1980). Quality planning and analysis. Nueva York: McGraw-Hill.
Kahn, R. L. y Cannell, C. F. (1957). The dynamics of interviewing: Theory, techiques and cases. Nueva York: John Wiley and Sons.
Katz, D. (1942). «Do interviewers bias polls?». Public Opinion Quartely, 6, 248-268. https://doi.org/10.1086/265548
Klecka, W. R. (1980). Discriminant analysis. Beverly Hills: Sage.
Kreuter, F.; Presser, S. y Tourangeau, R. (2008). «Social desirability bias in CATI, IVR, and web surveys: The effects of mode and question sensitivity». Public Opinion Quarterly, 72, 847-865. https://doi.org/10.1093/poq/nfn063
Krosnick, J. A.; Presser, S.; Fealing, K. H.; Ruggles, S. y Vannette, D. L. (2015). The future of survey research: Challenges and opportunities. Arlington, VA: National Science Foundation. http://www.nsf.gov/sbe/AC_Materials/The_Future_of_Survey_Research.pdf
Lepkowski, J. M. y Couper, M. P. (2002). «Nonresponse in the second wave of longitudinal household surveys». En: Groves, R. M. et al. Survey nonresponse. Nueva York: John Wiley and Sons, 121-134.
Loosveldt, G. y Storms, V. (2008). «Measuring public opinions about surveys». International Journal of Public Opinion Research, 20 (1), 74-89. https://doi.org/10.1093/ijpor/edn006
Lyberg, L. (2012). «Survey quality». Survey Methodology, 38 (2), 107-130.
Lyberg, L.; Biemer, P.; Collins, M.; Leeuw, E. de; Dippo, C.; Schwarz, N. y Dennis Trewin, D. (1998). Survey measurement and process control. Nueva York: John Wiley and Sons.
Medway, R. L. y Tourangeau, R. (2015). «Response quality in telephone surveys. Do prepaid cash incentives make a difference?». Public Opinion Quarterly, 79 (2), 524-543. https://doi.org/10.1093/poq/nfv011
Meyer, B. D.; Mok, W. K. C. y Sullivan, J. X. (2015). «Household surveys in crisis». Journal of Economic Perspectives, 29 (4), 199-226. https://doi.org/10.3386/w21399
Muscio, B. (1916). «The influence of the form of a question». The British Journal of Psychology, 8, 351-389. https://doi.org/10.1111/j.2044-8295.1916.tb00138.x
Preisendörfer, P. y Wolter, F. (2014). «Who is telling the truth? A validation study on determinants of response behavior in surveys». Public Opinion Quarterly, 78 (1), 126-146. https://doi.org/10.1093/poq/nft079
Presser, S. (1992). «Survey sponsorship, response rates and response effects». Social Science Quarterly, 73, 699-702.
Rice, S. A. (1929). «Contagious bias in the interview: A methodological note». American Journal of Sociology, 35, 420-423. https://doi.org/10.1086/215055
Rogelberg, S.; Fisher, G.; Maynard, D.; Hakel, M. y Horvath, M. (2001). «Attitudes toward surveys: Development of a measure and its relationship to respondent behavior». Organizational Research Methods, 4 (3), 3-25. https://doi.org/10.1177/109442810141001
Rookey, B. D.; Le, L.; Littlejohn, M. y Dillman, D. A. (2012). «Understanding the resilience of mail-back survey methods: An analysis of 20 years of change in response rates to national park surveys». Social Science Research, 41, 1.404-1.414. https://doi.org/10.1016/j.ssresearch.2012.06.004
Rosen, J.; Murphy, J.; Peytchev, A.; Riley, S. y Lindblad, M. (2011). «The effects of differential interviewer incentives on a field data collection effort». Field Methods, 23, 24-36. https://doi.org/10.1177/1525822x10383390
Schonlau, M. y Couper, M. P. (2017). «Options for conducting web surveys». Statistical Science, 32 (2), 279-292. https://doi.org/10.1214/16-sts597
Schuman, H. y Presser, S. (1981). Questions & answers in attitude surveys. Experiments on question form, wording and context. Nueva York: Academic Press.
Simmons, A. D. y Bobo, L. D. (2015). «Can non-full-probability internet surveys yield useful data?». Sociological Methodology, 45 (1), 357-387. https://doi.org/10.1177/0081175015570096
Singer, E. (2003). «Exploring the meaning of consent: participation in research and beliefs about risks and benefits». Journal of Official Statistics, 19, 273-286.
Singer, E. y Cond, Y. (2013). «The use and effects of incentives in surveys». The Annals of the American Academy of Political and Social Science, 645, 112-141. https://doi.org/10.1177/0002716212458082
Stocké, V. (2006). «Attitudes toward surveys, attitude accessibility and the effect on respondents’ susceptibility to nonresponse». Quality and Quantity, 40, 259-288. https://doi.org/10.1007/s11135-005-6105-z
Stocké, V. y Langfeldt, B. (2004). «Effects of survey experience on respondents’ attitudes towards surveys». Bulletin of Sociological Methodology, 81, 5-32. https://doi.org/10.1177/075910630408100103
Sudman, S. y Bradburn, N. M. (1974). Response effects in surveys: A review and synthesis. Chicago: Aldine Publishing Co.
Vercruyssen, A.; Putte, B. van de y Stoop, I. A. L. (2011). «Are they really too busy for survey participation? The evolution of busyness and busyness claims in Flanders». Journal of Official Statistics, 27 (4), 619-632.
Williams, D. y Brick, M. (2017): «Trends in U.S. face-to-face household survey nonresponse and level of effort». Journal of Survey Statistics and Methodology, 6(2), 186-211. https://doi.org/10.1093/jssam/smx019
Willimack, D. K.; Schuman, H.; Pennell, B. E. y Lepkowski, J. M. (1995). «Effects of a prepaid nonmonetary incentive on response rates and response quality in a face-to-face survey». Public Opinion Quartely, 59, 78-92. https://doi.org/10.1086/269459
Zhang, X.; Kuchinke, L.; Woud, M. L.; Velten, J. y Margraf, J. (2017). «Survey method matters: online/offline questionnaires and face-to-face or telephone interviews differ». Computers in Human Behavior, 71, 172-180. https://doi.org/10.1016/j.chb.2017.02.006
Zillmann, D.; Schmitz, A.; Skopek, J. y Blossfeld, H. P. (2014). «Survey topic and unit nonresponse». Quality & Quantity, 48 (4), 2.069-2.088. https://doi.org/10.1007/s11135-013-9880-y
Published
Downloads
Copyright (c) 2022 M.ª Ángeles Cea D'Ancona
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
