How much carelessness is too much? Quantifying the impact of careless responding

Abstract

Responses to rating-scale items are often plagued by biases stemming from careless responding. While there is consensus that carelessness can jeopardize the validity of survey measures through a variety of psychometric issues, the question when carelessness becomes a serious problem is less well studied. Existing literature on this question are based on simulations, which are always limited to certain design choices. We leverage statistical robustness theory to derive analytical expressions for the bias caused by careless responding, with a focus on biases in Pearson’s correlation coefficient. We find that carelessness incidence rates of as low as 5% can suffice to cause sign flips in correlational estimates. Furthermore, we show that the impact of carelessness varies by the type of carelessness and that psychometric scales that are designed to be reliable are those most vulnerable to the adverse effects of careless responding.