The Ethics of Web Crawling and Web Scraping in Cybercrime Research: Navigating Issues of Consent, Privacy, and Other Potential Harms Associated with Automated Data Collection

The abundance of data easily accessible on the internet has made it an appealing domain for conducting social science research. Because of the volume of data available, automated collection software, such as web crawlers and web scrapers, are increasingly being deployed. Ethical guidelines that have developed over the decades, however, are not always directly applicable to online research. As a result, there can be ambiguity as to when and how informed consent needs to be obtained, how privacy can be protected, and what potential harms websites, their owners, and even researchers could experience, and how to address them. In this chapter, we argue that automated data collection practices need not be unethical, provided special care is taken by researchers to acknowledge and explicitly address the complexities surrounding informed consent, privacy, and a myriad of other risks and potential harms (to subjects, websites, and researchers).