Cognition and statistical-based crowd evaluation framework for ER-in-house crowdsourcing system: Inbound contact center

Entity identification and resolution has been a hot topic in computer science from last three decades. The ever increasing amount of data and data quality issues such as duplicate records pose great challenge to organizations to efficiently and effectively perform their business operations such as customer relationship management, marketing, contact centers management etc. Recently, crowdsourcing technique has been used to improve the accuracy of entity resolution that make use of human intelligence to label the data and make it ready for further processing by entity resolution (ER) algorithms. However, labelling of data by humans is an error prone process that affects the process of entity resolution and eventually overall performance of crowd. Thus controlling the quality of labeling task is an essential for crowdsourcing systems. However, this task becomes more challenging due to unavailability of ground data. In this paper, we address the above mentioned challenge and design and develop framework for evaluating performance of ER-In-house crowdsourcing system using cognition and statistical-based techniques. Our methodology is divided into two phases namely before-hand evaluation and in-process evaluation. In before-hand evaluation a cognitive approach is used to filter out workers with an inappropriate cognitive style for ER-labeling task. To this end, analytic hierarchy process (AHP) is used to classify the existing four primary cogitative styles discussed in the literature either as suitable or not-suitable for labelling task under consideration. To control the quality of work by crowd-workers, we extend and use the statistical approach proposed by Joglekar et al. during second phase i.e. in-process evaluation. To illustrate effectiveness of our approach; we have considered the domain of Inbound Contact Center and using Customer Service Representatives (CSRs) knowledge for ER-labeling task. In the proposed ER-In-house crowdsourcing system CSRs are considered as crowd-workers. Synthetic dataset is used to demonstrate the applicability of the proposed cognition and statistical-based CSRs evaluation approaches.