The Interaction Between Supervised Learning and Crowdsourcing

URL:

Abstract:

In this paper we report insights on combining supervised learning methods and crowdsourcing to annotate the sentiment of a large number of economic news articles. The application entailed using annotations from a group of non-expert annotators on a small subset of articles to train a classifier that would annotate a large corpus of articles. This presents an active learning problem where the challenge is to make the best use of the annotators’ efforts. We discuss the trade- off between determining consensus annotations and maximizing coverage on the training data. We also demonstrate that classifier uncertainty (a popular criterion for example selection in active learning) and disagreement between annotators are not the same thing. This finding provides an important insight into the interplay between supervised learning and crowdsourcing.

• Tagged
• XML
• BibTex