Discovery of newsworthy events in Twitter

Abstract

The new communication paradigm established by Social Media, along with its growing popularity in recent years contributed to attract an increasing interest by several research fields. One such research field is the field of event detection in Social Media. The purpose of this work is to implement a system to detect newsworthy events in Twitter. A similar system proposed in the literature is used as the base of this implementation. For this purpose, a segmentation algorithm implemented using a dynamic programming approach is proposed in order to split the tweets into segments. Wikipedia is then leveraged as an additional factor in order to rank these segments. The top k segments in this ranking are then grouped together according to their similarity using a variant of the Jarvis-Patrick clustering algorithm. The resulting candidate events are filtered using an SVM model trained on annotated data, in order to retain only those related to real-world newsworthy events. The implemented system was tested with three months of data, representing a total of 4,770,636 tweets created in Portugal and mostly written in the Portuguese language. The precision obtained by the system was 76.9 % with a recall of 41.6%.