실시간 크롤링을 통한 웹 마이닝 최적화 분산처리시스템 설계

Abstract

The media such as newspapers, televisions, radios and magazines played the important role in informing people about the facts that have occurred. Nowadays, the development of information and communication technology and the Internet makes it possible to access to various news more easily, and the influence of the media providing information that can form public opinion is larger and larger. According to the statistics of registration notification for “Internet newspaper" published by the Ministry of Culture, Sports and Tourism, the number of Internet newspapers was 286 in 2005, 2,484 in 2010, and 6,605 in 2015. With the spread of the Internet, the emergence of mobile devices, and the development of networks, the number has increased more than 23 times in 10 years. It has been utilized as a research subject of many researchers as reflecting the use value of users. However, since most article analysis studies do not reflect up-to-date information, it is difficult to predict accurately the direction of public opinions that change over time. Therefore, it is necessary to study the method for quickly analyzing news data generated in real-time with the latest information. This research constructed a real-time analysis system for 150 Internet newspaper companies in Korea as the objects of the study. Through the web mining process of desired Internet news articles, various visualizations were presented through real-time data collection and text mining analysis, and the Hadoop system, a distributed processing technology for rapid processing, was also applied. The opinion mining analysis of Netizen’s comments in real-time collected news and standard dictionaries will be opened to the public, which will be helpful for future researchers. Listening to customers and analyzing consumer’s patterns and needs quickly will help the companies to establish operations and strategies. We expect that this study will contribute to the development of big data processing system that can deal with real-time processing not only simple frequency analysis but also association analysis, cluster analysis, classification analysis, and predictive analysis.