Abstract
A news archive is the core operational tool a media relations team depends on in order to effectively feed a data-hungry organization. An ingrained approach to news archiving in existence is the use of a relational database. As a consequence, integrating search engines that support full-text search is practically impossible due to the strict data schema that is defined in relational database systems. Therefore, there is a need for news archives that support full-text search with relevance ranking of news. In this paper, an approach that supports full-text search is proposed. The process is started by crawling newswire websites for news that are relevant with respect to some predefined keywords and extracting them. Then, they are stored in a data structure known as an inverted-index which supports full-text search, aggregation, and relevance ranking of search results. Search results are ranked and returned to a user in the order of decreasing relevance to the search term. We were able to provide a software solution written in java, the jsoup library for HTML parsing, and an elasticsearch implementation of a search engine. We tested our solution on nine newswires using ten keywords and were able to retrieve a total of 42 relevant news matching seven keywords. The approach proposed in this paper when compared to the manual approach performed better in terms of retrieval speed and accuracy. We conclude that three main components are important in a good digital archive: relevance, extraction, and search. This work is an integration of a good relevance marking technique, an extraction method, and a search engine.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
García, R., Perdrix, F., Gil, R.M.: Ontological infrastructure for a semantic newspaper. In: Proceedings 15th World Wide Web Conference on Semantic Web Annotations for Multimedia (SWAMM), pp. 1–12 (2006)
Crescenzi, V., Mecca, G., Merialdo, P.: RoadRunner: towards automatic data extraction from large web sites. In: VLDB 2001 - Proceedings of 27th International Conference on Very Large Data Bases, pp. 109–118 (2001)
Laender, A.H.F. Ribeiro-Neto, B.A., da Silver, A.S., Teixerira, J.S.: A brief survey of web data extraction tools. ACM SIGMOD Rec. 31(2), p. 84 (2002). https://doi.org/10.1145/565117.565137
Wong, K.-F. et al.: Utilizing microblogs for automatic news highlights extraction. In: Series on Language Processing, Pattern Recognition, and Intelligent Systems. Social Media Content Analysis, pp. 277–296 (2017). https://doi.org/10.1142/9789813223615_0019
Prasad, J., Paepcke, A.: {CoreEx}: content extraction from online news articles. In: CIKM 2008 Proceeding 17th ACM Conference on Information and Knowledge. Management. pp. 1391–1392 (2008). https://dl.acm.org/doi/10.1145/1458082.1458295
Liu, Q., Shao, M., Wu, L., Zhao, G.: Main content extraction from web pages based on node characteristics. J. Comput. Sci. Eng. 11(2), 39–48 (2017). https://doi.org/10.5626/JCSE.2017.11.2.39
Connolly, T., Begg, C.: Database Systems: A practical Approach to Design, Implementation and Management (Sixth Edition). Pearson, Boston (2014)
Dehghani, M., Rothe, S, Alfonseca, P., Fleury, P.: Learning to attend, copy, and generate for session-based query suggestion. In: Proceedings of the 2017 ACM International Conference on Information and Knowledge Management, pp. 1747–1756 (2017). https://doi.org/10.1145/3132847.3133010
Negm, N., ElKafrawy, P., Salem, A.B.: A survey of web information extraction tools. Int. J. Comput. Appl. 43(7), 19–27 (2012). https://doi.org/10.5120/6115-8296
Yarlagadda, M., Gangadhara Rao, K., Srikrishna, A.: Frequent itemset-based feature selection and Rider Moth Search Algorithm for document clustering. J. King Saud Univ. Comput. Inf. Sci. (2019). https://doi.org/10.1016/j.jksuci.2019.09.002
Bun, K.K., Ishizuka, M.: Topic extraction from news archive using TF*PDF algorithm In: Proceedings of the Third International Conference on Web Information Systems Engineering WISE 2002, IEEE, pp. 73–82 (2002). https://doi.org/10.1109/wise.2002.1181645
Zhang, H., Boons, F., Batista-Navarro, R.: Whose story is it anyway? automatic extraction of accounts from news articles. Inf. Process. Manag. 56, 1837–1848 (2019). https://doi.org/10.1016/j.ipm.2019.02.012
Weninger, T., Hsu, W., Han, J.: CETR - content extraction via tag ratios. In: Proceedings of the 19th International Conference on the World Wide Web, WWW 2010. pp. 971–980 (2010). https://doi.org/10.1145/1772690.1772789
Yenicag, A., İnternet, H., İçin, S., İçerik, S-B., Yöntemi C.: A Template-Independent Content Extraction Approach for News (2012)
Karnalim, O.: Improving scalability of java archive search engine through recursion conversion and multithreading. CommIT (Commun. Inf. Technol.) J. 10, 15–26 (2016). https://doi.org/10.21512/commit.v10i1.832
Lin, J., Nogueira, R., Yates, A.: Pretrained Transformers for Text Ranking: BERT and Beyond (2020)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL HLT 2019 – 2019 Conference North American Chapter of the Association for Computational Linguistics Human Language Technologies - Proceeings Conference volume 1, pp. 4171–4186 (2019)
Croft, W.B., Metzler, D., Strohman, T.: Search Engines Information Retrieval in Practice. Pearson, Prentice Hall (2015)
Nakamura, T.A., Calias, P.H., de Castro Reis, D., Lemos, A.P.: An anatomy for neural search engines. Inf. Process. Manag. 480, 339–353 (2019). https://doi.org/10.1016/j.ins.2018.12.041
Guo, J., Fan, Y., Ai, Q., Croft, W.B.: A deep relevance matching model for Ad-hoc retrieval. In: Proceedings of the 25th ACM International Conference on Information and Knowledge Management, pp. 55–64 (2016). https://doi.org/10.1145/2983323.2983769
Liu, B.: Web data mining: exploring hyperlinks, contents, and usage data. In: Carey, M.J. and Ceri, S. Data Centric Systems and Applications. (Second Edition). Springer, Berlin (2015). https://doi.org/10.1007/978-3-642-19460-3
Dixit, B., Kuć, R., Rogoziński, M., Chhajed, S.: Elasticsearch A Complete Guide. Packt Publishing, Sebastopol (2017)
Gormley, C., Tong, Z.: Elasticsearch: The Definitive Guide: A Distributed Real-Time Search and Analytics Engine. O’Reilly Media, Sebastopol (2015)
Brutlag, J.D. Hutchinson, H., Stone, M.: User preference and search engine latency. In: JSM Proceedings, Quality and Productivity Research Conference, American Statistical Association, vol. 12, pp. 1–13 (2008)
Schildt, H.: Java The Complete Reference (Eleventh Edition). McGraw-Hill, New York (2018)
Liang, Y.D.: Introduction to Java Programming Comprehensive Version. Pearson, Prentice Hall (2016)
Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein C..: Introduction to Algorithms. MIT Press (3rd Edition), Cambridge (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Muhammad-Bello, B.L., Lukman, M., Salim, M. (2021). A Novel Approach to News Archiving from Newswires. In: Misra, S., Muhammad-Bello, B. (eds) Information and Communication Technology and Applications. ICTA 2020. Communications in Computer and Information Science, vol 1350. Springer, Cham. https://doi.org/10.1007/978-3-030-69143-1_42
Download citation
DOI: https://doi.org/10.1007/978-3-030-69143-1_42
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-69142-4
Online ISBN: 978-3-030-69143-1
eBook Packages: Computer ScienceComputer Science (R0)