More Web Proxy on the site http://driver.im/

review-article

Enhancing systematic reviews: : An in-depth analysis on the impact of active learning parameter combinations for biomedical abstract screening

Authors:

Regina Ofori-Boateng,

Tamy Goretty Trujillo-Escobar,

Magaly Aceves-Martins,

Nirmalie Wiratunga,

Carlos Francisco Moreno-GarciaAuthors Info & Claims

Volume 157, Issue C

https://doi.org/10.1016/j.artmed.2024.102989

Published: 01 November 2024 Publication History

Abstract

Systematic Review (SR) are foundational to influencing policies and decision-making in healthcare and beyond. SRs thoroughly synthesise primary research on a specific topic while maintaining reproducibility and transparency. However, the rigorous nature of SRs introduces two main challenges: significant time involved and the continuously growing literature, resulting in potential data omission, making most SRs become outmoded even before they are published. As a solution, AI techniques have been leveraged to simplify the SR process, especially the abstract screening phase. Active learning (AL) has emerged as a preferred method among these AI techniques, allowing interactive learning through human input. Several AL software have been proposed for abstract screening. Despite its prowess, how the various parameters involved in AL influence the software’s efficacy is still unclear. This research seeks to demystify this by exploring how different AL strategies, such as initial training set, query strategies etc. impact SR automation. Experimental evaluations were conducted on five complex medical SR datasets, and the GLM model was used to interpret the findings statistically. Some AL variables, such as the feature extractor, initial training size, and classifiers, showed notable observations and practical conclusions were drawn within the context of SR and beyond where AL is deployed.

Highlights

•

This study explores optimal Active Learning (AL) combinations for systematic reviews (SRs).

•

Smaller initial training samples improve performance metrics in datasets.

•

TF-IDF consistently outperformed Doc2Vec and S-BERT.

•

Certainty and Uncertainty strategies gave comparative results and effectively interacted with the TF-IDF.

•

The impact of AL variables in SR automation varies according to the specific dataset.

References

[1]

Khan K.S., Kunz R., Kleijnen J., Antes G., Five steps to conducting a systematic review, J R Soc Med 96 (3) (2003) 118–121. [Online]. Available: https://doi.org/10.1258/jrsm.96.3.118.

[2]

Clarke J., What is a systematic review?, Evid-Based Nurs 14 (3) (2011) 64. [Online]. Available: https://doi.org/10.1136/ebn.2011.0049.

[3]

Stevens K., Systematic reviews: The heart of evidence-based practice, AACN Clin Issues 12 (2001) 529–538. [Online]. Available: https://doi.org/10.1097/00044067-200111000-00009.

[4]

Kitchenham B., Brereton O.P., Budgen D., Turner M., Bailey J., Linkman S., Systematic literature reviews in software engineering – A systematic literature review, Inf Softw Technol 51 (1) (2009) 7–15. [Online]. Available: https://doi.org/10.1016/j.infsof.2008.09.009.

Digital Library

[5]

Marshall I.J., Wallace B.C., Toward systematic review automation: A practical guide to using machine learning tools in research synthesis, Syst Rev 8 (1) (2019) 1–10. Avaiable: https://doi.org/10.1186/s13643-019-1074-9.

[6]

Bannach-Brown A., Przybyła P., Thomas J., Rice A.S.C., Ananiadou S., Liao J., Macleod M.R., Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error, Syst Rev 8 (1) (2019) 1–12. [Online]. Available: https://doi.org/10.1186/s13643-019-0942-7.

[7]

Bastian H., Glasziou P., Chalmers I., Seventy-five trials and eleven systematic reviews a day: How will we ever keep up?, PLoS Med 7 (9) (2010) [Online]. Available: https://doi.org/10.1371/journal.pmed.1000326.

[8]

Borah R., Brown A.W., Capers P.L., Kaiser K.A., Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry, BMJ Open 7 (2) (2017) 1–7. [Online]. Available: https://doi.org/10.1136/bmjopen-2016-012545.

[9]

Bornmann L., Mutz R., Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references, J Assoc Inf Sci Technol 66 (11) (2015) 2215–2222. [Online]. Available: https://doi.org/10.1002/asi.23329.

[10]

van de Schoot R., de Bruin J., Schram R., Zahedi P., de Boer J., Weijdema F., Kramer B., Huijts M., Hoogerwerf M., Ferdinands G., Harkema A., Willemsen J., Ma Y., Fang Q., Hindriks S., Tummers L., Oberski D.L., An open source machine learning framework for efficient and transparent systematic reviews, Nat Mach Intell 3 (February) (2021) 125–133. [Online]. Available: https://doi.org/10.1038/s42256-020-00287-7.

[11]

O’Mara-Eves A., Thomas J., McNaught J., Miwa M., Ananiadou S., Using text mining for study identification in systematic reviews: a systematic review of current approaches, Syst Rev 4 (1) (2015) 1–22. [Online]. Available: https://doi.org/10.1186/2046-4053-4-5.

[12]

Blaizot A., Veettil S.K., Saidoung P., Moreno-Garcia C.F., Wiratunga N., Aceves-Martins M., Lai N.M., Chaiyakunapruk N., Using artificial intelligence methods for systematic review in health sciences: A systematic review, Res Synth Methods 13 (3) (2022) 353–362. [Online]. Available: https://doi.org/10.1002/jrsm.1553.

[13]

Cohen A.M., Hersh W.R., Peterson K., Yen P.-Y., Reducing workload in systematic review preparation using automated citation classification, J Am Med Inform Assoc 13 (2) (2006) 206–219. [Online]. Available: https://doi.org/10.1197/jamia.m1929.

[14]

Johnson E.E., O’Keefe H., Sutton A., Marshall C., The systematic review toolbox: keeping up to date with tools to support evidence synthesis, Syst Rev 11 (1) (2022) [Online]. Available: https://doi.org/10.1186/s13643-022-02122-z.

[15]

Settles B., Active learning literature survey, 2010.

[16]

Deploying an interactive machine learning system in an evidence-based practice center: Abstrackr, in: IHI’12 - proceedings of the 2nd ACM SIGHIT international health informatics symposium, 2012, pp. 819–823. Available: https://doi.org/10.1145/2110363.2110464.

[17]

Ouzzani M., Hammady H., Fedorowicz Z., Elmagarmid A., Rayyan-a web and mobile app for systematic reviews, Syst Rev 5 (1) (2016) 1–10. Available: https://doi.org/10.1186/s13643-016-0384-4.

[18]

Cheng S.H., Augustin C., Bethel A., Gill D., Anzaroot S., Brun J., DeWilde B., Minnich R.C., Garside R., Masuda Y.J., Miller D.C., Wilkie D., Wongbusarakum S., McKinnon M.C., Using machine learning to advance synthesis and use of conservation and environmental evidence, Conserv Biol 32 (4) (2018) 762–764. Blackwell Publishing Inc. [Online]. Available: https://doi.org/10.1111/cobi.13117.

[19]

Chai K.E.K., Lines R.L.J., Gucciardi D.F., Ng L., Research screener: a machine learning tool to semi-automate abstract screening for systematic reviews, Syst Rev 10 (1) (2021) 1–13. Systematic Reviews, [Online]. Available: https://doi.org/10.1186/s13643-021-01635-3.

[20]

Howard B.E., Phillips J., Tandon A., Maharana A., Elmore R., Mav D., Sedykh A., Thayer K., Merrick B.A., Walker V., Rooney A., Shah R.R., SWIFT-active screener: Accelerated document screening through active learning and integrated recall estimation, Environ Int 138 (2019) (2020) [Online]. Available: https://doi.org/10.1016/j.envint.2020.105623.

[21]

van Dijk S.H.B., Brusse-Keizer M.G.J., Bucsán C.C., van der Palen J., Doggen C.J.M., Lenferink A., Artificial intelligence in systematic reviews: promising when appropriately used, BMJ Open 13 (7) (2023) BMJ, [Online]. Available: https://doi.org/10.1136/bmjopen-2023-072254.

[22]

van de Schoot R., Sijbrandij M., Depaoli S., Winter S.D., Olff M., van Loey N.E., Bayesian PTSD-trajectory analysis with informed priors based on a systematic literature search and expert elicitation, Multivariate Behav Res 53 (2) (2018) 267–291. Informa UK Limited. [Online]. Available: https://doi.org/10.1080/00273171.2017.1412293.

[23]

Hughes M., Hasta la vista, baby - will machine learning terminate human literature hasta la vista, baby - will machine learning terminate human literature reviews in entrepreneurship?, 2021.

[24]

Ferdinands G., Schram R., de Bruin J., Bagheri A., Oberski D.L., Tummers L., van de Schoot R., Active learning for screening prioritization in systematic reviews - a simulation study, Center for Open Science, 2020, [Online]. Available: https://doi.org/10.31219/osf.io/w6qbg.

[25]

Ferdinands G., Schram R., de Bruin J., Bagheri A., Oberski D.L., Tummers L., Teijema J.J., van de Schoot R., Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the average time to discover relevant records, Syst Rev 12 (1) (2023) [Online]. Available: https://doi.org/10.1186/s13643-023-02257-7.

[26]

Yu Z., Kraft N.A., Menzies T., Finding better active learners for faster literature reviews, Empir Softw Eng 23 (6) (2018) 3161–3186. [Online]. Available: https://doi.org/10.1007/s10664-017-9587-0.

[27]

Almeida H., Meurs M.-J., Kosseim L., Tsang A., Data sampling and supervised learning for HIV literature screening, IEEE Trans Nanosci 15 (4) (2016) 354–361. IEEE. [Online]. Available: https://doi.org/10.1109/bibm.2015.7359733.

[28]

van Dinter R., Tekinerdogan B., Catal C., Automation of systematic literature reviews: A systematic literature review, Inf Softw Technol 136 (2021) [Online]. Available: https://doi.org/10.1016/j.infsof.2021.106589.

Digital Library

[29]

Le Q.V., Mikolov T., Distributed representations of sentences and documents, 2014, arXiv preprint. Available: arXiv:1405.4053.

[30]

Reimers N., Gurevych I., Sentence-BERT: Sentence embeddings using siamese BERT-networks, in: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing, EMNLP-IJCNLP, Association for Computational Linguistics, 2019, [Online]. Available: https://doi.org/10.18653/v1/d19-1410.

[31]

Cortes C., Vapnik V., Saitta L., Support-vector networks editor, Mach Learn 20 (1995) 273–297. Kluwer Academic Publishers.

[32]

Lewis D.D., Naive (Bayes) at forty: The independence assumption in information retrieval, in: ECML, 1998.

[33]

Hans C., Elastic net regression modeling with the orthant normal prior, J Amer Statist Assoc 106 (496) (2011) 1383–1393. [Publisher: Taylor & Francis].

[34]

Breiman L., Random forests, Mach Learn 45 (1) (2001) 5–32. [Online]. Available: https://doi.org/10.1023/A:1010933404324.

Digital Library

[35]

Thrun S.B., Exploration in active learning, in: Handbook of brain and cognitive science, 1995, pp. 381–384. [Online]. Available: http://robots.stanford.edu/papers/thrun.arbib-handbook.ps.gz.

[36]

Lewis D.D., Gale W.A., A sequential algorithm for training text classifiers, in: Proceedings of the 17th annual international ACM SIGIR conference on research and development in information retrieval, Vol. 1994, SIGIR, ISBN 038719889X, 1994, pp. 3–12. [Online]. Available: https://doi.org/10.1007/978-1-4471-2099-5_1.

[37]

Hochreiter S., Schmidhuber J., Long short-term memory, Neural Comput 9 (8) (1997) 1735–1780. [Online]. Available: https://doi.org/10.1162/neco.1997.9.8.1735.

Digital Library

[38]

Przybyła P., Brockmeier A.J., Kontonatsios G., Le Pogam M.A., McNaught J., von Elm E., Nolan K., Ananiadou S., Prioritising references for systematic reviews with RobotAnalyst: A user study, Res Synth Methods 9 (3) (2018) 470–488. [Online]. Available: https://doi.org/10.1002/jrsm.1311.

[39]

EPPI-reviewer 3.5: software for research synthesis, EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, 2007.

[40]

This B. The Rct, Cochrane Crowd, N. H. S. Eed, and Cochrane Rct. Machine learning functionality in EPPI-Reviewer. [s.l.], 1–9.

[41]

Bozada T., Borden J., Workman J., Del Cid M., Malinowski J., Luechtefeld T., Sysrev: A FAIR platform for data curation and systematic evidence review, Front Artif Intell 4 (August) (2021) 1–18. Available: https://doi.org/10.3389/frai.2021.685298.

[42]

Aceves-Martins M., López-Cruz L., García-Botello M., Gutierrez-Gómez Y.Y., Moreno-García C.F., Interventions to treat obesity in mexican children and adolescents: Systematic review and meta-analysis, Nutr Rev 80 (3) (2022) 544–560. Oxford University Press. [Online]. Available:.

[43]

Kwok K.T.T., Nieuwenhuijse D.F., Phan M.V.T., Koopmans M.P.G., Virus metagenomics in farm animals: A systematic review, Viruses 12 (1) (2020) 107. 022. [Online]. Available: https://doi.org/10.3390/v12010107.

[44]

Ofori-Boateng R., Aceves-Martins M., Jayne C., Wiratunga N., Moreno-Garcia C.F., Evaluation of attention-based LSTM and bi-LSTM networks for abstract text classification in systematic literature review automation, Procedia Comput Sci 222 (2023) 114–126. [Online]. Available: https://doi.org/10.1016/j.procs.2023.08.149.

[45]

Howard B.E., Phillips J., Miller K., Tandon A., Mav D., Shah M.R., Holmgren S., Pelch K.E., Walker V., Rooney A.A., Macleod M., Shah R.R., Thayer K., SWIFT-review: a text-mining workbench for systematic review, Syst Rev 5 (1) (2016) [Online]. Available: https://doi.org/10.1186/s13643-016-0263-z.

[46]

Timsina P., Liu J., El-Gayar O., Advanced analytics for the automation of medical systematic reviews, Inf Syst Front 18 (2) (2015) 237–252. [Online]. Available: https://doi.org/10.1007/s10796-015-9589-7.

[47]

Olorisade B.K., Brereton P., Andras P., The use of bibliography enriched features for automatic citation screening, J Biomed Inform 94 (2019) [Online]. Available: https://doi.org/10.1016/j.jbi.2019.103202.

[48]

Singh A.K., Mogalla S., Vectorization of text documents for identifying unifiable news articles, Int J Adv Comput Sci Appl 10 (7) (2019) 305–310. [Online]. Available: https://doi.org/10.14569/ijacsa.2019.0100742.

[49]

Dharma E.M., Gaol F.L., Leslie H., Warnars H.S., Soewito B., The accuracy comparison among word2vec, glove, and fasttext towards convolution neural network (cnn) text classification, J Theor Appl Inf Technol 100 (2) (2022) 31.

[50]

Toshevska M., Stojanovska F., Kalajdjieski J., Comparative analysis of word embeddings for capturing word similarities, 2020, arXiv preprint arXiv:2005.03812.

[51]

Mikolov Tomas, Chen Kai, Corrado Greg, Dean Jeffrey. Efficient estimation of word representations in vector space. In: 1st international conference on learning representations, ICLR 2013 - workshop track proceedings. 2013, p. 1–12.

[52]

Devlin J., Chang M.-W., Lee K., Toutanova K., BERT: Pre-training of deep bidirectional transformers for language understanding, 2019, ArXiv, abs/1810.04805.

[53]

Kleinbaum David G., Modeling strategy guidelines, Logist Regres (1994) 161–189. [Online]. Available: https://doi.org/10.1007/978-1-4757-4108-7_6.

[54]

Sarker Iqbal H., Machine learning: Algorithms, real-world applications and research directions, SN Comput Sci 2 (3) (2021) 1–21. [Online]. Available: https://doi.org/10.1007/s42979-021-00592-x.

Digital Library

[55]

Quinlan J.R., Induction of decision trees, Mach Learn 1 (1) (1986) 81–106. [Online]. Available: https://doi.org/10.1007/bf00116251.

[56]

Rokach L., Maimon O., Decision trees, in: Lecture notes in mathematics, vol. 1928, 2008, pp. 67–86. [Online]. Available: https://doi.org/10.1007/978-3-540-75859-4_5.

[57]

Sarker I.H., A machine learning based robust prediction model for real-life mobile phone data, Internet Things (Netherlands) 5 (2019) 180–193. [Online]. Available: https://doi.org/10.1016/j.iot.2019.01.007.

[58]

Moreno-García C.F., Jayne C., Elyan E., Class-decomposition and augmentation for imbalanced data sentiment analysis, in: International joint conference on neural networks, IJCNN, IEEE, 2021, pp. 1–7.

[59]

Yu Z., Kraft N.A., Menzies T., Finding better active learners for faster literature reviews, Empir Softw Eng 23 (6) (2018) 3161–3186. [Online]. Available: https://doi.org/10.1007/s10664-017-9587-0.

[60]

Fisher R.A., On the interpretation of χ 2 from contingency tables, and the calculation of P, J R Stat Soc 85 (1) (1922) 87. [Online]. Available: https://doi.org/10.2307/2340521.

[61]

Claeskens G., Hjort N.L., Akaike’s information criterion, in: Model selection and model averaging, cambridge series in statistical and probabilistic mathematics, Cambridge University Press, Cambridge, 2008, pp. 22–69. [Online]. Available: https://doi.org/10.1017/CBO9780511790485.003.

[62]

Automated confidence ranked classification of randomized controlled trial articles: An aid to evidence-based medicine, J Am Med Inform Assoc 22 (3) (2015) 707–717. [Author not provided]. [Online]. Available: https://doi.org/10.1093/jamia/ocu025.

Index Terms

Enhancing systematic reviews: An in-depth analysis on the impact of active learning parameter combinations for biomedical abstract screening

Index terms have been assigned to the content through auto-classification.

Recommendations

Low Cost Active Learning Framework for Short Answer Scoring
Knowledge Management and Acquisition for Intelligent Systems
Abstract
Short answer scoring (SAS) that automatically scores learner’ answers based on given rubric. Because SAS uses different rubrics for different prompts, it is necessary to generate training data for each prompt, which is time-consuming and labor-...
Efficient Learning from Few Labeled Examples
ISNN '09: Proceedings of the 6th International Symposium on Neural Networks on Advances in Neural Networks

Active learning and semi-supervised learning are two approaches to alleviate the burden of labeling large amounts of data. In active learning, user is asked to label the most informative examples in the domain. In semi-supervised learning, labeled data ...
Analysis of active semi-supervised learning
SAC '23: Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing

In many real-world applications, labeled instances are costly and infeasible to obtain large training sets. This way, learning strategies that do the most with fewer labels are calling attention, such as semi-supervised learning (SSL) and active learning ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Artificial Intelligence in Medicine

Artificial Intelligence in Medicine Volume 157, Issue C

Nov 2024

404 pages

Issue’s Table of Contents

The Authors.

Publisher

Elsevier Science Publishers Ltd.

United Kingdom

Publication History

Published: 01 November 2024

Author Tags

Qualifiers

Review-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 28 Jan 2025

Other Metrics

View Author Metrics

Citations

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents