Validation of large language models for detecting pathologic complete response in breast cancer using population-based pathology reports

Simple item page
dc.contributor.authorCheligeer, Ken
dc.contributor.authorWu, Guosong
dc.contributor.authorLaws, Alison
dc.contributor.authorQuan, May L.
dc.contributor.authorLi, Andrea
dc.contributor.authorBrisson, Anne-Marie
dc.contributor.authorXie, Jason
dc.contributor.authorXu, Yuan
dc.date.accessioned2024-10-06T00:04:23Z
dc.date.available2024-10-06T00:04:23Z
dc.date.issued2024-10-03
dc.date.updated2024-10-06T00:04:22Z
dc.description.abstractAbstract Aims The primary goal of this study is to evaluate the capabilities of Large Language Models (LLMs) in understanding and processing complex medical documentation. We chose to focus on the identification of pathologic complete response (pCR) in narrative pathology reports. This approach aims to contribute to the advancement of comprehensive reporting, health research, and public health surveillance, thereby enhancing patient care and breast cancer management strategies. Methods The study utilized two analytical pipelines, developed with open-source LLMs within the healthcare system’s computing environment. First, we extracted embeddings from pathology reports using 15 different transformer-based models and then employed logistic regression on these embeddings to classify the presence or absence of pCR. Secondly, we fine-tuned the Generative Pre-trained Transformer-2 (GPT-2) model by attaching a simple feed-forward neural network (FFNN) layer to improve the detection performance of pCR from pathology reports. Results In a cohort of 351 female breast cancer patients who underwent neoadjuvant chemotherapy (NAC) and subsequent surgery between 2010 and 2017 in Calgary, the optimized method displayed a sensitivity of 95.3% (95%CI: 84.0–100.0%), a positive predictive value of 90.9% (95%CI: 76.5–100.0%), and an F1 score of 93.0% (95%CI: 83.7–100.0%). The results, achieved through diverse LLM integration, surpassed traditional machine learning models, underscoring the potential of LLMs in clinical pathology information extraction. Conclusions The study successfully demonstrates the efficacy of LLMs in interpreting and processing digital pathology data, particularly for determining pCR in breast cancer patients post-NAC. The superior performance of LLM-based pipelines over traditional models highlights their significant potential in extracting and analyzing key clinical data from narrative reports. While promising, these findings highlight the need for future external validation to confirm the reliability and broader applicability of these methods.
dc.identifier.citationBMC Medical Informatics and Decision Making. 2024 Oct 03;24(1):283
dc.identifier.urihttps://doi.org/10.1186/s12911-024-02677-y
dc.identifier.urihttps://hdl.handle.net/1880/119953
dc.language.rfc3066en
dc.rights.holderThe Author(s)
dc.titleValidation of large language models for detecting pathologic complete response in breast cancer using population-based pathology reports
dc.typeJournal Article
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
12911_2024_Article_2677.pdf
Size:
1.79 MB
Format:
Adobe Portable Document Format
Download
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.71 KB
Format:
Item-specific license agreed upon to submission
Description:
Download
Collections
Open Access Publications