本次分享论文：MEDFUZZ: EXPLORING THE ROBUSTNESS OF LARGE LANGUAGE MODELS IN MEDICAL QUESTION ANSWERING
Basic Information
原文作者：Robert Osazuwa Ness, Katie Matton, Hayden Helm, Sheng Zhang, Junaid Bajwa, Carey E. Priebe, Eric Horvitz
作者单位：Microsoft Research, Massachusetts Institute of Technology (MIT), Helivan Research, Johns Hopkins University
Keywords: large language model, medical Q&A, robustness, MedFuzz, benchmarking
Original link: https://arxiv.org/pdf/2406.06573
Open source code: Not available
Thesis Essentials
Introduction:
This paper proposes an adversarial method called MedFuzz to evaluate the robustness of large language models in medical Q&A benchmarks. The study modifies the benchmark question to explore how the model behaves when the baseline assumptions are broken. Experimental results show that the MedFuzz method can effectively reveal the potential problems and limitations of the model in complex practical environments, and provide a new perspective for evaluating its reliability in real clinical applications.
Research Objectives:
The purpose of this article is to evaluate whether the performance of large language models in medical Q&A benchmarks can be generalized to real-world clinical settings. The study examines how LLMs perform when hypotheses are violated by introducing an adversarial method called MedFuzz that attempts to modify the questions in the benchmark without changing the correct answers. The article also explores how this approach can provide insights to assess the robustness of LLMs in more complex real-world environments.
introduction
Currently, large language models perform well in medical Q&A benchmarks, even reaching the human level. However, this high accuracy does not mean that the model will perform equally well in a real-world clinical setting. Benchmarks often rely on specific assumptions that may not hold true in an open clinical setting. In order to explore the performance of LLMs in more complex real-world environments, an adversarial method called MedFuzz is introduced. Drawing on fuzzing methods in software testing and cybersecurity, MedFuzz exposes the failure mode of a system by intentionally "breaking" it by entering unexpected data. This paper demonstrates MedFuzz's method by modifying the questions in the MedQA benchmark, and a successful "attack" can turn an LLM from a correct answer to a false answer without confusing medical experts. Further, this paper also introduces a permutation verification technique to ensure the statistical significance of the attack.
Background:
In recent years, medical Q&A has become a critical task in evaluating large language models. Multiple medical Q&A benchmarks have emerged to statistically assess the performance of LLMs. For example, the MedQA benchmark is based on the United States Medical Licensing Examination (USMLE) and is designed to assess reasoning skills in clinical decision-making. The latest generation of large language models have achieved significant performance improvements on MedQA, such as Med-PaLM 2 and GPT-4, achieving 85.4% and 90.2% accuracy, respectively. While these results are impressive, the assumptions in the benchmark may not be applicable in a real-world clinical setting. Therefore, evaluating the performance of LLMs when these assumptions are violated is critical to understanding their robustness in practical applications.
Research Methods:
The MedFuzz method proposed in this paper uses adversarial LLMs to modify the questions in the benchmark so that these modifications go against the assumptions of the benchmark but do not change the correct answers. Based on the historical output of the target LLM, the adversarial LLM gradually optimizes and modifies the scheme until the target LLM gives an incorrect answer or reaches a predetermined number of iterations. With this approach, it is possible to evaluate the performance of LLMs in more complex real-world environments. Specific steps include selecting assumptions to violate, prompting changes against LLMs, re-evaluating benchmark performance, and identifying interesting case studies.
Experimental analysis
GPT-3.5 and GPT-4 were evaluated using the MedQA benchmark. The counter-LLM is modified several times, and the target LLM answers the revised question. The results show that the accuracy of the benchmark gradually decreases as the number of attacks increases, revealing the vulnerability of the model when assumptions are violated. The specific experimental analysis included multiple attempts to modify the question, and recorded the changes in the target LLM's responses, and finally evaluated the robustness of the LLM in a more complex real-world environment by comparing the performance statistics before and after the benchmark. The case study further demonstrates the inadequacy of LLMs in dealing with bias and complexity.
Findings:
Experimental results show that the use of the MedFuzz method can significantly reduce the performance of LLMs on the MedQA benchmark, suggesting that these models may perform poorly in more complex real-world environments. Specifically, as the number of attacks increases, the accuracy of LLMs gradually decreases, showing their vulnerability when benchmark assumptions are breached. Through case studies, this paper also finds that LLMs are susceptible to interference when dealing with questions with biased and unfair assumptions, resulting in wrong answers.
Conclusion of the dissertation
In this paper, the Robustness of a large language model in medical Q&A benchmarking is evaluated by introducing the MedFuzz method. Research shows that while LLMs perform well in benchmarks, their performance can drop significantly in more complex real-world environments. The MedFuzz method not only reveals the potential problems of LLM when assumptions are violated, but also provides a way to evaluate its robustness in practical applications. Future research can further extend this method and apply it to benchmarking in other fields to fully evaluate the practical application potential of large language models.
Original author: Interpreting agents of the paper
Proofreading: Little Coconut Wind