Original title: Shen Tao et al.: Key points of risk control of AI in the field of pharmaceutical supervision - interpretation of the list of typical application scenarios of artificial intelligence in drug supervision
Artificial intelligence (AI), one of today's most disruptive technologies, is changing the world and the way people live at an unprecedented rate. With the continuous development of AI technology, as far as the pharmaceutical field is concerned, AI can obtain new important trends by analyzing a large amount of data generated in the whole process of the industry, bringing convenience and efficiency to the pharmaceutical and health industry, and promoting the vigorous development of the industry.
1. A brief history of artificial intelligence
In August 1956, John McCarthy proposed the term Artificial Intelligence at the Dartmouth Conference, which was unanimously approved by the participants, marking the official birth of artificial intelligence. In 1997, the parallel computer "Deep Blue" developed by IBM defeated Kasparov, who had been the world chess champion for 12 years, marking a new era in the history of chess. In 2016, the deep learning system AlphaGo defeated Go world champion Lee Sedol and became the first program to prevent a piece from defeating a professional Go player. In 2022, OpenAI released a phenomenal language model called ChatGPT, which has now been widely used around the world and is regarded as opening a new era of artificial intelligence. [1]
2. The application of artificial intelligence in the pharmaceutical industry
According to a 2023 survey of 86 biopharmaceutical and medical device company leaders conducted by Definitive Healthcare, "the global AI market in the life sciences market is expected to reach $7.09 billion by 2028, with a compound annual growth rate (CAGR) of 25.23%, and half of the 50 largest pharmaceutical companies have entered into partnerships or licensing agreements with AI companies" [2]. This highlights the huge interest and investment in the use of AI in the life sciences industry.
At present, AI is widely used in the medical and health field, and can be seen in new drug research and development, medical imaging, and comprehensive auxiliary diagnosis, which has greatly changed the operation of the pharmaceutical industry.
Through the AI system, medical staff can quickly and accurately obtain information about the patient's condition, providing a strong basis for diagnosis and treatment. For example, in the early identification and screening of diseases, as well as the subsequent processing of complex clinical research data, AI can help improve the quality of diagnosis and reduce the rate of misdiagnosis by establishing predictive models to analyze the impact of specific interventions on different patient populations. In addition, AI can also assist medical staff in drug research and development, surgical planning, etc., to improve the professional level of medical services.
At present, the main application fields of AI can be divided into three categories: (1) medical management and administrative work, mainly for the statistical work and data entry of orders and prescriptions; (2) Diagnostic support, such as pathological examination, symptom analysis, image recognition, etc.; and (3) treatment strategy assistance, such as personalized medicine, digital therapeutics, etc. [3] In the following article, we will focus on a new application area of AI, the regulation of the pharmaceutical industry.
3. The practice of using AI for medical supervision abroad
(1) United States
目前美国并未就AI在医药行业的监管功能出台相关法案，仅就对AI本身的监管出台相关法案，例如FDA于2019年4月2日发布《人工智能医疗器械软件变更监管框架提议（讨论稿）》（Discussion Paper: Proposed Regulatory Framework for Modification to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device, SaMD）[4]，于2023年4月3日发布《人工智能/机器学习支持设备软件功能的预定变更控制计划的营销提交建议的指南草案（Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence/Machine Learning（AI/ML）-Enabled Device Software Functions, Draft Guidance）[5]。
But there are relevant practices in other areas. For example, on April 6, 2024, the Pennsylvania Department of Insurance issued a notice on the use of artificial intelligence systems (AI) by insurers; Notice 2024-04[54 Pa.B. 1910]）[6]。 The circular highlights the need for insurers to comply with all applicable insurance laws and regulations when using AI systems, and also addresses potential risks that insurers should consider when using AI systems, such as inaccuracies, unfair discrimination, data vulnerability, and lack of transparency and explainability.
In addition, Pennsylvania launched a pilot project led by ChatGPT Enterprise in January this year to test the real-world effects of generative AI technology in the daily work of government employees. ChatGPT will provide services such as manuscript editing, job description drafting, and policy guide lookup for employees in the Pennsylvania government's OA department [7].
(2) European Union
Compared to the United States, the development of AI itself and its regulation is more remarkable in the European Union. As far as the regulation of AI itself is concerned, on March 13, 2024, the European Parliament passed and approved the EU Artificial Intelligence Act (AI Act). The Act is the world's first comprehensive legal framework to regulate AI to regulate the development and use of AI. In the pharmaceutical industry, the bill sets strict standards and evaluation procedures for high-risk AI medical device applications.
In terms of the use of AI for regulation, the European Medicines Agency's (EMA) Big Data Steering Group published a draft Reflection paper on the use of artificial intelligence in the lifecycle of medicines in July 2023 [8], An overview of current thinking on how AI can be used in the lifecycle of pharmaceutical products. The EMA said its goal is to reflect on the scientific rationale relevant to regulatory review when using such emerging technologies to support drug development. In the post-marketing authorization phase, AI tools can effectively support pharmacovigilance activities, including adverse event reporting management and signal detection.
In December 2023, the EMA and the Head of Medicines Agency (HMA) developed a work plan on AI for 2023-2028 in the European Medicines Regulatory Network (EMRN) [9]. The program was developed to responsibly promote the use of AI in the pharmaceutical and medical device industries. These include implementing and monitoring AI for internal regulatory purposes, enhancing network-wide analytics capabilities, and more.
4. Artificial Intelligence and Drug Supervision in China
(1) The evolution of mainland policy
In July 2017, the State Council pointed out in the "Development Plan for the New Generation of Artificial Intelligence" [10] that "the wide application of artificial intelligence in education, medical care, elderly care, environmental protection, urban operation, judicial services and other fields will greatly improve the precision level of public affairs and comprehensively improve the quality of people's lives", pointing out the development direction of AI in the field of medical and health care in mainland China.
On July 10, 2023, the National Development and Reform Commission (NDRC), the Ministry of Education (MOE) and six other government departments jointly issued the Interim Measures for the Administration of Generative AI Services[11], mentioning that "innovative applications of generative AI technologies in various industries and fields are encouraged".
In this regard, on June 18, 2024, the General Department of the NMPA issued the List of Typical Application Scenarios of Artificial Intelligence for Drug Regulation[12] (the "Regulatory List"). The "Regulatory List" lists 15 application scenarios that are leading and demonstrative, have development potential, address work pain points, and have more urgent needs, aiming to promote the research and exploration of artificial intelligence technology in the field of drug supervision and promote artificial intelligence to empower the drug regulatory system.
(2) Interpretation of the "Regulatory List".
1. Credit-related scenarios
Among the 15 scenarios listed in the Regulatory List, the following three scenarios mention "credit": application 4 "remote supervision", application 12 "business data inquiry", and application 15 "risk management".
In the above scenarios, the Regulatory List points out that AI can be embedded in the "marketing authorization holder credit information" and "drug safety credit file" in a timely manner to carry out operations such as information retrieval, content integration and data analysis.
Take the "drug safety credit file" as an example: drug safety credit information mainly includes the basic information, supervision and inspection information, information on violations of laws and regulations, and other relevant good behavior information of drug safety credit subjects. Before AI intervened, IT technicians usually obtained relevant information in the traditional mode of combining conditional data queries, and then integrated and analyzed the original information to form a report. However, after embedding the corresponding AI technology, thanks to the powerful retrieval and processing capabilities of AI, it can not only easily perform one-click data capture, but also realize advanced functions such as fuzzy query of complex conditions, summary query of associated data, and generation of data charts. In addition, AI can not only analyze based on more detailed data, but also make reports more visual and intuitive, helping business personnel quickly and accurately obtain the required data and information, effectively reducing communication costs and labor and material costs, and ultimately providing strong support for regulatory decision-making.
2. Age-appropriate transformation of drug instructions
In June 2023, the State Food and Drug Administration (NMPA) issued the Pilot Work Plan for the Reform of Drug Instructions for Aging (Draft for Comments)[13], which clearly requires medical institutions, drug retail enterprises and other units to meet the needs of different patient groups and solve problems such as "unclear" drug instructions, especially the need for age-appropriate reform.
The "Regulatory List" is specifically mentioned in Application Scenario 11, which will "empower the transformation of drug instructions for the elderly with artificial intelligence" to improve the drug experience of the elderly.
The "Regulatory List" points out that AI models can be used to simplify drug instructions and carry out customized instructions, and assist the elderly in obtaining drug information by using multiple rounds of human-computer voice dialogue, voice broadcasting, and generating large-print instructions QR codes.
In practice, Baidu Health adopted a similar line of thought when it released an application called "AI Medication Instructions" in October 2023. With the help of the Wenxin model and the digital human innovation model, Baidu Health's AI medication instructions realize real-time interaction between users and virtual humans using the image of pharmacists, and are committed to solving the problems of unclear and incomprehensible traditional paper instructions, as well as the problems of too small words [14]. In addition, Baidu Health's AI medication instructions support patients to ask questions through text and voice, and generate answers based on the questions entered by patients, answering professional drug knowledge in an easy-to-understand way.
3. Application of AI in pharmacovigilance
Pharmacovigilance runs through the whole development of drugs and is an important part of the whole life cycle of drugs. Due to the general lack and limitations of the understanding of possible adverse reactions before the marketing of drugs, the post-marketing monitoring of drugs is particularly important.
However, for a long time, post-marketing surveillance of drugs has been affected by the large number of reports and poor report quality, and the detection efficiency of safety signals is low. Therefore, an important challenge in the post-market testing stage is how to collect and analyze relevant observational data and draw convincing conclusions.
In 2019, the Pfizer research team published a pilot study using AI technology and machine processing procedures to automate the management of adverse event reports, and the results showed that AI is feasible for both adverse event data extraction and report evaluation [15].
According to the relevant content of the 8th application scenario of the Regulatory List, with the help of AI technology, key information can be automatically extracted from the "individual case security report" to achieve structured data processing and eliminate duplicate reports. Then, based on the quality of the extracted information, the content quality is automatically graded, and the security reports that contain sufficient information and have evaluation value are screened for subsequent analysis.
It is expected that in the future, more and more enterprises and regulatory agencies will begin to explore the application of AI in clinical trials and post-marketing pharmacovigilance, especially the analysis and evaluation of safety risks that cannot be discovered during clinical trials as the primary task, and ultimately promote pharmacovigilance to become a highly mature link in the application of AI in the whole life cycle of drugs.
(3) Risks and responses
There are still certain risks and challenges in the application of AI in the field of pharmaceutical supervision, such as incorrect model output, incomplete data representativeness, data leakage, and algorithm discrimination.
(1) The model output is incorrect
Earlier this month, Google announced its withdrawal from the AI-powered search feature "AI Overview," but within a few days of its release, users discovered its inherently unreliable, such as advising users to water their pizza and eat at least one small rock a day.
Why is this happening? The reason for this is that if a Retrieval Augmented Generation system encounters conflicting information, such as significant differences in indications and medication recommendations between the old and new versions of the drug instructions, the AI cannot determine which version to use to generate the response. At this point, it instead combines the information of the two to create an answer that may be misleading.
Although AI is able to provide high-quality information in most cases, there is still the possibility of "making mistakes". In fact, not only is it difficult for AI to draw conclusions because the network is flooded with a large amount of useless and redundant information, but also from a technical point of view, the more specific the topic provided to the AI, the higher the probability of misinformation in its output. If it were just the scenario in the above example, everyone might laugh and give up, but if it involves the use of AI for document processing, specific review, and medication instructions, such a mistake is unacceptable. In addition, the introduction of AI is to a certain extent to reduce manpower, but if it is necessary to manually review the conclusions of AI in order to prevent AI from "making mistakes", the purpose of reducing labor costs has not been achieved.
(2) Data leakage and incomplete data representativeness
AI analysis is based on the collection and analysis of large amounts of data, and the quality of the input data determines the quality of AI output, so ensuring the reliability, integrity, relevance, and representativeness of the data involved is the key to making the most of AI.
At the same time, AI is a double-edged sword, bringing significant innovation and convenience, but also huge data security concerns. The application scenarios in the Regulatory List involve personal information and confidential information in the registration and application materials for drugs and medical devices, the credit information of marketing authorization holders, work plans, etc., and it is necessary to carefully consider the preservation of relevant information and how to prevent leakage and attack.
In terms of the risk of incomplete data representativeness, effective measures should be taken in the data collection stage to improve the online access rate of relevant information, ensure the integrity of data acquisition to the greatest extent, and reduce the risk of unreliable analysis conclusions due to lack of data. As far as the risk of data leakage is concerned, measures such as data de-identification and anonymization, data encryption, and restricting data access can be used to prevent it.
(3) Algorithmic discrimination
Whether AI is semi-autonomous learning or autonomous learning, it needs to passively obtain or automatically capture a large amount of data for self-analysis and build relevant models for this purpose. In the precursor link, the inaccuracy of the data it "obtains" will cause the deviation of the results, that is, the generation of "algorithmic discrimination". Taking AI-assisted business handling as an example, it is currently uncertain whether the following situation will occur in the future: if all the preconditions are met, A submits the materials first and B submits the materials later, but the platform preferentially locks in the quota to B who submits later through the algorithm, because the platform finds that B has more commercial value than A after calculation. If such a situation occurs, it will inevitably violate the fairness of the market, and how to avoid it needs to be solved through technical and other means. Especially when AI enters the regulatory field, whether it adheres to "algorithm neutrality" is extremely important.
In the face of the negative impact of algorithmic discrimination, it is necessary to start from the algorithm itself, regulate its technology and underlying algorithm logic, and lead the positive development of algorithms. In addition, it is also necessary to formulate relevant laws, regulations, and operational guidelines from the outside, and clarify the specific regulatory requirements for AI, such as the specific criteria for determining algorithmic discrimination, and the circumstances under which the AI model must be prohibited.
V. Conclusion
Since the application of AI in the supervision of the pharmaceutical industry is still in the exploratory stage, the "Regulatory List" only lists the possible adaptation scenarios of AI, and does not involve the specific implementation plans and supervision of the follow-up. At present, a more systematic programmatic document on AI application in the field of pharmaceutical industry supervision in mainland China has yet to be formed.
In general, in the face of the rapid development of AI, it is necessary for the mainland drug regulatory authorities to establish AI application rules in the supervision of medicine and related industries as soon as possible, and pay attention to risk prevention while rationally using AI technology to promote efficiency. We also look forward to the safe development of AI and the greater opportunities for the development of the pharmaceutical industry.
Special Statement:
Dentons strictly adheres to its obligations to protect clients' information, and the content of the client projects involved in this article is taken from public information or obtained the consent of the client. The content and opinions expressed in this article are for reference only and do not represent any position of Dentons, nor should they be regarded as issuing any form of legal advice or recommendation. If you need to reprint or quote any content of the article, please communicate the authorization matter by private message, and indicate the source at the beginning of the article when reprinting. Unauthorized reproduction or use of any content in such articles is not permitted.
