Obligation for AI Systems in Healthcare: Prepare for Trouble and Make it Double?

In the field of medicine, artificial intelligence (AI) is expected to improve early detection, diagnosis, innovative therapies, personalised medicine and disease surveillance. AI is slowly transforming medical practice as AI applications extend into domains that were previously regarded as being solely in the domain of human expertise.

The use of AI has spread widely in every domain due to the evident benefits that it brings. However, as a consequence of their rapid development, AI systems are not easy subjects to regulate. In particular, the rapid development of AI processors makes it difficult for European legislators to keep relevant legislation updated [6]. Several attempts have been made to impose legal standards and limitations on AI applications. The general approach is soft law. Codes of conducts, recommendations, and declarations have been published by European Union (EU) institutions over the last 6 years. The main concerns of legislators and stakeholders are the unintelligibility of AI systems. Currently, most AI technology produces results that are not very traceable or explainable [5]. This issue puts individual and collective rights at risk. If AI results (i.e. outputs) are not explainable, any legal claim against them becomes demanding. In any case, the explicability of AI paths leading from input to output is fundamental, and the degree to which explicability is required depends on the significance of erroneous or inaccurate outputs i.e. the consequences of erroneous or inaccurate outputs.

The explicability principle is a pivotal principle for the development of good AI systems in society. It was established in 2018 by the AI4People Scientific Committee. The notion of explicability has two components. Intelligibility, like transparency in AI practice, answers analytical inquiries such as “Why is this the output?, What steps did you take to get there?, What was the logical path?”. The second component is accountability, which addresses the critical problem of who is responsible for AI mistakes or the harm caused by those mistakes [8]. The issue remains open. The issue remains open. Currently, it is possible to explain steps of AI training, whereas intelligibility of AI functions are not easy to obtain, and consequently accountability is not easy to address.

To address the complex nature of artificial intelligence, the European Commission recently proposed a risk-based regulatory framework called the Artificial Intelligence Act (AI Act). It establishes which AI practices must be banned, defines dangerous AI practices and specifies which AI practices pose limited or zero risks to the citizen’s safety.^{Footnote 1}

Among the different application fields of AI, one that could bring major changes is healthcare [2, 11]. There are many benefits to incorporating AI into the healthcare ecosystem. Through the analysis of large volumes of data, AI can automate tasks such as detecting anomalies, finding the causes of and preventing disease [1, 4]. Overall, AI could deliver faster, higher quality healthcare services at a lower cost. AI could help tailor healthcare services to the individual patient [12]. However, the medical field is highly regulated and is by nature a high-risk sector. Therefore, every time that a new piece of legislation comes into force, several standards and requirements need to be checked, balanced and validated against the EU legislation already in force. AI systems for medical purposes would have to meet legislative standards on AI as well as the standards set out in Medical Devices Regulation 2017/745 EU (MDR).^{Footnote 2} Yet, whenever a new regulation is proposed by the European Commission, its interaction with other EU frameworks is not set out.

This article aims to analyse the relationship between the proposed AI Act and the Medical Device Regulation by highlighting similarities and redundancies and presenting the new set of obligations. The main objective of the article is to examine the relationship between the new EU Artificial Intelligence Act and the existing Medical Device Regulation, and how they may intersect. The focus is on exploring the possible implications of these overlapping regulations and how they may create unnecessary obstacles for the development and implementation of AI technologies used in medical devices. This affects the design and implementation of artificial intelligence techniques employed in medical devices.

The next section describes the legal background of EU legislation on AI systems and provides an overview of the Medical Device Regulation. Section 3 shows the possible intersection of obligations. Section 4 presents some considerations regarding classifications, outlining challenges and problems regarding intersections between the regulations. Section 5 tests the intersection on selected case studies and Sect. 6 presents our conclusions and future work.

Achieving a balance between different interests and principles is an ongoing process for EU legislators, especially when a framework is still at its earliest stages.

On a juridical level, it is essential to analyse legal sources and their intersection to understand the intended objectives, how they change over time and how the legislators intend to trade them off.

No legislation explicitly balances these factors in advance, so it is important to expand on this topic and to investigate every jurisdiction that is contemplating the regulation of AI systems (not only in the EU but also Canada, China, etc.). Our analysis is based on the two regulations, the AI Act in the first subsection and the MDR in the second. The final goal of this article is to create a double risk assessment for AI systems for medical purposes, which will be articulated in the third section.

2.1 EU Policy on AI Systems

The European Union has stated more than once its interest in building strategic leadership on AI applications. Before the AI Act proposal, the EU approach to AI standards was developed through soft law.

The first relevant soft law was the High-Level Expert Group on AI’s Ethics Guidelines for Trustworthy Artificial Intelligence. The final version of that document assumed the AI4people approach and focused on the principle of explicability which subsumed the other “more traditional” principles related to AI systems: beneficence, non-maleficence, autonomy and justice. In addition, it set out three goals to ensure that AI systems achieve “trustworthiness” in their production and use. The guidelines were formulated to provide practical ethical principles. These goals are:

• lawful processing: AI systems perform all their actions in a lawful manner, complying with all applicable EU laws.

• ethical operation: AI processing complies with all ethical principles established within the EU context.

• robust performance: technical and social safeguards to avoid any possible harm from AI.

The second relevant document is the White Paper on Artificial Intelligence: an European Approach to Excellence and Trust.^{Footnote 3} Proposed by the European Commission as a preparative work for the EU framework on AI, it defined the EU regulatory approach to AI systems. It suggested a technology-neutral method to regulate AI in the EU, focusing on their function, their outcomes and their effects on society and individuals [7]. The AI Act’s risk-based approach is built on the recommendations of the White Paper.

On the 21st of April 2021, the European Commission presented the AI Act proposal. The proposal establishes a risk-based classification of structural risks embedded in AI systems based on their design and function, and the potential impact of their errors.

The risk-based classification approach adopted in the AI Act is based on a technical report on the use of AI: ISO/IEC TR 24030:2021.^{Footnote 4} That technical report looked at societal concerns and presented a set of classifications for AI systems according to the risk levels associated with potential failure or unexpected behaviour. The risk levels were determined by looking at the severity of such behaviours. The ISO report details the relevant aspects for assessing the level of risk. The European Commission took this approach into consideration and highlighted that there is no safety regulation covering the high-risk functions detected in the 132 use cases described in the report. The AI Act should be applicable to those use cases, thereby filling these legal and protection gaps.

At the EU level, the same role of identifying applicable standards for AI systems and AI risk management is currently carried out by the European Committe for Standardisation and European Committee for Electrotechnical Standardization (CEN-CENELEC) Joint Technical Committee on Artificial Intelligence and other technical committees in liaison with the European Commission.

The European Commission has decided to build its proposal on risk approach based on this common work.

The risk-based classification of the AI Act is structured as follows:

• unacceptable AI practices: listed in Art. 5 of the AI Act. This is the top-level risk category and it contains four subcategories:

  • manipulative practices, described as:

    • \(\bullet\) actions deploying subliminal techniques to exploit the vulnerabilities of certain groups, and

    • \(\bullet\) actions aimed at manipulating individuals in ways that are likely to cause them harm;

  • Social scoring practices which implies the automatic classification of an individual’s trustworthiness carried out by AI systems only on the behalf of public authorities;

  • real-time biometric identification systems banned by the EU with the only exception of those used for the purpose of law enforcement (e.g. imminent threats, terrorist attacks, locating criminals etc.)

• high-risk AI practices: as defined in Art. 6 of the AI Act. This is the second level of risk category and is based on two main conditions:

  • the AI system is a safety component of a product, or is itself a product, covered by Union harmonization legislation.

  • the AI System is required to undergo a third-party conformity assessment.

• The AI practices with limited risks: defined in Art. 52 of the AI Act. The draft outlines four types of technologies under this category:

  • Deep fake technologies.

  • AI systems that interact with people (chatbots).

  • AI systems with emotion recognition.

  • biometric categorization mechanisms.

2.2 Regulation of Medical Devices

The goal of this section is to underline what can be considered a medical software device by presenting step by step the requirements prescribed by the Medical Device Regulation. This section is useful for understanding whether an AI system with medical purposes can fall under the scope of the Medical Device Regulation.

In general, at the EU level, the Medical Device Regulation (MDR) is the relevant legal framework for medical devices. It sets quality and risk management obligations for medical device companies.

Under Art. 2(1) a medical device is defined as any instrument, apparatus, appliance, software, implant, reagent, material or other article intended by the manufacturer to be used, alone or in combination, for human beings for one or more specific medical purposes:[...]diagnosis, prevention, monitoring, prediction, prognosis, treatment or alleviation of disease[...].

The MDR states that a software can be classed as a medical software device (MDSW) if it is a set of instructions that processes input data and creates output data,^{Footnote 5} and it is intended by the manufacturer to be used for specific medical purposes [10].

The fundamental feature of a medical device software under the scope of the MDR is its “intended use”: the producer must put the software on the market, or put it in use within the European Union, and it must be used for medical purposes. Before the Medical Device Regulation, the Medical Device Directive (MDD)^{Footnote 6} was the relevant framework for medical devices. That framework also referred to software as “medical devices” only where they had an intended medical purpose. The interpretation of “intended purpose”, as defined by the MDD, was stated in a preliminary ruling by the Court of Justice of the EU. In the “Brain Products” case (C-219/11), the Court affirmed that for a to tool to have a medical purpose, it is not sufficient that the tool is used in a medical context. “[I]t is also necessary that the intended purpose, defined by the manufacturer, is specifically medical”. This concept was confirmed in the “Snitem and Philips France” (C-329/16) case. It stated that a software that has medical purposes (such as detecting side-effects, drug interactions and excessive doses) is a medical device within the meaning of the MDD when the manufacturer’s intended purpose is clearly set out [10]. The definition of “medical device” was not subject to major changes in wording when the MDR replaced the MDD, and the “intended purpose” feature remains unaffected. Conversely, tasks like e-mailing, web or voice messaging, data parsing, word processing, and making back-ups of medical information (e.g. patient data) are not considered to have a medical purpose.

On those grounds, an AI system that processes medical data to give diagnostic information could fall under the definition of MDSW.

This is a fundamental step in MDR analysis. Under the conditions mentioned, an AI medical tool having medical purposes can be deemed as a medical device. Only where an AI medical tool falls under both the MDR and AI Act do we have an intersection of requirements/obligations. The notion of “medical purpose” is detailed in the next section.

By falling under the MDR, an AI medical tool would also be subject to a risk classification as a medical device. The risk classification of the MDR defines the dangerousness of a medical device on a scale that goes from class I (lowest risk) to class III (highest risk).

Annex VIII of the MDR establishes rules to classify the risk level of a medical device. There are several types of medical devices, and each type of software with medical purposes has its own classifications. The ones we consider in this paper are non-invasive MDSW with diagnosis or therapeutic purposes. This selection is linked to the consideration that most AI systems used for medical purposes are diagnostic AI systems that work on large datasets.

The general rule (Rule 11) of Annex VIII of the MDR establishes that non-invasive software should fall under class IIa. That is a medium-risk class, since non-invasive software usually provide information for therapeutic and diagnostic purposes. If a medical decision prompted by such information could lead to serious deterioration of health and/or surgical intervention, the MDSW class is raised to IIb. Where it could lead to irreversible deterioration of health or death, it falls under class III. All other uses of non-invasive MDSW are classified as class I. This information is useful for addressing the obligation related to each class in the next section. Plus, they help to understand Table 1 which shows the intersection between MDR classes and AI Act classes.

If an AI system is intended to be used for medical purposes, it raises the question “Can an AI system be legally considered a medical device?”

This question is critical from the manufacturer’s perspective. The answer determines the number of obligations that must be followed, depending on whether they include only obligations from the AI Act or obligations from both the AI ACT and MDR. We detail separate tracks of obligations, because there are no cross-references between the two legislative sources. We start our analysis with definitions.

The AI Act defines an AI system as software developed for a set of human-defined objectives that generate outputs, such as content, predictions, recommendations or decisions influencing the environment they interact with.

However, the most recent version of the compromise text of the AI Act^{Footnote 7} has narrowed its scope. An AI system is now defined as a machine-based system that is designed to operate with varying levels of autonomy and that can, for explicit or implicit objectives, generate outputs such as predictions, recommendations, or decisions that influence physical or virtual environments. This definition might seem similar to the earlier one, but it is significantly more oriented towards machine-learning and leaves out open categories previously listed in Annex I. Plus, the definition of general-purpose AI is now addressed in Article 3.1(a), a total innovation in the AI Act. They are defined as AI systems that can be used in and adopted to a wide range of applications for which they were not specifically designed (multiple intended and unintended purposes). On the contrary, the new version of Art. 2 (the Scope) lists excluded AI systems. These are national security AI systems, military AI systems and AI systems with the sole scope of research and development.

The MDR defines (medical device) software as a set of instructions that processes input data and creates output data for medical purposes. An AI system that processes medical data in order to give diagnostic information could fall under the definition of MDSW. This would also mean that the resulting risk classification would be based on a double assessment [12]. The MDR and the AI Act prescribe obligations for each class of risk.

Table 1 compares and these obligations and shows how they intersect. The columns indicate the AI Act classification, the rows indicate the MDR classification. An “X” indicates that an intersection is not possible. An “+” indicates that there is an obligation to be added, the added obligation is the one that follows on the right. “Not in the EU” means that the AI system cannot be produced, sold or used in the EU market. Classes IIa and IIb are considered together (Class II) for the sake of comparison with the “high-risk” class. Currently, “unacceptable risk” devices cannot enter the EU market or be used in EU territories. As we can read from Table 1, both regulations have created a risk-based classification with obligations for manufacturers.

Manufacturers of medical devices have to draft a “quality management system” (QMS) to assess the compliance of a medical tool with MDR provision. The strictness of the requirements set by the QMS depend on the risk level of medical tools. As defined by Art. 10 of MDR, the QMS has to cover general safety and performance requirements, conformity assessment procedures, risk management, management responsibility, resource management, product realisation management, clinical evaluation and post-market surveillance.

Starting from analysis of the first row, Class I is the MDR class with the lowest risk. Manufacturers of Class I devices have to ensure that prescribed documents are compliant with the MDR obligations. Class I manufacturers have fewer obligations that any other manufacturer: they have to draft QMS, supply technical documentation, create the European Conformity (CE) mark and an EU declaration of conformity, and establish a monitoring plan for medical devices that has entered the market. They must also write a post-market report on the device. In Table 1, the monitoring plan goes by the name “Post-Market Surevillance Plan (MDR)” as prescribed by the MDR for each and every class of medical device. The report is defined in Table 1 as “Periodic Safety Update Report (MDR)” as prescribed by the MDR for higher classes.

Manufacturers have to lodge an application for the assessment of the QMS by a notified body. The involvement of the notified body in the QMS assessment depends on the class of medical device. For Class I measuring devices, the involvement of the notified body extends only to examining their conformity with metrological requirements.

All medical devices using AI algorithms currently fall under the “high-risk” category since they are products covered by Union harmonization legislation, according to Annex II of the AI Act. Nevertheless, the AI Act is still under discussion among EU institutions and its provisions could change. We imagine that by the publication of the AI Act, some medical software devices with more basic functions could be moved to the “limited risk” class.

One example of this “lowered” classification is an app intended to provide basic health information such as the user’s fertility status via an AI chatbot. Such an app would gather and use individuals’ health-related details to provide information about their fertility status with human-like interaction. Such apps are currently classified as class I in the MDR.

The “limited risk” class prescribes few obligations. The major obligation is to notify users when they are interacting with an AI system. Obligations to inform natural persons that they are interacting with an AI system, in any operation they are exposed to, are called transparency obligations. In Table 1, such obligations are displayed as “Notification to Users”. The intersection between class I and the limited risk class would create the lowest level of obligations possible for manufacturers.

Continuing with the analysis of the same row, there is an intersection between the two middle classes, Class II in the MDR and the high-risk class in the AI Act. These classes are subject to similar requirements. They both prescribe a Quality Management System(QMS) to obtain, certify and maintain compliance with rules set out in the regulations. Hence, in Table 1, there are two sets of the same requirement: “Quality Management System (MDR)” and “Quality Management System (AI Act)” are essentially the same obligation prescribed to MD manufacturers and AI manufacturers.

The AI QMS involves test and validation procedures, technical documentation, data management procedures, communication with national competent authorities, and risk management systems. The AI QMS has to be submitted to a notified body.

The second common feature is certifications. In Table 1, there are two set of “certifications”. One is prescribed by the MDR, the “Certifications (MDR)”. The second is prescribed by the AI Act and is displayed in Table 1 as “Certifications (AI Act)”. The AI Act and the MDR require manufacturers to create a CE mark for medium-to-high-level risk devices. The two regulations also require manufacturers to write an EU declaration of conformity attesting compliance with harmonised standards.

Notwithstanding the above commonalities, the AI Act and the MDR prescribe obligations that are strictly related to specific products they cover. The AI Act requires providers of high-risk AI systems to keep a record of events (“Logs”) for a certain period of time. Logs must be kept and retained automatically as part of evidence gathering for documentation on the compliance of the AI system. This obligation applies to all devices classified as “high-risk” (class I, class II and class III). As obligations specifically related to medical devices, the MDR prescribes that a “Periodic Safety Update Report” must be provided for class II devices. The report must collate the results and conclusions of the Post-Market Surveillance Plan.

Finally, Class III is the highest classification for medical devices. There are further obligations prescribed for the manufacturers of class III devices relating to pre-and-post-monitoring procedures. Manufacturers of class III devices may consult an expert panel to review their clinical development strategy and clinical investigation plan. Clinical investigations are a mandatory requirement and must be carried out as part of the post-market clinical follow-up. Manufacturers must update the Post-Market Clinical Follow-up evaluation report and provide a summary of safety and clinical performance of class III devices on an annual basis. In Table 1, the additional obligations are displayed as “Clinical Investigations”.

Prescribing conformity checks with similar features could bring about an overlapping compliance procedure. Recital 85 of the AI Act delegates harmonisation issues to the European Commission. However, the text of the AI Act does not specify the procedures for harmonisation. The issue of overlapping concerns stakeholders on multiple levels: for the MDR obligation of post-market surveillance, post-market controls prescribed by the AI Act would be added. One example of a layer of obligation added with the AI Act are procedures for AI systems presenting risks at national level as defined by Art. 65 and Art. 67 of the AI Act. In such cases, AI medical devices would be involved in double follow-up inspections in order to comply with both regulations. It is arguable that this could lead to an excessive burden on manufacturers, especially for small and medium-sized enterprises.

Art. 11 provides one attempt at harmonisation. It provides for a single set of technical documentation for AI devices under other European Union harmonisation legislation. This strategy could avoid duplication of the QMS and risk management system, and could lead to a single declaration of conformity. The AI ACT mentions but does not specify harmonising a single set of technical documentation. It is likely that the next version of the regulation will address the issue in more depth. On the same note, the selected MDR authorities will continue to serve as the surveillance authorities for post-market monitoring of AI clinical systems, preventing the need for a new supervisory authority. The adjustment on supervisory authorities implies that member states designating required MDR authorities for market surveillance and enforcement should also appoint authorities with expertise in AI. Such solutions have been criticised for being too “pragmatic” [14] since some member states still need to enforce MDR provisions. As a matter of fact, the MDR is a newly adopted regulation and several aspects still require clarification and interpretation.

The most likely scenario resulting from the interaction between MDR and the AI Act is a classification of high-risk medical AI systems.

On the basis of the MDR framework, AI medical devices are mostly software subject to classification IIa or higher. Under this classification, almost all medical software must undergo a strict conformity procedure with third-party assessment.

As they are listed in Annex II, all AI medical devices fall under Art. 6 of the AI Act and are consequently classified as high-risk systems. This mainly indicates that it is likely that manufacturers of AI medical devices will be subject to strong obligations in the AI Act, as the classification seems to be almost a “blanket classification” [13].

However, Recital 31 of the AI Act establishes that where a product is classified as high-risk under the scope of the Act, it does not automatically mean that it also poses a high-risk in the context of its use:

The classification of an AI system as high-risk pursuant to this Regulation should not necessarily mean that the product whose safety component is the AI system, or the AI system itself as a product, is considered “high-risk” under the criteria established in the relevant Union Harmonization Legislation that applies to the product. This is notably the case for Regulation (EU) 2017/745 of the European Parliament.

It may be unreasonable to inflict the same high-risk obligations on a manufacturer whose AI system merely evaluates data as those for a manufacturer whose device performs direct surgical activities on patients [13].

The next section will introduce three case studies based on real research projects. They have been selected to test the current classifications in the two regulations and experiment on the blanket classification. It will also show the number of obligations that the manufacturer will have to comply with, highlighting the duplication and overlapping issues. The selected projects are:

• OAD-CRONO an NLP system developed for the project CANP - La CAsa Nel Parco.^{Footnote 8} The project aims to hospitalise patients in their own home with the support of AI and telemedicine. It is a Piedmont Regional project with European funding—POR FESR PIEMONTE 2014–2020, in collaboration with one of the biggest hospitals in Europe, A.O.U. Città della Salute e della Scienza di Torino^{Footnote 9} (Italy).

• ICONS—Infervision Coronavirus Neural Network Study for the use of AI system (InferRead CT Lung Covid 19) for an experimental study on COVID detection.^{Footnote 10} The project is in collaboration with Infervision who have signed an ITC agreement with the European Commission, Fondazione Compagnia di San Paolo and Istituto di Radiologia Diagnostica e Interventistica dell’Università di Torino, as part of the A.O.U. Città della Salute e della Scienza di Torino.

• DeepHealth^{Footnote 11} for the use of an AI system to detect lung cancer nodules. In collaboration with the European Union and A.O.U. Città della Salute e della Scienza di Torino.

These projects were chosen because the relevant software were developed by our research group during the last 6 months to 1 year period. The Infervision and DeepHealth AI applications for medical imagery are particularly suitable because this is one of the most popular uses of AI applications in the medical field. Although OAD CRONO has a different function and AI method, it is also a useful example, because it shows how requirements related to NLP can be classified under the MDR and the AI Act.

The selected devices are decision support software as defined in the Guidelines on MDSW, i.e. computer-based tools which combine general medical information databases and algorithms with patient-specific data. In the guidelines, the focus is on their role in medical procedures, they are intended to provide healthcare professionals with recommendations for diagnosis, prognosis, monitoring and treatments of patients. Thus, the role of decision support software is to provide processed health information to medical personnel who will proceed with the decision-making phase of diagnosis and treatment. This approach to the role of software in medical proceedings seems to retrace the traditional “Human In The Loop” (HITL) approach, with AI systems having a supporting rather than decision-making role in diagnosis and treatments.

The selected projects will be classified on the basis of the legislative articles taken into consideration in this section. As the AI Act is still a proposal, the considerations described here might change over time. However, to assess the modern approach of the European Union to artificial intelligence, it is essential to acknowledge how the institutions are proceeding with technological advancements. This assessment is pivotal to understanding the next steps of European institutions.

To further proceed with the case study assessment, we provide some contextual information:

• The Hospital institution that developed the AI system analysed in the next section is A.O.U. Città della Salute e della Scienza di Torino. This hospital complex is currently hosting several projects on AI systems with medical purposes: three AI projects developed in the healthcare hotspot will be analysed as case studies in this section.

• In Italy, the the Comitato Etico Interaziendale (CEI) ethical committee has the power to accept or refuse any experimental development. Without the approval of CEI, observational and experimental studies cannot start or progress. Any change in the planning, features or outcomes of projects must be submitted to the committee. The CEI was established in the 1990s with the adoption of Directive 91/507/CEE on the Approximation of the Laws of Member States relating to Analytical, Pharmaco-Toxicological and Clinical Standards and Protocols in Respect of the Testing of Medical Products. The Directive was ultimately adopted as national legislation with the Ministerial Decree of April 27th, 1992. The main function of the CEI is ethical guidance for clinical practice, from opinions on the use of certain medications and devices to assessments of clinical and surgery procedures. As such, CEI assessments of experimental studies could be relevant input for third-party assessments carried out on high-risk AI systems.

In the next section, the CEI’s assessment will always be cited as it provides a preliminary ethical analysis on the functionalities of an AI medical device. The assessment that the CEI carries out as regards each and every experimental study covers ethical use of the medical tool and includes some relevant issues also for the AI Act, e.g. transparency of data used by the medical software. Their assessment could be useful material for the third-party assessment prescribed by Art.6 of the AI Act.

5.1 DeepHealth

DeepHealth for Lung Cancer Diagnosis is an AI application developed within an European project called Deep Health. The application considered in this paper is the Lung Cancer application developed by the Città della Salute e della Scienza di Torino institution in Turin.

Is it an AI application under the AI Act? DeepHealth for Lung Cancer Diagnosis is an application for carrying out the human-defined objectives of segmenting lungs and detecting nodules. CT scans are fed to an algorithm, which produces masks targeting lung nodules and describing their features (e.g. their length, density etc.). Deephealth is based on a convolutional neural network (CNN). Its architecture consists of an encoder, which takes images as input, and a decoder, which reconstructs images and gives segmentation maps as outputs (each pixel is classified simply as background or as a nodule). The Deephealth CNN is deep learning method which is included in the AI techniques and approaches recognised in Annex I of the AI Act.

Is it a medical device under the MDR? The DeepHealth application has the medical purpose of diagnosing and monitoring lung cancer. It works in conjunction with tomography machines as the algorithm analyses CT scans. However, the functions carried out by the AI system and the functions carried out by the tomography machines are different. They both have their own medical purpose. Consequently, the DeepHealth use case should be considered on its own as a medical software device covered by the MDR.

The role of the AI system is to “aid diagnosis”, i.e. its outputs support doctors in their decision-making process. As the tasks set as the objectives of the DeepHealth application are imagining examinations, nodule spotting and providing a supporting role to medical professionals, it falls under class IIa of the MDR.

The resulting intersection is a high-risk AI system and class IIa medical device. DeepHealth manufacturers will face obligations based on this double classification, with ex-ante obligations, in-the market obligations and post-marketing obligations.

The manufacturers, i.e. the IT Department of the University of Turin and Città della Salute e della Scienza di Torino, will have to comply with standards set by the two regulations.^{Footnote 12} The providers have to draft the Quality Management System established both by Article 17 of the AI Act and Article 10 of MDR.

For DeepHealth, a medical device of class IIa, the QMS has to be submitted to a notified body, together with specific technical documentation^{Footnote 13} on its clinical characteristics, i.e. intended patient population targeted by the medical device and medical diagnosis, side-effects, warnings, etc.^{Footnote 14}

The risk management system is a common feature of conformity assessments set out in the two regulations. It involves identifying the risk, estimating its impact under foreseeable conditions, and post-marketing monitoring. Where risks cannot be eliminated, they must be handled with mitigation and control measures, and any residual risks must be communicated to users.^{Footnote 15} As this is also prescribed by the MDR, the DeepHealth manufacturer should provide two QMSs and two risk management systems.

Moreover, certifications for the Deephealth application should be drafted: the AI Act and the MDR both require the creation of CE marks for high-risk systems in conjunction with an EU declaration of conformity.^{Footnote 16} These certifications are needed to attest their compliance with the required standards.

Providers of DeepHealth will have to implement and update the post market monitoring plan, as required by both regulations. As described above, this would involve gathering, recording and analysing data on the safety, quality and performance of the AI system throughout its lifetime.

Since the DeepHealth application is a high-risk AI system, the AI Act prescribes obligations about keeping logs for a period of time that is appropriate in the light of the intended purpose of the high-risk AI system and applicable legal obligations under European Union or national law.^{Footnote 17}

As a class IIa medical device, manufacturers of the DeepHealth application will have to provide for an annual Periodic Safety Update Report. It will include the conclusions of the risks/benefits assessment based on the findings of the Post Market Surveillance Plan and the volume of sales of the application. Finally, they will have to provide an estimate of the size and other characteristics of the population that will use the device along with the usage frequency of the device, as required by Art. 86 of the MDR.

This outline of checks based on the two conformity assessments with similar features clearly shows how interaction between the two regulations could bring about an overlapping compliance procedure. This would easily result in providing the certification bureaucracy with several similar documents.

5.2 OAD CRONO

The second case study involves the AI technology of natural language processing (NLP) applied to hospital at home.

NLP is a series of computer-aided techniques to analyse, understand, and extract information from unstructured and semi-structured data [15]. NLP technologies provide methods to extract relevant information from selected text (e.g. electronic medical records). The use of NLP for medical purposes has been proven to enhance the accuracy of prediction models and better support providers in clinical decision-making.

The OAD-CRONO experimental study was developed in the context of the CANP project. It is intended to discover common features that characterise patients involved with the Hospital at Home service. This is carried out with machine learning and deep learning methods. The specific goal of the experimental study is to find clinical, management and social factors related to the transfer of patients hospitalised at home to a hospital emergency department (ED).

Is it an AI application under the AI Act? The OAD-CRONO algorithm is fed with patient data from hospital discharge forms (input), which are analysed with NLP methods in order to produce the success/failure rate of home hospitalisation compared to hospitalisation in ED (output). In the OAD-CRONO experimentation study, NLP technique are used to extract keywords from medical texts and classify them in non a priori categories. Machine learning is then used to create patients’ clinical profiles (‘models’) and reconstruct the clinical trajectories according to the available data. The OAD-CRONO system uses statistical approaches and machine learning to produce its output, which are both listed in Annex I of recognised AI techniques and approaches. Consequently, OAD-CRONO falls under the definition of AI in the means of the AI Act.

Is it a medical device under the MDR? The OAD-CRONO application has two modules applicable for research and diagnostic purposes respectively. The second module produces outcomes of the probability of the patient being hospitalised. As for DeepHealth system, its role will be to “aid diagnosis”, hence it will have an impact on patients’ clinical treatments. On that note, OAD-CRONO could be classified as a class IIa medical device. However, its practical task involves a specific kind of patient: it will affect patients with “acute diseases” who are already in precarious situations. It will take part in the decision-making process on hospitalisation. As it might involve conditions of irreversible deterioration or even death, it is more likely that its classification will end up being class III. It is arguable that the “research module” might affect clinical treatments too. New healthcare patterns might emerge through the detection of shared variables and consequently change current methods of diagnosis. Would the objective of the second module (the “research module”) be enough to class OAD-CRONO as a medical device? The current formulation of Rule 11 (‘software intended to provide information which is used to take decisions with diagnosis or therapeutic purposes is classified as class IIa[..]’) means that this is questionable and further interpretation and jurisprudence will probably address the issue. Nevertheless, the OAD-CRONO diagnostic module can be considered a medical device in the light of the above-mentioned features, thus the analysis will continue on the first module.

As OAD-CRONO is a medical software device, it comes under the category of safety products in Annex II. Thus, it qualifies as a high-risk class application under Art. 6.

The resulting intersection is high-risk AI and class IIa medical device. For OAD-CRONO, a similar number of obligations to those already examined for DeepHealth application are required. The manufacturer must have a QMS, risk assessment, management of technical documentation, a post-market monitoring system, they must register the system to EU databases and they must keep automatically saved logs. They must draft the EU declaration, manage the CE mark and carry out clinical evaluation within the QMS context.

Nevertheless, the intersection might also between high-risk and class III devices. In this scenario, manufacturers would encounter the maximum number of obligations: the shared concept between the AI Act and the MDR is the protection of users from any harm from tools carrying out certain functions. The resulting scheme of obligations is the most stringent, considering that high-risk class III AI medical systems are the most likely to affect patients (the only stricter framework would be banned AI systems, which for medical purposes seems unlikely).

The additional obligations prescribed for class III devices relate to pre- and post-monitoring: for class III devices, manufacturers may consult an expert panel to review their clinical development strategy and clinical investigation plan. Clinical investigations are part of the Post Marketing Clinical Follow-up (PMCF) and they are a mandatory requirement for class III devices. A clinical investigation is a document describing “the rationale, objectives, design, methodology, monitoring, statistical considerations, organisation and conduct of a clinical investigation”.^{Footnote 18} Clinical investigations must take place if the information necessary on device performance, safety, and clinical benefit can be obtained only by testing the device on humans.^{Footnote 19} Nevertheless, a clinical investigation is always mandatory for class III devices, regardless of the amount of information that can be retrieved from other sources.^{Footnote 20} Therefore, the manufacturers of OAD-CRONO should provide the results of a clinical investigation within a PMCF and update the PMCF evaluation report and summary of safety and clinical performance once a year.

5.3 InferRead CT Lung COVID-19

InferRead CT Lung COVID-19 was developed in 2020 as an AI solution for the diagnosis and management support of COVID pneumonia. It uses neural network technology to analyse CT images and is built on an online server to which TC images are automatically sent after being recorded.

The module’s algorithm works on the detection of COVID lung lesions and the segmentation of lung lobes (right upper lobe, middle lobe, right lower lobe, left upper lobe, left lower lobe) [3]. The AI application estimates a patient’s chance of being infected with COVID: it assesses the probability that the patient has interstitial pneumonia caused by the virus.

Is it an AI system under the AI Act? InferRead is a software with a human-established set of objectives. It was chosen by programmers to obtain outcomes (predictions) that influence the surrounding environment. The software uses ML and deep learning techniques and is currently in its experimental phase. However, it is intended to be sold and used in EU countries. On these grounds, it is an AI application within the scope of the AI Act, as defined by Art.3, and pertains to the AI techniques and approaches recognised in Annex I.

Is it a medical device under the MDR? InferRead CT Lung Covid 19 is an AI tool of built to support Covid diagnoses. It has investigative, diagnostic and monitoring functionalities. It is used in conjunction with the CT imaging machines of several hospital institutions. However, the AI system has its own goal and functions. Altogether, InferRead CT Lung COVID-19 is a medical software device under the definition of the MDR.

InferRead CT Lung COVID-19 is not the first AI tool developed by Infervision that works on computed tomography. InferRead CT Pneumonia and InferRead CT Lung are also software produced by Infervision that work by processing CT images through algorithms that set precise objectives to be carried out. These AI products have already been officially classified as medical devices

With software that intends to impact diagnosis or monitor vital physiological process, the class of reference is usually that of “medium risk”, thus they have both already received the classification of class IIa MDSW. They have both received the CE mark, thus they can be sold and used inside the EU.

On these grounds, InferRead CT Lung COVID-19 should also fall under class IIa, as it has similar functions of aiding in diagnosis as the two system described above. As it is still in the experimental phase, the role that InfeRead will play in therapeutic decision-making might change the final classification of the AI tool. Considering that studies on COVID confirms that it can present long-term effects such as permanent respiratory complications, and even cause death [9], a diagnostic tool such as InferRead might also be classified in a higher class, from class IIb to III. If the AI system is of class IIa, InferRead will be considered a “medium risk” medical device, whereas if it is of class III, it will be recognised as a “high-risk” medical device.

InferRead also falls under the high-risk category within the meaning of Art.6. It fulfills the requirement of being a safety product, as it is a MDSW.

The two classifications entail obligations for InferRead providers that refer to levels of danger embedded in the InferRead CT Lung COVID-19 device. The producers will have to adjust their management system to comply with the provisions of both the AI Act and the MDR.

The resulting intersection belongs to the high-risk class, but it is uncertain if it should be classified as IIa, class IIb or even class III.

The obligation resulting from the classification intersections are:

• If InferRead is classified as a medical device of class II (either class IIa or class IIb) Infervision Inc., the manufacturer, will have to provide for a conformity assessment procedure to be carried out as an internal control. The manufacturer must establish a QMS involving a general strategy for compliance with the AI Act and the MDR. As part of the QMS for AI systems, the company will have to develop methods and systems to test and verify both the functioning and safety of the AI system, including standard of data management, the draft and examination of technical documentation, a risk management system and a post-market monitoring system. Then, the Infervision company will have to produce a written EU declaration of conformity and the CE mark. Conversely, as part of the QMS for medical devices, a risk management system must be set up to evaluate hazards and risks related to the medical device’s functionality. Also, for MDR provisions, the CE mark and the EU declaration of conformity must be provided by the manufacturer.

• If InferRead is classified as a class III device, the only additional requirement is the post-marketing clinical follow-up (PMCF). As for OAD CRONO, the InferRead manufacturer will have to carry out a clinical investigation as part of the PMCF.

The European Union has decided to take on the challenge of developing one of the first legal frameworks in the world to impose obligations at a national and international level, with provisions targeting participants across the AI value chain such as importers, distributors, authorised representatives and even users.^{Footnote 21}

Indeed, the Union has made no secret of its wish to export its values and principles across the world, based on an approach that respects fundamental rights, including human dignity, pluralism, inclusion, non-discrimination and the protection of privacy and personal data.^{Footnote 22}

Fundamental questions remain concerning the interaction between AI systems and medical devices, and what will the regulations change. The proposed AI regulation, if approved, would add several provisions affecting various ways of implementing and marketing medical devices. The case studies in the previous section presented all the features, strengths and flaws that the classifications are likely to include.

As with any newly adopted EU regulation, several aspects require clarification and interpretation. The MDR is a recently enforced framework, thus it has not been fully implemented all over Europe. Additionally, its provisions have not yet been referred to in relevant EU documents such as the opinions of the Court of Justice of the European Union (CJEU). Furthermore, the AI Act is still a proposal and it will change by the time it will be published as an official EU Act. Also, there are many aspects of the AI Act that will be left to the legal profession to clarify (e.g. the scope of the AI Act, its classifications, the interaction between different obligations etc.)

The AI Act has the main aim of regulating the high-risk practices of AI systems to prevent harm. However, the current assessment of AI system categories could easily lead to overly cautious classification of many AI medical devices as “high-risk” even for products presenting no risk to fundamental rights and safety, especially when other EU frameworks require a device to undergo some non-AI related assessment. On that note, the AI Act would end up classifying devices such as ordinary apps for health as high-risk systems. The European Commission should amend the legislation to avoid over-regulating high-risk AI devices that in practice pose low or limited risks.

The AI Act is still at the beginning of its journey. The European Commission foresees it will require at least one and a half more years before it becomes applicable, with the second half of 2024 being the earliest foreseeable time.^{Footnote 23} Until that time, technology could change in a way that will lead to the need to add provisions and major variations to the AI Act.

As an experiment, we will apply these considerations to another real case study belonging to our projects: C.o.Rsa, a selected and finalised research project developed by the Piedmont Region under “INFRA-B—Support for projects for the construction, strengthening and expansion of public infrastructures”. It involves A.O.U. Città della Salute e della Scienza di Torino and the University of Turin. The objective of Co.R.S.A. is to develop and install a state-of-the-art AI system to assist with the diagnosis of COVID pneumonia from chest X-ray (CXR) and evaluate its impact.

The choice of C.o.RSA as the next case study makes it possible to further analyse the relationship between the MDR and the AI Act. It has been chosen out of all the ongoing experimental projects developed by our research group for A.O.U Città della Salute e della Scienza di Torino as the technological development could bring interesting insights and analyses on the AI Act. Also, the compromise text is already a second version of the AI Act Proposal. With C.oRSA as a case study, it will be possible to monitor changes in the EU’s approach to AI, and amendments to the provisions of the AI Act until its final published form.

The experimental activity in the involved radiology sector will allow us to evaluate the impact of the AI system on patient management processes, from the critical phase of admitting patients up to hospital discharge, and including the adoption of appropriate precautionary procedures of isolation. The experimental study will be fundamental to the early identification of COVID patients and remove them from the common waiting areas of emergency and urgency departments.