AI in healthcare: The BDF warns of risks to fundamental rights and calls for stronger regulation
Due to their health condition, people with disabilities are required to use healthcare services on a regular basis. The BDF notes that the rise of AI in healthcare poses a significant risk to fundamental rights and that the AI Act, the GDPR and the NIS2 legislation constitute absolutely essential safeguards for these rights, which must be consolidated and strengthened.
1. Personal data
Healthcare services hold a vast amount of sensitive data. People with disabilities expect maximum protection to prevent their data from being stolen, traded or falling into the wrong hands. To this end, EU Directive 2022/2555 (NIS2) is essential for regulating the cybersecurity of providers of essential services, such as hospitals. The latter are regularly targeted by cyberattacks. Furthermore, the Belgian law on patients’ rights is based on the principles of the GDPR, on the rights of access, consent and objection. Weakening the GDPR and these principles means ipso facto a weakening of patients’ fundamental rights.
2. The quality of AI systems embedded in healthcare
Apart from systems that determine whether someone is eligible for healthcare services, AI systems embedded in medical devices are not considered high-risk systems. Nevertheless, some of these devices ‘may lead to […] patterns of discrimination […] against […] persons with disabilities persisting’ (AI-ACT, recital 57). The issue of bias is central. After all, a posteriori bias, which enables the personalisation of care, can be beneficial to the patient. However, this does not apply to a priori bias, which results from design choices. These can lead to the exclusion of certain groups of people, for example people with a particular disability, with serious consequences for the adequacy of care provision and therefore for the health of these individuals. It is absolutely essential to enhance the quality of the development of medical AI systems, even if these are not considered high-risk systems. To this end, involving people with disabilities in the development of AI systems would raise developers’ awareness of this bias and help to reduce it. Furthermore, the training of healthcare professionals who use AI is of crucial importance. They must continue to master these AI tools and prevent us from sliding into a two-tier healthcare system, namely care with AI and care without AI. Quality requirements for development and training are key principles of the AI-ACT. These must also be applicable to medical AI systems which, whilst not considered high-risk systems, nevertheless have a significant impact on patients’ health.
3. The options available to a patient who is the subject of a medical AI decision
The options are limited and are primarily based on the provisions of the AI-ACT and the GDPR: public access to the EU database for AI systems; powers regarding information, investigation and appeal held by bodies responsible for the protection of fundamental rights; powers regarding investigation and supervision held by the “AI Office”; cooperation between bodies responsible for the protection of fundamental rights and market surveillance authorities; mandatory registration of high-risk AI systems; protection of the rights of access, information, consent and objection guaranteed by the GDPR. Without these provisions, patients would have no effective legal remedies against all-powerful AI systems.
Finally, the BDF calls on the European Commission to uphold and strengthen the provisions of the AI Act, the GDPR and the NIS2 legislation, which protect citizens’ fundamental rights and their capacity to act when these rights are threatened.
