Artificial Intelligence (AI) in Human Research

The purpose of this guidance is to develop guardrails around the use of AI technology in human research that prioritize the protection of human participants, research integrity, and innovation, while minimizing risks to participants, including data security risks. This guidance will continue to evolve with the advancement of AI technology. If you prefer this guidance in PDF format, please click here. 

What is AI?

 

“At its simplest form, artificial intelligence is a field, which combines computer science and robust datasets, to enable problem-solving. These disciplines are comprised of AI algorithms which seek to create expert systems which make predictions or classifications based on input data.” (IBM).

 

Throughout this guidance AI will be referred to generally as AI technology due to the differing terminology utilized, depending on the stage of the research or project as well as the intentions of the technology.

 

Some technologies may be an algorithm foundation for a mobile application or software while others may be a chatbot that intervenes or interacts with participants. Click below for additional guidance on various terminologies that have been used to describe AI technologies in research protocols as well as additional background information about AI. 

Additional Background Information on AI

Responsibilities of Researchers Using AI Technology

 

Most importantly, investigators should ensure compliance with regulations (e.g., copyright law, privacy regulations such as HIPAA and FERPA), local laws, and institutional policies (e.g., Sharing Data and Biological Samples with Third Parties, confidentiality agreements and intellectual property).

 

Researchers also have a responsibility of discretion, verification, and disclosure.

Discretion, Verification, & Disclosure AI Guidance from ISC

Clinical Research Considerations

Patient Privacy Protection

It is not permissible under HIPAA nor Penn Medicine policy to share patient or research participant information in connection with public AI/ML services, such as ChatGPT, even if the data is de-identified. This is because, as currently configured, such public services can use and share any data without regard to HIPAA restrictions and other protections. Therefore, individual patient data and patient data sets (even if de-identified) may not be exposed to any Open Source AI/ML services. It is recommended that internal, HIPAA compliant tools be used whenever possible. If external tools must be used, OpenSource tools are not permitted. Penn Medicine offers Penn AI Chat (upenn.edu) as a validated, HIPAA compliant version of Chat GPT 4.0 for both research and QI purposes.

Patient Care Delivery

External AI and ML services, including ChatGPT, may not be used as an adjunct to facilitate patient care. Novel applications of AI in patient care will be thoughtfully evaluated against validated clinical standards and best practices in AI, privacy, and security prior to deployment. Note that services deployed within our electronic medical records systems have been vetted and are acceptable.

Please direct any questions on these matters to governanace.ai@pennmedicine.upenn.edu.

Penn Medicine AI Governance Review

Clinical Research within Penn Medicine that involves the use or development of Artificial Intelligence (AI) technology as defined above is required to be reviewed by the Penn Medicine AI Governance Committee. This includes any research that involves the patients or protected health information being exposed to AI technology. This review can be commenced via completion of the REDCap form.

AI Governance Form

Clinical Investigations

AI technologies may be considered a medical device based on its use in the protocol if they are “…intended for use in the diagnosis of disease or other conditions, or in the cure, mitigation, treatment, or prevention of disease, in man…” FDA intends to exercise enforcement discretion for some clinical decision support software and for software /mobile applications that:

  • Help patients (i.e., users) self-manage their disease or conditions without providing specific treatment or treatment suggestions; or
  • Automate simple tasks for health care providers.

 

If AI technology meets the definition of a medical device AND the protocol investigates the safety and / OR effectiveness of the technology, the protocol is an FDA regulated clinical investigation. In these cases, please consult with OCR Regulatory.

OCR Regulatory Services
Additional Penn Medicine Guidance

IRB Review Considerations

 

In order for the IRB to conduct an ethical review, the IRB must have a clear understanding of the AI technology. Certain basic information about the technology should be included in the application or standalone protocol. The IRB recommends a standalone protocol for this type of research, given its complexities.

 

Basic Information

  • Purpose of the technology [e.g., prediction model, mining text records, automation, biometric recognition (face, voice, etc.), etc.]
  • What kind of technology is being utilized? [Machine Learning, Deep Learning, Natural Language Processing (NLP), Unsupervised Learning, Reinforcement Learning, etc.]
  • Is the technology adaptative (learns in real time) or locked?
  • Is the technology automated?
  • How the technology is being used on the protocol?

  • Is the technology intended to “inform” or to “drive” decisions?

Ethical Considerations

Respect for Persons

To ensure risks are minimized by using sound scientific design, does the protocol describe …

  1. The data characteristics used to train the model / algorithm?
  2. How the model / algorithm(s) function, including the process and role of the model’s output in final decision-making?
  3. The method and sources of data collection?
  4. Continuous training/iteration, updating, and monitoring of model (to account for data change, or model drift over time)
  5. What will happen to the data when this specific project is complete? (Will the model / algorithm continue using the data for future training? Will the model / algorithm be shared or marketed?)

 

Does the protocol consider…

  1. To ensure participant autonomy, are participants prospectively informed when the product will impact their care or wellbeing?

Justice

To ensure equitable subject selection…

  1. Is there diversity (including but not limited to: race, skin tone, gender, socio-economic, disability, etc.) in the data source that meets the needs of the study design and procedures to ensure equitable selection?
  2. Does the protocol describe how algorithmic decisions minimize disparities and unjust impacts, such as health disparities, when comparing data across different demographics or affected communities and individuals?

Beneficence

To ensure an appropriate risk-benefit ratio, does the protocol clearly describe …

  1. Who will directly benefit from this technology
  2. How findings and general knowledge benefit the populations of which the data originates

 

Potential risks of the technology; Risks may include but are not limited to:

  • Confidentiality: e.g., reidentification of data
  • Future / Secondary: e.g., discrimination and/or population-level harms due to bias, technological errors, or vulnerabilities in the algorithm
  • Risks to Participant Wellbeing: i.e., in cases where AI gives health advice (e.g., chatbot) or makes decisions impacting participant’s health (e.g., device software)

 

Risk Mitigation Strategies and Monitoring Plan

  • Adequate controls in place for preventing abuse during the research, and after the research is complete.
  • Describes iteration requirements and plans for continuous monitoring and evaluation of the data (retraining model) to intervene when there are undesirable outcomes
  • Describes monitoring of participant interactions/interventions and escalation to humans

References & Resources