Consider This Comprehensive Approach for Evaluating AI

Artificial intelligence (AI) is poised to save clinicians time by automating clinical documentation.

While several healthcare organizations have quickly adopted AI, Cleveland Clinic is taking a methodical, data-driven approach to evaluate several software products before implementing the technology.

“When considering AI software, we’re interested in a strategic partnership that will evolve as new technologies are introduced,” says Eric Boose, MD, Cleveland Clinic’s Associate Chief Medical Information Officer.

Dr. Boose, a family medicine physician, recognizes AI’s potential to improve efficiency while reducing clinician burnout.

He explains, “Most clinicians would welcome a tool that expedites documentation and allows for more face-to-face time with their patients – that’s why we choose this profession.”

To quicken clinical documentation, providers can use ambient listening AI software to record conversations with their patients. Their discussion is automatically generated into a structured clinical note. The provider reviews, revises as needed and accepts the note as part of the patient’s electronic health record (EHR).

But adding new technology doesn’t always lead to positive outcomes.

“Oftentimes technology investments can be burdensome when it’s just another thing to do,” says Jennifer Owens, Senior Business Relationship Manager within Cleveland Clinic’s Information Technology Division.

Owens adds, “Incorporating new technology needs to provide a smooth care delivery workflow that’s optimal for our caregivers and patients. Identifying the best product comes with a thorough evaluation process.”

Kaitlyn Apodaca, a Cleveland Clinic Children’s Program Manager, agrees: “From the project’s onset, there’s been clear direction to remain an industry leader while using data-based evidence to identify the AI vendor that best fits with our strategy.”

“We’re evaluating a partnership, not just purchasing a product,” says Apodaca.

Charting the course

Building the AI evaluation program focused on the software’s capabilities and deliverables; the operational, clinical and financial impacts as well as provider and patient satisfaction.

By applying a multifactorial approach, and with Cleveland Clinic Innovations’ input, the team narrowed their selection process to identify the appropriate vendors.

Dr. Boose explains, “Guidance from the Innovations team helped identify vendors by adapting their market evaluation tool with our operational metrics. We wanted a wide sampling of vendors from established organizations to start-ups that offer stand-alone technology or an integrated tool within our EHR.”

Selected vendors must adhere to our cybersecurity requirements and complete our legal contracting process.

Building expectations

To effectively evaluate the software, pilot participants range from self-volunteers, providers with varying technical expertise and those who spend ample time documenting their notes. Data was collected on adoption, clinician experience, documentation quality and operational efficiency.

During each three-month pilot, participants attend a 45-minute training session and agree to use the tool at least once per clinic day. They also agree to complete surveys at different intervals throughout the evaluation period.

“The sweet spot for an IT solution is when the technology is good at something that isn’t a good use of caregivers’ time.” Owens continues, “To get this level of detail, we administer several industry-standard surveys to capture objective and subjective data. It’s not worth investing in a digital tool that doesn’t alleviate pain points.”

Preliminary pilot data shows increased provider satisfaction with a steady decline in note writing burden and self-reported burnout.

Quality matters

Caregivers from the Coding, Compliance and Legal departments are interested in the team’s findings.

Dr. Boose describes the workflow from a quality perspective: “All AI generated notes must be reviewed and accepted by the provider before being entered into the EHR. Coders are also reviewing my notes for accuracy. There have been a few clarification requests, such as a more specific diagnosis, but it’s easy to adjust my templates as needed.”

Similarly, Compliance and Legal are reviewing the workflow to ensure the technology meets regulatory standards, including keeping patients informed. Although covered within existing patient consent forms, flyers and provider talking points ensure patients understand and feel comfortable with the AI recording process and workflow.

“Patient feedback is very positive, with many stating that they appreciate their provider’s attention instead of looking at the computer or their notes,” says Dr. Boose. “And providers describe improved conversations in a more relaxed setting.”

Multiple teams will continue to compare pre- and post-notes to monitor the AI software’s accuracy.

Positive praise supports approach

The team is confident with their approach to vet several vendors before reaching a final decision.

Owens describes, “There are so many interesting, and unexpected, nuances with each vendor. That level of product differentiation reinforces the importance of a multifactorial review process to determine the best solution for our caregivers.”

For Dr. Boose, “The vendors have been receptive to our feedback and making changes to improve their product,” says Dr. Boose. “That responsiveness and collaboration signals greater opportunities for future growth and advancements.”

Since launching the AI evaluation program, Apodaca has transitioned to a new role supporting providers who are part of the pilot.

“As an administrative fellow, I worked on the back-end tasks to start the AI evaluation program, says Apodaca. “Now I’m working with clinicians who are trialing the technology. Their encouraging feedback shows that our comprehensive approach was the right decision.”

In the end, they all agree, “Adopting new technology needs to make a positive impact. Throughout this project, we’re consistently hearing that this solution is making a significant difference in people's day-to-day lives – and that’s rewarding.”