How Augmented Reality Shaped Cleveland Clinic’s Total Face Transplant

When Cleveland Clinic biomedical engineer Karl West, MS, got the call that a donor had been found for Cleveland Clinic’s first total face transplant, he knew his team had only about 24 hours to perform its mission.

That mission was to take CT scans of the donor’s face, segment out the information that was pertinent to the pending transplant surgery and import it into the Microsoft HoloLens augmented reality device. The surgical team would then use the HoloLens headset to overlay a 3D holographic representation of the donor’s anatomy onto the recipient’s face to assess alignment and refine their surgical plans.

West, who serves as Director of Medical Device Solutions in Cleveland Clinic’s Lerner Research Institute, had been on standby for such a call for months along with the two biomedical engineers on his team for the project. “We worked all night to extrapolate the important information out of the DICOM data set that we got from the donor’s CTs,” he explains.

First use of augmented reality in a face transplant

One reason the team had made it such a priority was the innovative nature of the project. Their work on holographic representations of the faces of both the donor and the recipient made possible the first-ever deployment of an augmented reality platform in a face transplant surgery.

“It was a remarkable advance,” says Francis (Frank) Papay, MD, Chairman of Cleveland Clinic’s Dermatology & Plastic Surgery Institute and a leader of the surgical team for this total face transplant, performed in May 2017, as well as Cleveland Clinic’s two earlier near-total face transplants. “You put the HoloLens headset on and you don’t want to take it off. It’s like having X-ray vision, because you can look at a patient’s anatomy in three-dimensional space. You can walk around the holographic image and even interact with it. It gave me, as a surgeon, an amazing perspective I’d never had before.”

The HoloLens augmented reality platform in action during the face transplant surgery.

How it works

West and his Cleveland Clinic colleagues had been exploring HoloLens augmented reality applications in vascular surgery when Dr. Papay contacted him about using the technology in planning the face transplant. That was about a year before the operation took place, when the recipient was preparing for the transplant (as detailed in this Consult QD post) but before a donor had been found.

“When we started experimenting with the technology, we realized we could take patient-specific images from a CT or MRI data set, do a 3D reconstruction of the images and project them onto the headset lenses,” West says. “HoloLens allows you to see the visual environment around you but to augment it with additional information projected onto the lenses.”

This convinced West and the surgical team that the technology had great potential value for the face transplant, both for thoroughly visualizing the recipient’s anatomy before the transplant and to help match the donor’s anatomy to the recipient immediately before and during the transplant.

“So we took the transplant candidate’s patient-specific images derived from her CT data and registered them to her face using markers,” West explains. “This allowed the surgeons to see the DICOM or CT data set superimposed on the patient’s face as a full volumetric rendering. We extrapolated out the most important information — the skull, bony structures and vascular structures — so the surgeons could see these structures in relation to one another to plan their surgical approach.”

“It really helped us see these various structures and how they fit together, even guiding us in mock surgeries,” says Dr. Papay. “We could see the entire vascular structure up into the skull — which was critical in planning this transplant — as well as how bone structures would overlay each other in various potential scenarios.”

Months of fine-tuning…

These capabilities were made possible by months of back-and-forth between West’s team and the surgeons to pinpoint the most valuable information to visualize. “People don’t realize how much information is contained in a CT scan,” West notes. “There’s so much there that we don’t extrapolate. A key to success is knowing which structural data to focus on for the greatest clinical utility.”

That work was done solely by West’s team in consultation with the surgeons. “Microsoft’s role was limited to providing us with the HoloLens device, which is essentially just a computer with sophisticated features,” he says. “The software was put together from scratch by my lab, with particularly notable contributions by my biomedical engineering colleague Jeffrey Yanof, PhD. We developed all the HoloLens apps used for the face transplant.”

… and then a 24-hour sprint

All the HoloLens work specific to the recipient was done far in advance of the transplant, after which West’s team largely had to wait for the call about a donor to come. When that happened, much of the three-person team’s time during their 24-hour window was spent segmenting out pertinent information for the 3D reconstructions of the donor face for importation to the HoloLens. “It demands a lot of time at the computer just doing image segmentation,” West explains.

An additional team member worked on generating a 3D-printed stereolithographic model of the donor’s face, which took even longer. “The donor was already on the operating table when the 3D printer finally finished,” West notes.

But the team delivered both the holographic and 3D-printed modeling tools in time, and the donor’s DICOM data set was overlaid onto the recipient using HoloLens to allow the surgeons to assess structural alignments shortly before and during the transplant surgery.

An ethos of innovation

“Intraoperatively, HoloLens was particularly valuable in helping determine how much structure needed to be removed from both donor and recipient and in visualizing the underlying bony structures, which were otherwise hard to see because of all the blood and vascularity involved,” Dr. Papay says.

He adds that HoloLens and the stereolithographic modeling (which was done for both the recipient and the donor) complemented each other well. “The 3D-printed models, which we had used in Cleveland Clinic’s two prior face transplants, are good for visualizing larger structures from the CT data set. It’s also helpful that they’re scaled at actual size. In contrast, HoloLens is really helpful for visualizing small structures, like small vessels, which you can’t really appreciate in a 3D-printed model. And HoloLens is able to display various scenarios to help you determine which is the best fit and guide your choice of surgical strategy.”

Surgeons reference a 3D-printed stereolithographic model of the recipient’s face during the transplant surgery.

Dr. Papay says his team plans to publish a paper on how the use of augmented reality with HoloLens impacted this transplant surgery, in terms of various objective measures. “This initial use of augmented reality in face transplantation stems from Cleveland Clinic’s ethos of collaborative innovation — the same ethos that made us the first U.S. institution to perform a face transplant,” he notes. “We look forward to helping define the utility of augmented reality for future face transplants. It’s a natural application of this technology, and we’re just getting started.”