Predicting Embryo Morphokinetic Annotations Using Convolutional Neural Networks

Convolutional neural networks (CNN) can predict morphokinetic annotations of early development of many embryos with high frame-level accuracy, according to results from a study being presented at the upcoming American Society for Reproductive Medicine Scientific Congress & Expo by Cleveland Clinic resident physician Julian Gingold, MD, PhD.

“Incorporating machine learning into embryo assessment has significant potential to transform how we evaluate and select embryos,” says Dr. Gingold, who worked on the retrospective cohort study with Cleveland Clinic IVF Laboratory Director Nina Desai, PhD, and computer science collaborators at the University of California and Carnegie Mellon University.

“The analysis in the IVF lab currently is limited in that it requires a trained embryologist to review video images one by one, frame by frame to determine when the embryo is changing,” Dr. Gingold explains.

“We employed a data set of over a thousand different embryos and 100,000 different images that were individually reviewed and annotated,” he says. “The motivation for our study was to apply the technology of machine learning to figure out when developmental milestones occur and ultimately improve the speed and accuracy of our predictions.”

The study sought to generate a convolutional (feed-forward) neural network to predict the morphokinetic annotations (describing developmental progression with a “when did this happen?” assessment) from the raw time-lapse images without the need for human image review.

Images analyzed first 70 hours of development

The study involved 113 Cleveland Clinic Fertility Center patients from 2014-16. Images were collected every 15 minutes while in culture using EmbryoScope™. Data from 236,595 frames from 1,309 embryos were divided as training/validation/test sets with 93/10/10 patients, respectively.

“We wanted to initially construct a simple problem, so we restricted our analysis to the first 70 hours, the very first few stages of development” says Dr. Gingold. “In training the model to take advantage of prior knowledge about embryo development, we found we were able to improve our prediction accuracy by incorporating information from adjacent frames.

“Also, because we know an embryo should be expected to only progress forward in development, we were able to do what amounts to a smoothing of these predictions to further improve our accuracy.”

The study found CNN model raw frame-level accuracy is greater than 82 percent in predicting early embryo morphokinetic annotations versus the human embryologist gold standard. Incorporation of adjacent frame data using a late fusion approach improves accuracy to greater than 84 percent.

Figure: Convolutional neural network (CNN) model accurately identifies early embryo morphokinetic annotations. (Left) Raw image from a three-cell embryo in IVF lab. (Right) Morphokinetic annotations of a developing embryo from 3 different tested models. Gold standard human annotation (purple) is compared with model predictions made using a single-frame CNN (green), a CNN incorporating 15 adjacent frames (red), and a CNN incorporating 15 adjacent frames after smoothing (blue). Horizontal black line notes developmental time of corresponding raw image at left.

What’s next?

“Further optimization and automation are going to continue to improve our accuracy numbers,” notes Dr. Gingold. “We also will want to expand the model, so we can apply it to the entire course of an embryo development in the IVF lab to save time and expense and avoid the potential inter-observer variability associated with different embryologists reviewing images frame by frame.

“There’s obviously value in having a more consistent, objective way of scoring embryos,” Dr. Gingold continues. “Longer term, we’d like to see if we can generate a model using the time-lapse images that can better predict the potential of an embryo to implant and be chromosomally normal, which would be very valuable for use in clinical contexts in the IVF lab for patients who are undergoing fertility treatment.”

This is the first reported study using neural networks and machine learning to automate the prediction process of early embryo development and morphokinetic annotations.