Automated Screening Reduces Errors in Radiation Treatment Planning

An automated software program developed by Cleveland Clinic to prospectively screen radiation treatment planning parameters for common errors is highly effective at reducing mistakes, but requires updating to remain optimal as radiotherapy equipment and workflows change.

Those are the conclusions of a new study assessing the Department of Radiation Oncology’s Automatic Plan Consistency Check (APCC) program, which has been in use since 2014. The results were presented by Cleveland Clinic medical physicist Ping Xia, PhD, the study’s senior author and the co-developer of the APCC, at the 2019 annual meeting of the American Society for Radiation Oncology (ASTRO).

“Serious errors in radiation oncology treatment plans impact patient safety, and minor errors impact effective communication and can delay treatment,” Dr. Xia says. “The use of automation enables us to improve treatment plan consistency and integrity while avoiding planner errors.”

Development of the APCC

ASTRO’s Radiation Oncology Incident Learning System, a national radiation oncology-specific error-tracking and analysis effort in operation since 2014, has reported that 33% of known radiation oncology safety incidents occur during treatment planning and pre-treatment review/verification.

To address that issue, Cleveland Clinic’s APCC program was developed to automatically review radiation treatment parameters in the Department of Radiation Oncology’s treatment planning system, checking for treatment planning errors most frequently encountered in the department.

The APCC is embedded in the treatment system and can be run by medical dosimetrists, medical physicists and other treatment team members as often as needed throughout the planning process. Its error and inconsistency checklist evolves with feedback from planners, and as treatment devices change. Currently the checklist contains between 20 and 25 items, including such things as multiple treatment isocenters, non-deliverable beam angles and improper beam-naming conventions. A summary error report is attached to the finalized radiation treatment plan.

After some initial skepticism, radiation oncology team members recognized the APCC’s value.

“It took us a bit of effort at first to convince the planners that this was a useful tool,” Dr. Xia says. “The more they used it, the more they appreciated its value. A typical complex treatment plan takes about two days to complete. If a mistake isn’t detected upstream, that time is wasted. Our dosimetrists run the APCC multiple times because they want to correct errors early, upstream.”

What the study found

To evaluate the APCC program’s impact, Dr. Xia and her colleagues examined the volume of planning parameter errors — mistakes that are input to treatment machines — before and after APCC implementation. Parameter errors are reported by radiation therapists using the Department of Radiation Oncology’s institutional Workflow Enhancement tracking system.

The researchers reviewed 253 parameter error reports submitted between 2013 and 2018. They graded the individual severity of the 272 issues documented in the reports, using one of five descriptors: serious (S), near-miss with safety net (NM), clinical interruption (CLI), minor impediment (MI), and book-keeping (BK).

Their definitions:

• S — Potential patient harm if not caught (for example, mismatched beam isocenters).
• NM — Potential patient harm but caught by a safety net in the process (such as incorrect shifts caught by portal images).
• CLI — An event that could halt the clinical process until cleared (such as an ambiguous gantry angle for volumetric-modulated arc therapy beams).
• MI — A minor disruption that does not stop the clinical process (such as a missing source to surface distance).
• BK — A mislabeling in the plan (such as beam labels that do not match institution standards).

The overall number of reported planning parameter errors declined 24.1% from 2014 — the year the APCC was implemented — to 2015, and fell even more precipitously, by 59.1%, from 2015 to 2016, the researchers found. Errors with the greatest potential consequence, in categories S and NM, dropped from 14 in 2014 to 3 in 2015, and remained low (5 or fewer) through the end of the analysis period in 2018.

Keeping up with technology changes

While the APCC program has demonstrated its effectiveness in decreasing radiation treatment planning errors, it must be continuously modified to keep pace with new radiotherapy equipment and resultant workflow changes. New technology creates the potential for errors if staff lacks proper training or familiarity. Likewise, the APCC may not detect planning errors if it is not adapted to account for equipment and process changes.

For example, in 2017, five of six radiotherapy machines in Cleveland Clinic’s Department of Radiation Oncology were replaced with new models, necessitating changes in the treatment planning processes. The researchers’ analysis of planning parameter error reports found an increase in total errors, from 18 in 2016 to 46 in 2017. Most were classified as CI or MI. The total number of errors declined to 33 in 2018 as the APCC was revised.

Future plans: Dissemination and automation

Dr. Xia hopes to share the APCC with other cancer centers, and to add additional layers of automation to the radiation oncology treatment planning process to improve quality and further reduce the number and severity of errors.

“Radiation planning for each disease treatment site is different; therefore, developing the automation that can apply to the specific treatment site is pretty tricky,” Dr. Xia says. An evaluation of her team’s initial efforts in automated breast cancer treatment planning will be presented in the coming months.

The ambitious goal of fully automated treatment planning does not eliminate human medical professionals from the process, Dr. Xia says. The added technology is complementary.

“Humans are smarter than computers,” she says. “When we introduced our first-layer automation for intensity-modulated radiotherapy treatment, our dosimetrists felt their jobs might be threatened. Now they use the automation as their baseline, and they can fine-tune their plan on top of the automated plan. It raises the bar, so to speak.”