RSNA ML Pediatric Bone Challenge Showcases Promising New Tools

In designing an algorithm to predict skeletal age from pediatric hand x-rays, Mark Cicero, MD, and Alexander Bilbily, MD, demonstrated the potential of machine learning (ML) and artificial intelligence (AI) to aid radiologists in better serving patients.

The algorithm, which won first place in the 2017 RSNA ML Pediatric Bone Challenge, provides radiologists with an automated method for analyzing bone age — a significant improvement on current methods which can be cumbersome and time-consuming, Dr. Cicero said.

Winning the RSNA competition was just the beginning. Recently, Drs. Cicero and Bilbily, both radiology trainees at the University of Toronto, Ontario, have founded a medical diagnostics AI company, 16 Bit, which has partnered with an AI platform provider to distribute the algorithm to hundreds of U.S. hospitals — free for public research use.

“Artificial intelligence and machine learning will be the foundation of these next-generation tools and, ultimately, will allow us to provide faster, better and more reliable care to our patients,” Dr. Bilbily said.

ML Challenge Uses Stanford Data

Demonstrating how ML and AI can benefit radiology and improve diagnostic care was the goal of the RSNA ML Pediatric Bone Age Challenge launched by the RSNA Radiology Informatics Committee (RIC) in the fall of 2017.

The online competition drew more than 250 participants, comprised of radiologists, technology companies, computer scientists, engineers and other medical specialists. Participants worked in 29 teams to submit the outcomes of their algorithms. Teams with the most accurate predictions (one winning team and two runners-up) were announced at RSNA 2017 in December.

While there is an overall lack of high-quality validated data that is publicly available, the challenge used data sets generously donated by Stanford Children’s Hospital, CA, and Colorado Children’s Hospital, Aurora, CO. Stanford researchers published a 2017 Radiology study on deep learning networks to assess pediatric hand images. (See Web Extras).

“Access to quality image datasets in sufficient quantity to address the problem at hand is often the limiting factor in performing reasonable machine learning with diagnostic imaging,” said Adam Flanders, MD, former RIC chair and professor of radiology at Thomas Jefferson University, Philadelphia. “Repurposing large imaging datasets used in other types of image-based research is an attractive read-made source of data that could be used for machine learning, but in many instances that data is embargoed.”

Participants were given a set of hand radiographs and corresponding skeletal ages to develop and train their prediction models. Then, they were asked to predict the skeletal ages on a test dataset with skeletal ages withheld.

Most, but not all of the best performing submissions utilized deep neural network techniques based on one or multiple convolutional neural networks, said Jayashree Kalpathy Cramer, PhD, associate professor of radiology, Harvard Medical School, Boston, who built the platform that hosted the challenge. While similar methodologies may be used to derive the answer, the efficiencies provided through these algorithms make these tools useful for radiologists.

“Challenges can comprehensively assess the performance of algorithms by comparing them on common, sufficiently large and diverse data sets using realistic tasks and valid evaluation metrics,” said Dr. Kalpathy-Cramer, who also served on the RSNA ML Challenge Organizing Committee. “However, as this technology evolves and similar algorithms are developed, radiologists will have a choice of algorithms that perform similarly in terms of prediction performance. No one wants to use an algorithm that takes 10 times longer to run.”

A Collaborative Competition

“The RSNA ML Challenge provided a format for involving radiology in the transformative AI and ML tools that will impact their practices,” Dr. Bilbily said.

“Increased exposure to artificial intelligence and its applications in radiology gets more people involved, which advances the field faster through research, experimentation, and novel use-case development, since we all have challenges in our day-to-day work that can be made more efficient with this technology,” Dr. Cicero said.

Safwan Halabi, MD, clinical assistant professor of radiology and pediatric radiology at Stanford and a member of the RSNA ML Challenge Organizing Committee agrees that the benefits of this type of challenge are far-reaching.

“Instead of standing on the sidelines waiting for what they assume is going to happen – that the technology will take away their jobs – this challenge promoted the idea of radiology being collaborative with this technology,” Dr. Halabi said.

RSNA is planning a 2018 ML challenge and will announce details this summer. The winners will be announced during RSNA 2018.

Web Extras

• Access the study, “Performance of a Deep-Learning Neural Network Model in Assessing Skeletal Maturity on Pediatric Hand Radiographs,” at http://pubs.rsna.org/doi/full/10.1148/radiol.2017170236