QUT developing multimodal imaging system for robot surgeons – Software

Minimally invasive surgeries like keyhole operations often mean doctors work with a limited field of view, but robots will need a clearer picture if they’re ever to perform surgery autonomously.

Researchers at the Queensland University of Technology are working to fix that by developing a multimodal imaging system that combines miniature camera technology, 3D ultrasound and machine learning to provide unparalleled situational awareness to medical robots.

Dr Davide Fontanarosa from QUT’s Faculty of Health said that ultrasound technology is perfectly suited to the task since, unlike other medical imaging techniques, it works in real time and is much more compact than MRI or CT scanners.

Ultrasound scanners also avoid the risks of radiation exposure or magnetic interference.

“Tendons and ligaments, despite their many structural similarities, and several other soft tissues can be distinguished in ultrasound images,” Fontanarosa said.

“Ultrasound imaging is portable, completely harmless to patients and avoids side-effects such as claustrophobia but requires lengthy training to be able to interpret and use it.”

That’s where machine learning comes in, identifying structures detected by the ultrasound and combining it with camera technologies to inform robots on where to position instruments.

Specifically, Fontanarosa told iTnews the team is using deep learning convolutional neural networks (whose organisational structure resembles the visual cortex in animal brains, suiting them to image processing) for classification, segmentation and tracking in live images.

Knee tissue ultrasound. Source: QUT

Dr Ajay Pandey, a lecturer in Robotics and Autonomous Systems at QUT, highlighted how autonomous robots could be useful in knee surgeries, Australian surgeons surveyed in a study acknowledged “inadvertent damage” to cartilage tissue in ten percent of arthroscopic knee procedures.

“Procedures such as meniscal repair and anterior cruciate ligament (ACL) in knee arthroscopy require extreme care,” Pandey said.

“Advancing the medical imaging technology will greatly enhance surgeons’ clinical abilities and drastically lower the physical footprint and the cost of surgical procedures.”

While the technology is yet to be developed to the point of use in clinical trials with live patients, Fontanarosa said the team could reasonably expect a proof of concept prototype to be available in the next five years.