Abstract
Kidney stones require surgical removal when they grow too large to be
broken up externally or to pass on their own. Upper tract urothelial
carcinoma are also sometimes treated endoscopically in a similar
procedure. These surgeries are difficult, particularly for trainees who
often miss tumors, stones or stone fragments, requiring re-operation.
One cause of difficulty is the high cognitive strain surgeons experience
in creating accurate mental models during the endoscopic operation.
Furthermore, there are no patient-specific simulators to facilitate
training or standardized visualization tools for ureteroscopy despite
its high prevalence. We propose ASSIST-U, a system to automatically
create realistic ureteroscopy images and videos solely using
preoperative CT images to address these unmet needs. We train a 3D UNet
model to automatically segment CT images and construct 3D surfaces.
These surfaces are then skeletonized for rendering and camera position
tracking. Finally, we train a style transfer model using Contrastive
Unpaired Translation (CUT) to synthesize realistic ureteroscopy images.
Cross validation on the UNet model achieved a Dice score of 0.853
$\pm$ 0.084 for the CT segmentation step. CUT style
transfer produced visually plausible images; the Kernel Inception
Distance to real ureteroscopy images was reduced from 0.198 (rendered)
to 0.089 (synthesized). We also qualitatively demonstrate the entire
pipeline from CT to synthesized ureteroscopy. The proposed ASSIST-U
system shows promise for aiding surgeons in visualization of kidney
ureteroscopy.