FNPG-NH: A Reinforcement Learning Framework for Flexible Needle Path Generation with Nonholonomic Constraints

Path planning algorithms for minimally invasive neurosurgery involve avoiding critical structures such as blood vessels and ventricles while following needle kinematics. The majority of planning solutions proposed in the literature use sampling-based algorithms. This paper introduces a Flexible Needle Path Generation framework with Non-Holonomic constraints (FNPG-NH), an extension of our FNPG framework. FNPG-NH uses deep Reinforcement Learning (RL) based methods such as Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO), and Soft Actor-Critic (SAC) to obtain a kinematically feasible path for a bevel-tipped flexible needle using a nonholonomic model. RL algorithms presented in this work generate the control input for needle rotation based on the rewards generated by the environment. The deep RL algorithms are trained on an environment that consists of (1) ventricles segmented from T1 images of the healthy volunteers using atlas-based segmentation, (2) blood vessels segmented from MRA volumes of the same volunteer using thresholding, and (3) tumor volume from labeled BraTS 2020 dataset and placed at an anatomically relevant location. The paths generated by the reinforcement learning algorithm and the traditional sampling-based algorithm RRT are compared for various performance metrics. The reinforcement learning model was trained on 20 volumes and validated on 68 volumes, and RRT was evaluated on the same 68 validation volumes. The results show that the trajectories generated by the FNPG-NH framework are safer, shorter, and take less time than RRT while avoiding critical structures such as ventricles and blood vessels.