Image analysis for detecting deviations from a surgical plane

Systems, methods, and computer readable media related to detection in video of surgical instrumental deviations from surgical planes are disclosed. The methods may include receiving video frames from a surgical video feed. The frames are analyzed to identify a surgical instrument. The frames are evaluated with the identified surgical instrument to ascertain an interface area corresponding to a location of an interaction between the identified surgical instrument and tissue. Stored data is accessed characterizing a surgical plane corresponding to the interaction, and the stored data is used to determine whether the interface area is outside of the surgical plane. An out-of-surgical plane signal is outputted indicating a deviation from the surgical plane by the surgical instrument.