LieToMe: Preliminary study on hand gestures for deception detection via Fisher-LSTM

•Hand gestures exploited to detect deception in real-life trials.•Fisher Vector (FV) encoding leveraged to better represent hand gestures.•LSTM employed to fully exploit the FV representation of deception cues. The ability to discern lies, more broadly known as deception detection, is an invaluable skill that can strongly influence the outcome of relevant situations such as court trials and police interrogatories. Several devices currently exist and are being used (e.g., magnetic resonance and polygraphs) to ease those tasks; although, due to the subject awareness of such tools, their effectiveness can be compromised by the person intentional behavioural changes. Thus, alternative ways to discriminate lies without using physical devices, could become critical assets for the aforementioned situations, especially in ever improving smart cities environments. In this letter, we present an unorthodox deception detection approach, based on hand gestures found in RGB videos of famous trials. The proposed system first extrapolates hands skeletons from the RGB sequences, then computes meaningful features which are summarized into Fisher Vectors (FVs), and finally feeds this representation to a Long-Short Term Memory (LSTM) network, defined Fisher-LSTM, to try and discern if a lie is being told. In the experimental results, we show how the FV representation can help a LSTM network grasp hand gestures characteristics that could otherwise be missed. What is more, the devised Fisher-LSTM, due to its real-time computation, can be employed in smart environments as an alternative lie detector in situations requiring an immediate response, such as the aforementioned law enforcement examples.