ReflecTouch: Utilizing Corneal Reflection Images for Smartphone Grasp Posture Detection

Sensing how a user holds a smartphone enables adaptive user interfaces, such as those that automatically change the displayed content and position of graphical user interface (GUI) components based on how the phone is held. We propose ReflecTouch, a novel method to detect how a smartphone is being held by capturing images of the smartphone screen reflected on the cornea with a built-in front camera. In these images, the areas where the user places their fingers on the screen appear as shadows, which makes it possible to estimate the grasp posture. Since most smartphones have a front camera, this method can be used regardless of the device model; in addition, no additional sensor or hardware is required. In this paper, we report the identification accuracy of the proposed method through laboratory experiments under controlled conditions and crowdsourcing experiments that are closer to the actual situations.