Task execution based on real-world text detection for assistant systems

In one embodiment, a method includes accessing a visual signal including an image from a client system associated with a first user, the image depicting textual content in a real-world environment associated with the first user; identifying the text content based on a machine learning model and a visual signal; determining a context associated with the first user about a real-world environment based on the visual signal; executing a task for the first user, the task being determined based on the text content and the determined context; and sending an instruction to the client system for presenting the execution result of the task to the first user. 在一个实施例中，一种方法包括从与第一用户相关联的客户端系统访问包括图像的视觉信号，所述图像描绘了与第一用户相关联的真实世界环境中的文本内容；基于机器学习模型和视觉信号识别该文本内容；基于视觉信号确定与第一用户相关联的关于真实世界环境的上下文；为第一用户执行任务，所述任务是基于文本内容和所确定的上下文而确定的；以及向客户端系统发送用于向第一用户呈现任务的执行结果的指令。