Task execution based on real-world text detection for assistant systems

In one embodiment, a method includes accessing visual signals comprising images portraying textual content in a real-world environment associated with a first user from a client system associated with the first user, recognizing the textual content based on machine-learning models and the visual signals, determining a context associated with the first user with respect to the real-world environment based on the visual signals, executing tasks determined based on the textual content and the determined context for the first user, and sending instructions for presenting execution results of the tasks to the first user to the client system.