Control of an autonomous vehicle

A vehicle controller is provided for determining a contingency action for a primary vehicle which can be driven in an autonomous mode and is also operable by a driver in a manual mode. The method comprises: determining failure of the driver to accept a request 32 to switch the primary vehicle to the manual mode when the vehicle is in the autonomous mode; determining a primary vehicle driving state 38 comprising position and velocity. The state and a signal indicating the failure are sent to a remote server or controller having a processor for determining and communicating the contingency action 46 for the primary vehicle. Data from secondary vehicles may be received by the remote server through vehicle-to-vehicle communication. The invention also provides the remote controller and provides for safe continued motion when manual take over has not occurred.