PRIVACY-PROTECTING DISTRIBUTED SELF-SUPERVISED LEARNING

Methods, systems, and apparatus, including medium-encoded computer program products, for receiving, from a first set of user devices, embedding statistics that were determined by the user devices using sets of one or more training pairs. Global embedding statistics can be determined, at least in part, using the embedding statistics, and transmitted to a second set of user devices. Local parameter model updates that were determined, at least in part, using the global embedding statistics can be received from the second set of user devices. Global model updates can be determined at least in part and using at least a subset of the local model updates. Global model updates can be transmitted to a third set of user devices.