Message-based distributed synchronization in a storage system

Described is a synchronization technique that may be used to coordinate processing between endpoints using the connecting message fabric. Processors in a data storage system communicate using the message switch of the message fabric. Each processor is an endpoint within a data storage system. A first endpoint may mark the beginning of the synchronization period by specifying a processing point at which other processors and the first endpoint are to coordinate from the perspective of the first endpoint. Synchronization is performed using local state information about the processing state of each endpoint as reported by each endpoint. The first endpoint waits for successful synchronization within a timeout period in accordance with the first endpoint's local state information. If successful synchronization does not occur prior to the timeout period, the first endpoint broadcasts a message with a new synchronization point to other endpoints. This indicates that the synchronization has failed marking the end of the synchronization period from perspective of the first endpoint.