Distributed adaptive moment estimation method with dynamic learning rate boundary

Optimization methods based on adaptive gradient, such as ADAGRAD, RMSPROP, ADAM and the like, are widely applied to solving the problem of large-scale machine learning including deep learning. In existing work, many solutions have been provided for the communication parallelization problem of peripheral nodes and center nodes, but the communication cost is often high. In addition, an existing method is generally poor in generalization ability, and even convergence cannot be achieved due to instability and an extreme learning rate. In order to solve the existing problems, a new distributed adaptive moment estimation method (DADBOUND) with a dynamic learning rate boundary is developed and is used for carrying out online optimization on a decentralized network, so that data parallelization and decentralized calculation are realized. And the method utilizes the dynamic range of the learning rate to realize progressive stable transition from the adaptive method to the DSGD so as to eliminate the generalization gap