Generating and globally tuning application-specific machine learning accelerators

Methods, systems, and apparatus, including computer-readable media, are described for globally tuning and generating ML hardware accelerators. A design system selects an architecture representing a baseline processor configuration. An ML cost model of the system generates performance data about the architecture at least by modeling how the architecture executes computations of a neural network that includes multiple layers. Based on the performance data, the architecture is dynamically tuned to satisfy a performance objective when the architecture implements the neural network and executes machine-learning computations for a target application. In response to dynamically tuning the architecture, the system generates a configuration of an ML accelerator that specifies customized hardware configurations for implementing each of the multiple layers of the neural network.