Neural network model using peer-to-peer attention

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing network inputs using neural networks to generate network outputs. In one aspect, a method includes processing a network input using a neural network to generate a network output, where the neural network has a plurality of blocks, where each block is configured to process a block input to generate a block output, the method including: for each target block of the neural network, generating an attention weighted representation of a plurality of first block outputs; for each first block output, the method includes: processing a plurality of second block outputs to generate an attention factor; and generating an attention weighted representation for each first block output by applying the respective attention factor to the corresponding first block output; and generating a target block input from the attention weighted representation; and processing the target block input using the target block to ge