Generating sequences of data elements using cross attention operations

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a sequence of data elements that includes respective data elements at each location in a sequence of locations. In one aspect, a method includes, for each position after a first position...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: HAWTHORNE CURTIS GLENN-MCVEY, BASTIEN, VINYALS ORIOL, BOTVINIK, MATTHEW, KENGIA CATALINA-CODRUTA, SIMON, IAN, STEWART, JAEGLE ANDREW KURT, CARREIRA JOAO, DIELEMAN, SANDOR, ETIENNE, LEA, ZEGEDOR NEIL, MALINOWSKI MATEUSZ, SHEEHAN HANNAH RACHEL, NASH CHARLES THOMAS CURTIS, ALLAC JEAN-BAPTISTE, BORJODIT AWOCA, S
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a sequence of data elements that includes respective data elements at each location in a sequence of locations. In one aspect, a method includes, for each position after a first position in a sequence of positions: obtaining a current sequence of data element inlays comprising a respective data element inlay for each data element at a position prior to the current position, obtaining a potential inlay sequence, and processing, using a neural network, (i) the current sequence of data elements embedded, and (ii) the potentially embedded sequence to generate the data elements at the current location. The neural network comprises a sequence of neural network blocks comprising: (i) a cross-attention block, (ii) one or more self-attention blocks, and (iii) an output block. 用于生成包括在位置序列中的每个位置处的相应数据元素的数据元素序列的方法、系统和装置,其包括在计算机存储介质上编码的计算机程序。在一个方面,方法包括:针对在位置序列中的第一位置之后的每个位置:获得包括在当前位置之前的位置处的每个数据元素的相应数据元素嵌入的数据元素嵌