PROGRAMMABLE MATRIX PROCESSING ENGINE

In one embodiment, an integrated circuit (IC) chip is provided. The IC chip comprises: a plurality of processing units to collectively execute a matrix multiplication operation with matrix data by performing matrix processing at least partially in parallel, each processing unit of the plurality of processing units to process a portion of the matrix data to perform a corresponding partial matrix operation; a plurality of memories, each memory to store the portion of the matrix data to be processed by a corresponding processing unit of the plurality of processing units; a plurality of interconnects to couple the plurality of processing units, a processing unit of the plurality of processing units to send partial matrix data to a neighboring processing unit of the plurality of processing units or to receive partial matrix data from the neighboring processing unit over a corresponding interconnect; a first controller, wherein responsive to the first controller, the plurality of processing units are to collectively execute the matrix multiplication operation in accordance with at least one matrix multiplication command or instruction specifying a first input matrix and a second input matrix, the plurality of processing units to produce an output matrix by multiplying the first input matrix and the second input matrix; and a plurality of second controllers, each second controller associated with a processing unit of the plurality of processing units, the second controller to retrieve the portion of the matrix data to be processed by a corresponding processing unit of the plurality of processing units from a system memory and to store the portion of the matrix data to a corresponding memory of the plurality of memories.