Parallel Memories for Multidimensional Data Access

The subject disclosure is directed towards loading parallel memories (e.g., in one or more FPGAs) with multidimensional data in an interleaved manner such that a multidimensional patch/window may be filled with corresponding data in a single parallel read of the memories. Depending on the position of the patch, the data may be rotated horizontally and/or vertically, for example, so that the data in each patch is consistently arranged in the patch regardless of from which memory each piece of data was read. Also described is leveraging dual ported memory for multiple line reads and/or loading one part of a buffer while reading from another.