Method, apparatus and system for encoding and decoding a coding tree unit

Abstract METHOD, APPARATUS AND SYSTEM FOR ENCODING AND DECODING A CODING TREE UNIT A system and method of decoding a coding unit from a coding tree unit of an image frame from a video bitstream. The method comprises determining a scan pattern for a transform block of the coding unit, wherein the scan pattern traverses the transform block by progressing through a plurality of non-overlapping collections of sub-blocks of residual coefficients, the scan pattern progressing from a current collection to a next collection of the plurality of collections after completing scanning of the current collection; decoding residual coefficients from the video bitstream according to the determined scan pattern; determining a multiple transform selection index for the coding unit, the determining comprising: decoding (1676) the multiple transform selection index from the video bitstream if a last significant coefficient encountered along the scan pattern is at or within a threshold cartesian location of the transform block, and determining (1674) the multiple transform selection index to indicate that multiple transform selection is not used if the last significant residual coefficient position of the transform block along the scan pattern outside the threshold cartesian location; and transforming the decoded residual coefficients by applying a transform according to the multiple transform selection index to decode the coding unit 23954677_1