Non-volatile storage system with integrated compute engine and optimized use of local fast memory

A memory system (e.g. a solid state drive) includes one or more non-volatile memory die, a controller in communication with the memory die, a local memory connected to (or part of) the controller and a compute engine inside the memory system that is near the location of the data and can be used to perform common data manipulation operations. When the memory system receives instructions to use the compute engine to perform data manipulation operations, the local memory is reallocated such that an amount of space allocated in the local memory for logical to physical translation information is changed based on the one or more data manipulation instructions.