Lagrangian analysis on mechanism of forced convection heat transfer control for cylinder via deep reinforcement learning

Forced convection heat transfer plays a vital role in engineering; however, its control presents significant complexity. In this paper, a closed-loop deep reinforcement learning framework is introduced to optimize cooling tasks in a heat exchanger, where a cylindrical heat source is immersed in a narrow cavity. The online learning deep Q-networks (DQN) algorithm is implemented, and an offline learning conservative Q-learning soft actor-critic (CQL-SAC) algorithm is first proposed to learn based solely on preexisting databases without interacting with the environment. Taking the continuous blowing mode as the baseline, the CQL-SAC control obtains a temperature reduction of 49.1% greater than that under the optimal human control and consumes only 0.53% of the time required by online learning. While the DQN control achieves the best cooling performance, with a temperature reduction exceeding that of the optimal human control by 91.4%. The underlying mechanisms are analyzed. Particle tracking technology and Lagrangian coherence structures (LCS) are employed to identify regions of sufficient heat exchange and precisely map from where a cold particle can be captured to undergo sufficient heat exchange or swiftly escapes with inadequate heat exchange. The mechanism of the enhanced cooling effect under the DQN control is clarified from a particle capturing and escaping perspective. The greater overlap between the cold particle capture region and hotspots correlating with the more saddle points of the LCS within this region indicates more intense heat exchange in areas closer to hotspots, thus resulting in better cooling performance.