Forced convection heat transfer control for cylinder via closed-loop continuous goal-oriented reinforcement learning

Forced convection heat transfer control offers considerable engineering value. This study focuses on a two-dimensional rapid temperature control problem in a heat exchange system, where a cylindrical heat source is immersed in a narrow cavity. First, a closed-loop continuous deep reinforcement learn...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Physics of fluids (1994) 2024-11, Vol.36 (11)
Hauptverfasser: Liu, Yangwei, Wang, Feitong, Zhao, Shihang, Tang, Yumeng
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page
container_issue 11
container_start_page
container_title Physics of fluids (1994)
container_volume 36
creator Liu, Yangwei
Wang, Feitong
Zhao, Shihang
Tang, Yumeng
description Forced convection heat transfer control offers considerable engineering value. This study focuses on a two-dimensional rapid temperature control problem in a heat exchange system, where a cylindrical heat source is immersed in a narrow cavity. First, a closed-loop continuous deep reinforcement learning (DRL) framework based on the deep deterministic policy gradient (DDPG) algorithm is developed. This framework swiftly achieves the target temperature with a temperature variance of 0.0116, which is only 5.7% of discrete frameworks. Particle tracking technology is used to analyze the evolution of flow and heat transfer under different control strategies. Due to the broader action space for exploration, continuous algorithms inherently excel in addressing delicate control issues. Furthermore, to address the deficiency that traditional DRL-based active flow control (AFC) frameworks require retraining with each goal changes and cost substantial computational resources to develop strategies for varied goals, the goal information is directly embedded into the agent, and the hindsight experience replay (HER) is employed to improve the training stability and sample efficiency. Then, a closed-loop continuous goal-oriented reinforcement learning (GoRL) framework based on the HER-DDPG algorithm is first proposed to perform real-time rapid temperature transition control and address multiple goals without retraining. Generalization tests show the proposed GoRL framework accomplishes multi-goal tasks with a temperature variance of 0.0121, which is only 5.8% of discrete frameworks, and consumes merely 11% of the computational resources compared with frameworks without goal-oriented capability. The GoRL framework greatly enhances the ability of AFC systems to handle multiple targets and time-varying goals.
doi_str_mv 10.1063/5.0239718
format Article
fullrecord <record><control><sourceid>proquest_scita</sourceid><recordid>TN_cdi_scitation_primary_10_1063_5_0239718</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>3130939189</sourcerecordid><originalsourceid>FETCH-LOGICAL-c182t-740ae6f0dd49aadc7367f8b779a37f7d710011a293e18fea2dd716422a6f36683</originalsourceid><addsrcrecordid>eNp9kE9LAzEQxYMoWKsHv8GCJ4WtyWabbI5SrAoFL3pexvypKWlSk2yh396s7dnTvHnz4w08hG4JnhHM6ON8hhsqOOnO0ITgTtScMXY-ao5rxii5RFcpbTDGVDRsgvIyRKlVJYPfa5lt8NW3hlzlCD4ZHcdDjsFVJhR9cNarYu4tVNKFpFXtQtj9QdYPYUjVOoCrQ7Ta5xIbtfVm_LAte-U0RG_9-hpdGHBJ35zmFH0unz8Wr_Xq_eVt8bSqJemaXPMWg2YGK9UKACU5Zdx0X5wLoNxwxQnGhEAjqCad0dCoYrG2aYAZylhHp-jumLuL4WfQKfebMERfXvaUUCyoIJ0o1P2RkjGkFLXpd9FuIR56gvux1H7en0ot7MORTdJmGOv6B_4Fj0Z4wQ</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>3130939189</pqid></control><display><type>article</type><title>Forced convection heat transfer control for cylinder via closed-loop continuous goal-oriented reinforcement learning</title><source>AIP Journals Complete</source><creator>Liu, Yangwei ; Wang, Feitong ; Zhao, Shihang ; Tang, Yumeng</creator><creatorcontrib>Liu, Yangwei ; Wang, Feitong ; Zhao, Shihang ; Tang, Yumeng</creatorcontrib><description>Forced convection heat transfer control offers considerable engineering value. This study focuses on a two-dimensional rapid temperature control problem in a heat exchange system, where a cylindrical heat source is immersed in a narrow cavity. First, a closed-loop continuous deep reinforcement learning (DRL) framework based on the deep deterministic policy gradient (DDPG) algorithm is developed. This framework swiftly achieves the target temperature with a temperature variance of 0.0116, which is only 5.7% of discrete frameworks. Particle tracking technology is used to analyze the evolution of flow and heat transfer under different control strategies. Due to the broader action space for exploration, continuous algorithms inherently excel in addressing delicate control issues. Furthermore, to address the deficiency that traditional DRL-based active flow control (AFC) frameworks require retraining with each goal changes and cost substantial computational resources to develop strategies for varied goals, the goal information is directly embedded into the agent, and the hindsight experience replay (HER) is employed to improve the training stability and sample efficiency. Then, a closed-loop continuous goal-oriented reinforcement learning (GoRL) framework based on the HER-DDPG algorithm is first proposed to perform real-time rapid temperature transition control and address multiple goals without retraining. Generalization tests show the proposed GoRL framework accomplishes multi-goal tasks with a temperature variance of 0.0121, which is only 5.8% of discrete frameworks, and consumes merely 11% of the computational resources compared with frameworks without goal-oriented capability. The GoRL framework greatly enhances the ability of AFC systems to handle multiple targets and time-varying goals.</description><identifier>ISSN: 1070-6631</identifier><identifier>EISSN: 1089-7666</identifier><identifier>DOI: 10.1063/5.0239718</identifier><identifier>CODEN: PHFLE6</identifier><language>eng</language><publisher>Melville: American Institute of Physics</publisher><subject>Algorithms ; Closed loops ; Deep learning ; Flow control ; Forced convection ; Heat exchange ; Heat transfer ; Machine learning ; Particle tracking ; Real time ; Technology assessment ; Temperature control ; Tracking</subject><ispartof>Physics of fluids (1994), 2024-11, Vol.36 (11)</ispartof><rights>Author(s)</rights><rights>2024 Author(s). Published under an exclusive license by AIP Publishing.</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><cites>FETCH-LOGICAL-c182t-740ae6f0dd49aadc7367f8b779a37f7d710011a293e18fea2dd716422a6f36683</cites><orcidid>0009-0008-2941-334X ; 0000-0001-9653-5760 ; 0009-0008-5242-088X ; 0000-0002-2131-3810</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>314,776,780,790,4498,27901,27902</link.rule.ids></links><search><creatorcontrib>Liu, Yangwei</creatorcontrib><creatorcontrib>Wang, Feitong</creatorcontrib><creatorcontrib>Zhao, Shihang</creatorcontrib><creatorcontrib>Tang, Yumeng</creatorcontrib><title>Forced convection heat transfer control for cylinder via closed-loop continuous goal-oriented reinforcement learning</title><title>Physics of fluids (1994)</title><description>Forced convection heat transfer control offers considerable engineering value. This study focuses on a two-dimensional rapid temperature control problem in a heat exchange system, where a cylindrical heat source is immersed in a narrow cavity. First, a closed-loop continuous deep reinforcement learning (DRL) framework based on the deep deterministic policy gradient (DDPG) algorithm is developed. This framework swiftly achieves the target temperature with a temperature variance of 0.0116, which is only 5.7% of discrete frameworks. Particle tracking technology is used to analyze the evolution of flow and heat transfer under different control strategies. Due to the broader action space for exploration, continuous algorithms inherently excel in addressing delicate control issues. Furthermore, to address the deficiency that traditional DRL-based active flow control (AFC) frameworks require retraining with each goal changes and cost substantial computational resources to develop strategies for varied goals, the goal information is directly embedded into the agent, and the hindsight experience replay (HER) is employed to improve the training stability and sample efficiency. Then, a closed-loop continuous goal-oriented reinforcement learning (GoRL) framework based on the HER-DDPG algorithm is first proposed to perform real-time rapid temperature transition control and address multiple goals without retraining. Generalization tests show the proposed GoRL framework accomplishes multi-goal tasks with a temperature variance of 0.0121, which is only 5.8% of discrete frameworks, and consumes merely 11% of the computational resources compared with frameworks without goal-oriented capability. The GoRL framework greatly enhances the ability of AFC systems to handle multiple targets and time-varying goals.</description><subject>Algorithms</subject><subject>Closed loops</subject><subject>Deep learning</subject><subject>Flow control</subject><subject>Forced convection</subject><subject>Heat exchange</subject><subject>Heat transfer</subject><subject>Machine learning</subject><subject>Particle tracking</subject><subject>Real time</subject><subject>Technology assessment</subject><subject>Temperature control</subject><subject>Tracking</subject><issn>1070-6631</issn><issn>1089-7666</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><recordid>eNp9kE9LAzEQxYMoWKsHv8GCJ4WtyWabbI5SrAoFL3pexvypKWlSk2yh396s7dnTvHnz4w08hG4JnhHM6ON8hhsqOOnO0ITgTtScMXY-ao5rxii5RFcpbTDGVDRsgvIyRKlVJYPfa5lt8NW3hlzlCD4ZHcdDjsFVJhR9cNarYu4tVNKFpFXtQtj9QdYPYUjVOoCrQ7Ta5xIbtfVm_LAte-U0RG_9-hpdGHBJ35zmFH0unz8Wr_Xq_eVt8bSqJemaXPMWg2YGK9UKACU5Zdx0X5wLoNxwxQnGhEAjqCad0dCoYrG2aYAZylhHp-jumLuL4WfQKfebMERfXvaUUCyoIJ0o1P2RkjGkFLXpd9FuIR56gvux1H7en0ot7MORTdJmGOv6B_4Fj0Z4wQ</recordid><startdate>202411</startdate><enddate>202411</enddate><creator>Liu, Yangwei</creator><creator>Wang, Feitong</creator><creator>Zhao, Shihang</creator><creator>Tang, Yumeng</creator><general>American Institute of Physics</general><scope>AAYXX</scope><scope>CITATION</scope><scope>8FD</scope><scope>H8D</scope><scope>L7M</scope><orcidid>https://orcid.org/0009-0008-2941-334X</orcidid><orcidid>https://orcid.org/0000-0001-9653-5760</orcidid><orcidid>https://orcid.org/0009-0008-5242-088X</orcidid><orcidid>https://orcid.org/0000-0002-2131-3810</orcidid></search><sort><creationdate>202411</creationdate><title>Forced convection heat transfer control for cylinder via closed-loop continuous goal-oriented reinforcement learning</title><author>Liu, Yangwei ; Wang, Feitong ; Zhao, Shihang ; Tang, Yumeng</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c182t-740ae6f0dd49aadc7367f8b779a37f7d710011a293e18fea2dd716422a6f36683</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Algorithms</topic><topic>Closed loops</topic><topic>Deep learning</topic><topic>Flow control</topic><topic>Forced convection</topic><topic>Heat exchange</topic><topic>Heat transfer</topic><topic>Machine learning</topic><topic>Particle tracking</topic><topic>Real time</topic><topic>Technology assessment</topic><topic>Temperature control</topic><topic>Tracking</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Liu, Yangwei</creatorcontrib><creatorcontrib>Wang, Feitong</creatorcontrib><creatorcontrib>Zhao, Shihang</creatorcontrib><creatorcontrib>Tang, Yumeng</creatorcontrib><collection>CrossRef</collection><collection>Technology Research Database</collection><collection>Aerospace Database</collection><collection>Advanced Technologies Database with Aerospace</collection><jtitle>Physics of fluids (1994)</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Liu, Yangwei</au><au>Wang, Feitong</au><au>Zhao, Shihang</au><au>Tang, Yumeng</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Forced convection heat transfer control for cylinder via closed-loop continuous goal-oriented reinforcement learning</atitle><jtitle>Physics of fluids (1994)</jtitle><date>2024-11</date><risdate>2024</risdate><volume>36</volume><issue>11</issue><issn>1070-6631</issn><eissn>1089-7666</eissn><coden>PHFLE6</coden><abstract>Forced convection heat transfer control offers considerable engineering value. This study focuses on a two-dimensional rapid temperature control problem in a heat exchange system, where a cylindrical heat source is immersed in a narrow cavity. First, a closed-loop continuous deep reinforcement learning (DRL) framework based on the deep deterministic policy gradient (DDPG) algorithm is developed. This framework swiftly achieves the target temperature with a temperature variance of 0.0116, which is only 5.7% of discrete frameworks. Particle tracking technology is used to analyze the evolution of flow and heat transfer under different control strategies. Due to the broader action space for exploration, continuous algorithms inherently excel in addressing delicate control issues. Furthermore, to address the deficiency that traditional DRL-based active flow control (AFC) frameworks require retraining with each goal changes and cost substantial computational resources to develop strategies for varied goals, the goal information is directly embedded into the agent, and the hindsight experience replay (HER) is employed to improve the training stability and sample efficiency. Then, a closed-loop continuous goal-oriented reinforcement learning (GoRL) framework based on the HER-DDPG algorithm is first proposed to perform real-time rapid temperature transition control and address multiple goals without retraining. Generalization tests show the proposed GoRL framework accomplishes multi-goal tasks with a temperature variance of 0.0121, which is only 5.8% of discrete frameworks, and consumes merely 11% of the computational resources compared with frameworks without goal-oriented capability. The GoRL framework greatly enhances the ability of AFC systems to handle multiple targets and time-varying goals.</abstract><cop>Melville</cop><pub>American Institute of Physics</pub><doi>10.1063/5.0239718</doi><tpages>17</tpages><orcidid>https://orcid.org/0009-0008-2941-334X</orcidid><orcidid>https://orcid.org/0000-0001-9653-5760</orcidid><orcidid>https://orcid.org/0009-0008-5242-088X</orcidid><orcidid>https://orcid.org/0000-0002-2131-3810</orcidid></addata></record>
fulltext fulltext
identifier ISSN: 1070-6631
ispartof Physics of fluids (1994), 2024-11, Vol.36 (11)
issn 1070-6631
1089-7666
language eng
recordid cdi_scitation_primary_10_1063_5_0239718
source AIP Journals Complete
subjects Algorithms
Closed loops
Deep learning
Flow control
Forced convection
Heat exchange
Heat transfer
Machine learning
Particle tracking
Real time
Technology assessment
Temperature control
Tracking
title Forced convection heat transfer control for cylinder via closed-loop continuous goal-oriented reinforcement learning
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-02-03T17%3A00%3A21IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_scita&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Forced%20convection%20heat%20transfer%20control%20for%20cylinder%20via%20closed-loop%20continuous%20goal-oriented%20reinforcement%20learning&rft.jtitle=Physics%20of%20fluids%20(1994)&rft.au=Liu,%20Yangwei&rft.date=2024-11&rft.volume=36&rft.issue=11&rft.issn=1070-6631&rft.eissn=1089-7666&rft.coden=PHFLE6&rft_id=info:doi/10.1063/5.0239718&rft_dat=%3Cproquest_scita%3E3130939189%3C/proquest_scita%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=3130939189&rft_id=info:pmid/&rfr_iscdi=true