Forced convection heat transfer control for cylinder via closed-loop continuous goal-oriented reinforcement learning
Forced convection heat transfer control offers considerable engineering value. This study focuses on a two-dimensional rapid temperature control problem in a heat exchange system, where a cylindrical heat source is immersed in a narrow cavity. First, a closed-loop continuous deep reinforcement learn...
Gespeichert in:
Veröffentlicht in: | Physics of fluids (1994) 2024-11, Vol.36 (11) |
---|---|
Hauptverfasser: | , , , |
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | |
---|---|
container_issue | 11 |
container_start_page | |
container_title | Physics of fluids (1994) |
container_volume | 36 |
creator | Liu, Yangwei Wang, Feitong Zhao, Shihang Tang, Yumeng |
description | Forced convection heat transfer control offers considerable engineering value. This study focuses on a two-dimensional rapid temperature control problem in a heat exchange system, where a cylindrical heat source is immersed in a narrow cavity. First, a closed-loop continuous deep reinforcement learning (DRL) framework based on the deep deterministic policy gradient (DDPG) algorithm is developed. This framework swiftly achieves the target temperature with a temperature variance of 0.0116, which is only 5.7% of discrete frameworks. Particle tracking technology is used to analyze the evolution of flow and heat transfer under different control strategies. Due to the broader action space for exploration, continuous algorithms inherently excel in addressing delicate control issues. Furthermore, to address the deficiency that traditional DRL-based active flow control (AFC) frameworks require retraining with each goal changes and cost substantial computational resources to develop strategies for varied goals, the goal information is directly embedded into the agent, and the hindsight experience replay (HER) is employed to improve the training stability and sample efficiency. Then, a closed-loop continuous goal-oriented reinforcement learning (GoRL) framework based on the HER-DDPG algorithm is first proposed to perform real-time rapid temperature transition control and address multiple goals without retraining. Generalization tests show the proposed GoRL framework accomplishes multi-goal tasks with a temperature variance of 0.0121, which is only 5.8% of discrete frameworks, and consumes merely 11% of the computational resources compared with frameworks without goal-oriented capability. The GoRL framework greatly enhances the ability of AFC systems to handle multiple targets and time-varying goals. |
doi_str_mv | 10.1063/5.0239718 |
format | Article |
fullrecord | <record><control><sourceid>proquest_scita</sourceid><recordid>TN_cdi_scitation_primary_10_1063_5_0239718</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>3130939189</sourcerecordid><originalsourceid>FETCH-LOGICAL-c182t-740ae6f0dd49aadc7367f8b779a37f7d710011a293e18fea2dd716422a6f36683</originalsourceid><addsrcrecordid>eNp9kE9LAzEQxYMoWKsHv8GCJ4WtyWabbI5SrAoFL3pexvypKWlSk2yh396s7dnTvHnz4w08hG4JnhHM6ON8hhsqOOnO0ITgTtScMXY-ao5rxii5RFcpbTDGVDRsgvIyRKlVJYPfa5lt8NW3hlzlCD4ZHcdDjsFVJhR9cNarYu4tVNKFpFXtQtj9QdYPYUjVOoCrQ7Ta5xIbtfVm_LAte-U0RG_9-hpdGHBJ35zmFH0unz8Wr_Xq_eVt8bSqJemaXPMWg2YGK9UKACU5Zdx0X5wLoNxwxQnGhEAjqCad0dCoYrG2aYAZylhHp-jumLuL4WfQKfebMERfXvaUUCyoIJ0o1P2RkjGkFLXpd9FuIR56gvux1H7en0ot7MORTdJmGOv6B_4Fj0Z4wQ</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>3130939189</pqid></control><display><type>article</type><title>Forced convection heat transfer control for cylinder via closed-loop continuous goal-oriented reinforcement learning</title><source>AIP Journals Complete</source><creator>Liu, Yangwei ; Wang, Feitong ; Zhao, Shihang ; Tang, Yumeng</creator><creatorcontrib>Liu, Yangwei ; Wang, Feitong ; Zhao, Shihang ; Tang, Yumeng</creatorcontrib><description>Forced convection heat transfer control offers considerable engineering value. This study focuses on a two-dimensional rapid temperature control problem in a heat exchange system, where a cylindrical heat source is immersed in a narrow cavity. First, a closed-loop continuous deep reinforcement learning (DRL) framework based on the deep deterministic policy gradient (DDPG) algorithm is developed. This framework swiftly achieves the target temperature with a temperature variance of 0.0116, which is only 5.7% of discrete frameworks. Particle tracking technology is used to analyze the evolution of flow and heat transfer under different control strategies. Due to the broader action space for exploration, continuous algorithms inherently excel in addressing delicate control issues. Furthermore, to address the deficiency that traditional DRL-based active flow control (AFC) frameworks require retraining with each goal changes and cost substantial computational resources to develop strategies for varied goals, the goal information is directly embedded into the agent, and the hindsight experience replay (HER) is employed to improve the training stability and sample efficiency. Then, a closed-loop continuous goal-oriented reinforcement learning (GoRL) framework based on the HER-DDPG algorithm is first proposed to perform real-time rapid temperature transition control and address multiple goals without retraining. Generalization tests show the proposed GoRL framework accomplishes multi-goal tasks with a temperature variance of 0.0121, which is only 5.8% of discrete frameworks, and consumes merely 11% of the computational resources compared with frameworks without goal-oriented capability. The GoRL framework greatly enhances the ability of AFC systems to handle multiple targets and time-varying goals.</description><identifier>ISSN: 1070-6631</identifier><identifier>EISSN: 1089-7666</identifier><identifier>DOI: 10.1063/5.0239718</identifier><identifier>CODEN: PHFLE6</identifier><language>eng</language><publisher>Melville: American Institute of Physics</publisher><subject>Algorithms ; Closed loops ; Deep learning ; Flow control ; Forced convection ; Heat exchange ; Heat transfer ; Machine learning ; Particle tracking ; Real time ; Technology assessment ; Temperature control ; Tracking</subject><ispartof>Physics of fluids (1994), 2024-11, Vol.36 (11)</ispartof><rights>Author(s)</rights><rights>2024 Author(s). Published under an exclusive license by AIP Publishing.</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><cites>FETCH-LOGICAL-c182t-740ae6f0dd49aadc7367f8b779a37f7d710011a293e18fea2dd716422a6f36683</cites><orcidid>0009-0008-2941-334X ; 0000-0001-9653-5760 ; 0009-0008-5242-088X ; 0000-0002-2131-3810</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>314,776,780,790,4498,27901,27902</link.rule.ids></links><search><creatorcontrib>Liu, Yangwei</creatorcontrib><creatorcontrib>Wang, Feitong</creatorcontrib><creatorcontrib>Zhao, Shihang</creatorcontrib><creatorcontrib>Tang, Yumeng</creatorcontrib><title>Forced convection heat transfer control for cylinder via closed-loop continuous goal-oriented reinforcement learning</title><title>Physics of fluids (1994)</title><description>Forced convection heat transfer control offers considerable engineering value. This study focuses on a two-dimensional rapid temperature control problem in a heat exchange system, where a cylindrical heat source is immersed in a narrow cavity. First, a closed-loop continuous deep reinforcement learning (DRL) framework based on the deep deterministic policy gradient (DDPG) algorithm is developed. This framework swiftly achieves the target temperature with a temperature variance of 0.0116, which is only 5.7% of discrete frameworks. Particle tracking technology is used to analyze the evolution of flow and heat transfer under different control strategies. Due to the broader action space for exploration, continuous algorithms inherently excel in addressing delicate control issues. Furthermore, to address the deficiency that traditional DRL-based active flow control (AFC) frameworks require retraining with each goal changes and cost substantial computational resources to develop strategies for varied goals, the goal information is directly embedded into the agent, and the hindsight experience replay (HER) is employed to improve the training stability and sample efficiency. Then, a closed-loop continuous goal-oriented reinforcement learning (GoRL) framework based on the HER-DDPG algorithm is first proposed to perform real-time rapid temperature transition control and address multiple goals without retraining. Generalization tests show the proposed GoRL framework accomplishes multi-goal tasks with a temperature variance of 0.0121, which is only 5.8% of discrete frameworks, and consumes merely 11% of the computational resources compared with frameworks without goal-oriented capability. The GoRL framework greatly enhances the ability of AFC systems to handle multiple targets and time-varying goals.</description><subject>Algorithms</subject><subject>Closed loops</subject><subject>Deep learning</subject><subject>Flow control</subject><subject>Forced convection</subject><subject>Heat exchange</subject><subject>Heat transfer</subject><subject>Machine learning</subject><subject>Particle tracking</subject><subject>Real time</subject><subject>Technology assessment</subject><subject>Temperature control</subject><subject>Tracking</subject><issn>1070-6631</issn><issn>1089-7666</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><recordid>eNp9kE9LAzEQxYMoWKsHv8GCJ4WtyWabbI5SrAoFL3pexvypKWlSk2yh396s7dnTvHnz4w08hG4JnhHM6ON8hhsqOOnO0ITgTtScMXY-ao5rxii5RFcpbTDGVDRsgvIyRKlVJYPfa5lt8NW3hlzlCD4ZHcdDjsFVJhR9cNarYu4tVNKFpFXtQtj9QdYPYUjVOoCrQ7Ta5xIbtfVm_LAte-U0RG_9-hpdGHBJ35zmFH0unz8Wr_Xq_eVt8bSqJemaXPMWg2YGK9UKACU5Zdx0X5wLoNxwxQnGhEAjqCad0dCoYrG2aYAZylhHp-jumLuL4WfQKfebMERfXvaUUCyoIJ0o1P2RkjGkFLXpd9FuIR56gvux1H7en0ot7MORTdJmGOv6B_4Fj0Z4wQ</recordid><startdate>202411</startdate><enddate>202411</enddate><creator>Liu, Yangwei</creator><creator>Wang, Feitong</creator><creator>Zhao, Shihang</creator><creator>Tang, Yumeng</creator><general>American Institute of Physics</general><scope>AAYXX</scope><scope>CITATION</scope><scope>8FD</scope><scope>H8D</scope><scope>L7M</scope><orcidid>https://orcid.org/0009-0008-2941-334X</orcidid><orcidid>https://orcid.org/0000-0001-9653-5760</orcidid><orcidid>https://orcid.org/0009-0008-5242-088X</orcidid><orcidid>https://orcid.org/0000-0002-2131-3810</orcidid></search><sort><creationdate>202411</creationdate><title>Forced convection heat transfer control for cylinder via closed-loop continuous goal-oriented reinforcement learning</title><author>Liu, Yangwei ; Wang, Feitong ; Zhao, Shihang ; Tang, Yumeng</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c182t-740ae6f0dd49aadc7367f8b779a37f7d710011a293e18fea2dd716422a6f36683</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Algorithms</topic><topic>Closed loops</topic><topic>Deep learning</topic><topic>Flow control</topic><topic>Forced convection</topic><topic>Heat exchange</topic><topic>Heat transfer</topic><topic>Machine learning</topic><topic>Particle tracking</topic><topic>Real time</topic><topic>Technology assessment</topic><topic>Temperature control</topic><topic>Tracking</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Liu, Yangwei</creatorcontrib><creatorcontrib>Wang, Feitong</creatorcontrib><creatorcontrib>Zhao, Shihang</creatorcontrib><creatorcontrib>Tang, Yumeng</creatorcontrib><collection>CrossRef</collection><collection>Technology Research Database</collection><collection>Aerospace Database</collection><collection>Advanced Technologies Database with Aerospace</collection><jtitle>Physics of fluids (1994)</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Liu, Yangwei</au><au>Wang, Feitong</au><au>Zhao, Shihang</au><au>Tang, Yumeng</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Forced convection heat transfer control for cylinder via closed-loop continuous goal-oriented reinforcement learning</atitle><jtitle>Physics of fluids (1994)</jtitle><date>2024-11</date><risdate>2024</risdate><volume>36</volume><issue>11</issue><issn>1070-6631</issn><eissn>1089-7666</eissn><coden>PHFLE6</coden><abstract>Forced convection heat transfer control offers considerable engineering value. This study focuses on a two-dimensional rapid temperature control problem in a heat exchange system, where a cylindrical heat source is immersed in a narrow cavity. First, a closed-loop continuous deep reinforcement learning (DRL) framework based on the deep deterministic policy gradient (DDPG) algorithm is developed. This framework swiftly achieves the target temperature with a temperature variance of 0.0116, which is only 5.7% of discrete frameworks. Particle tracking technology is used to analyze the evolution of flow and heat transfer under different control strategies. Due to the broader action space for exploration, continuous algorithms inherently excel in addressing delicate control issues. Furthermore, to address the deficiency that traditional DRL-based active flow control (AFC) frameworks require retraining with each goal changes and cost substantial computational resources to develop strategies for varied goals, the goal information is directly embedded into the agent, and the hindsight experience replay (HER) is employed to improve the training stability and sample efficiency. Then, a closed-loop continuous goal-oriented reinforcement learning (GoRL) framework based on the HER-DDPG algorithm is first proposed to perform real-time rapid temperature transition control and address multiple goals without retraining. Generalization tests show the proposed GoRL framework accomplishes multi-goal tasks with a temperature variance of 0.0121, which is only 5.8% of discrete frameworks, and consumes merely 11% of the computational resources compared with frameworks without goal-oriented capability. The GoRL framework greatly enhances the ability of AFC systems to handle multiple targets and time-varying goals.</abstract><cop>Melville</cop><pub>American Institute of Physics</pub><doi>10.1063/5.0239718</doi><tpages>17</tpages><orcidid>https://orcid.org/0009-0008-2941-334X</orcidid><orcidid>https://orcid.org/0000-0001-9653-5760</orcidid><orcidid>https://orcid.org/0009-0008-5242-088X</orcidid><orcidid>https://orcid.org/0000-0002-2131-3810</orcidid></addata></record> |
fulltext | fulltext |
identifier | ISSN: 1070-6631 |
ispartof | Physics of fluids (1994), 2024-11, Vol.36 (11) |
issn | 1070-6631 1089-7666 |
language | eng |
recordid | cdi_scitation_primary_10_1063_5_0239718 |
source | AIP Journals Complete |
subjects | Algorithms Closed loops Deep learning Flow control Forced convection Heat exchange Heat transfer Machine learning Particle tracking Real time Technology assessment Temperature control Tracking |
title | Forced convection heat transfer control for cylinder via closed-loop continuous goal-oriented reinforcement learning |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-02-03T17%3A00%3A21IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_scita&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Forced%20convection%20heat%20transfer%20control%20for%20cylinder%20via%20closed-loop%20continuous%20goal-oriented%20reinforcement%20learning&rft.jtitle=Physics%20of%20fluids%20(1994)&rft.au=Liu,%20Yangwei&rft.date=2024-11&rft.volume=36&rft.issue=11&rft.issn=1070-6631&rft.eissn=1089-7666&rft.coden=PHFLE6&rft_id=info:doi/10.1063/5.0239718&rft_dat=%3Cproquest_scita%3E3130939189%3C/proquest_scita%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=3130939189&rft_id=info:pmid/&rfr_iscdi=true |