Deep reinforcement learning and adaptive policy transfer for generalizable well control optimization

Well control optimization is a challenging task but plays a critical role in reservoir management. Traditional methods independently solve each task from scratch and the obtained scheme is only applicable to the environment where the optimization process is run. In stark contrast, human experts are adept at learning and building generalizable skills and using them to efficiently draw inferences and make decisions for similar scenarios. Inspired by the recently proposed generalizable field development optimization approach, this work presents an adaptive and robust deep learning-based Representation-Decision-Transfer (DLRDT) framework to deal with the generalization problem in well control optimization. Specifically, DLRDT uses a three-stage workflow to train an artificial agent. First, the agent develops its vision and understands its surroundings by learning a latent state representation with domain adaptation techniques. Second, the agent is tasked with using high-performance deep reinforcement learning algorithms to train the optimal control policy in the latent state space. Finally, the agent is transferred and evaluated in several environments that were not seen during the training. Compared with previous methods that optimize a solution for a specific scenario, our approach trains a policy that is not only robust to variations in their environments but can adapt to unseen (but similar) environments without additional training. For a demonstration, we validate the proposed framework on waterflooding well control optimization problems. Experimental evaluations on two three-dimensional reservoir models demonstrate the trained agent has excellent optimization efficiency and generalization performance. Our approach is particularly favorable when considering the deployment of schemes in the real world as it can handle unforeseen situations. •A novel well control optimization framework that incorporates deep reinforcement learning and policy transfer techniques is proposed.•We uses a three-stage workflow to train an agent that learns the optimal control policy and adapts to new environments.•DLRDT is particularly favorable when considering the deployment in the real world as it can handle unforeseen situations.•DLRDT achieves excellent optimization efficiency and generalization performance on two 3D reservoir models.