On-line set-point optimization for intelligent supervisory control and improvement of Q-learning convergence

This paper proposes a design method of the Q-learning based intelligent supervisory control system (ISCS) for optimal operation of the three step kiln process and a new practical method for improvement of Q-learning convergence. First, the Q-learning based intelligent supervisory control system with two layer-structures is designed to find the on-line optimal set-points of control loops for the kiln process. Next, C4.5 is used to extract automatically the operational experience rules of the human operator from the historical data in the lower layer (i.e., process control layer) and the Q-function value is initialized by using the extracted rules in order to determine the optimal initial point of Q-learning in the higher layer (i.e., supervisory control layer). Hence, the convergence rate of Q-learning is extremely accelerated, so that the hierarchical ISCS can replace the human operator in the kiln process in which trial-and-error operation is not allowed. Through simulations and experiments, Q-learning convergence and the stability of the process operation have been evaluated sufficiently under the variable conditions of the states. •We proposed new method guaranteeing fast Q-learning convergence.•Method extracting experience rules from process database automatically was studied.•Optimal initialization algorithm of Q-value using experience rules was proposed.•Q-learning agent based Intelligent supervisory control system was designed.•Proposed system replaces human operator determining optimal set-points of controller.