A new relaying approach for protecting TCSC compensated transmission line connected to DFIG based wind farm

•Investigate the potential impacts of TCSC and DFIG wind farms on a conventional distance relaying-based transmission line protection scheme.•A novel fault detection algorithm for TCSC-compensated transmission lines connecting DFIG wind farms is developed. The proposed method uses the Complete Ensemble Empirical Mode Decomposition with Adaptive Noise-based Dominant Mode Algorithm and Hilbert Transform (CEEMDAN-DMA-HT) technique.•A novel fault classifier is developed for TCSC-compensated transmission lines linking DFIG wind farms. The proposed approach employs the Complete Ensemble Empirical Mode Decomposition with Adaptive Noise (CEEMDAN) to assist the Enhanced Jaya tuned Random Vector Functional Link Network (EJAYA-RVFL) in accurately classifying faults.•The effectiveness of the proposed relaying algorithm is assessed on a test system featuring a variety of potential faults and non-fault events. This evaluation thoroughly examines the algorithm's performance and reliability under different operating conditions and scenarios.•The proposed relaying algorithm is validated for the presence of the noise, double circuit lines, change in WF units rating, CT saturation, and different sampling frequency etc.•A list of features are computed using energy of phase and ground IMF which is used to train the EJAYA-RVFL classifier. In modern power systems, Thyristor Controlled Series Capacitor (TCSC)-compensated transmission lines are crucial in transporting bulk power from large wind farms. However, the performance of conventional distance relays is adversely impacted by the typical fault characteristics of the Doubly Fed Induction Generator (DFIG) and TCSC. To address this challenge, this paper presents a new relaying algorithm such as Complete Ensemble Empirical Mode Decomposition with Adaptive Noise-based Dominant Mode Algorithm-Hilbert Transform (CEEMDAN-DMA-HT) for fault detection and CEEMDAN assisted Enhanced Jaya Optimization-based Random Vector Functional Link Network (EJAYA-RVFL) for fault classification. In proposed fault detection algorithm, the differential current from both ends of the line is subjected to Complete Ensemble Empirical Mode Decomposition with Adaptive Noise, leading to the extraction of distinct intrinsic mode functions (IMFs). Employing the Dominant Mode Algorithm, the IMF with the highest Pearson correlation coefficient, referred to as the dominant IMF, is identified. Subsequently, a comparison between this dominant mode IMF and the origina