Deduplication method, device and equipment based on perceptual neural network and storage medium

The invention provides a duplicate removal method and device based on a perception neural network, equipment and a storage medium. Text macroscopic features of a to-be-stored text are determined by inputting the to-be-stored text into an overall perception layer, and a first comparison library is determined based on the text macroscopic features and a text library; inputting the to-be-stored text into a keyword matching layer to determine a keyword matching result, and determining a second comparison library based on the keyword matching result and the first comparison library; performing full-text retrieval on the second comparison library based on the to-be-stored text to determine a third comparison library; and determining whether a text matched with the to-be-stored text exists in the third comparison library based on a preset similarity algorithm, and inputting the to-be-stored text into the text library when the text does not exist in the third comparison library. Therefore, a human perception mechanis