Original data processing method

The invention relates to an original data processing method. The method is applied to an original data processing system. The system comprises multiple original data sources, multiple data collectingunits, a cluster storage, an original data processing platform, a data manager and a client. The original data processing method can perform repetition removal, contradiction removal and unreasonableremoval treatment on original data, on the basis of data record similarity, data record repetition removal treatment is performed, on the basis of the credibility, data records are selected for deletion treatment, the data repetition removal accuracy and efficiency are improved, the man-made participation workloads are lowered, the automatic degree is increased, and therefore the user experience of a client user is improved. 本发明涉及种原始数据处理方法，该方法应用于原始数据处理系统中，该系统包括多个原始数据源，多个数据收集单元，集群存储器，原始数据处理平台，数据管理器，客户端；该原始数据处理方法能够对原始数据进行去重复，去矛盾，去不合理处理，基于数据记录相似度进行数据记录的去重复处理，基于置信度选择数据记录作删除处理，提高了数据去重复的准确度和效率，减少了人为参与的工作量，提高了自动化程度，从而提高了客户端用户