A monitoring framework for transparency and fairness in big data platform

Summary Big data comprises large volume of complex, (semi) structured, and unstructured data which are processed and analyzed in order to help organizations in making strategic decisions. This paper investigates into privacy, transparency, and fairness of big data. It contends that these factors are crucial and must be taken into account when collecting, analyzing, and using large scale personal data. It proposes a novel framework that monitors personal data usage in big data platform in order to ensure that data usage is compliant with the transparency and fairness as stipulated by Privacy Transparency Fairness Agreement (PTFA) rules. The proposed framework is implemented using Python and Java. It is evaluated using multi‐nodes cluster with millions of lines of data and thousands of contracts. Various experiments are conducted in order to evaluate the efficiency and scalability of the proposed framework. These show that the proposed framework is efficient and scalable when monitoring transparency and fairness in big data collection, analysis, and usage. Visualization experiments are also conducted in order to derive relevant indicators, for instance, to monitor non‐compliance of PTFA rules, visualize the alerts, and show the proportion of personal data being violated as compared to the number of contracts.