Probabilistic modelling of deception-based security framework using markov decision process

•A deception-based security framework to plan and integrate deception.•A model to understand attackers behaviours on failed actions.•Quantification metrics to measure attackers and defenders performance.•IoT attacks are modelled as MDP and probabilistic properties verified using PRISM. Existing studies using deception are ad-hoc attempts and few theoretical models have been designed to plan and integrate deception. We theorise that a pre-planning stage should be a fundamental part to obtain information about the attackers’ behaviours and the attack process by analysing known attacks. This will help plan and take defence actions by actively interacting with the attackers and predicting their actions using a probabilistic approach. This paper proposes a framework that provides a theoretical understanding to plan and integrate deception systematically and strategically. We also present probabilistic modelling to predict attack actions by formalising a real case of attacks captured on simulated Internet of Things devices as an Markov Decision Process (MDP) and verifying related properties using Probabilistic Symbolic Model Checker (PRISM). MDP’s properties verification results reveal that the associated cost for defence actions can be decreased by successfully predicting attackers’ probable actions. Moreover, we identify several quantification metrics (e.g. cost, reward, trust, incentive and penalty) to evaluate the performance of actions performed by attackers and defenders.