Grammar-based transformations: attack and defence

Purpose – This research aims to propose an attack that de-obfuscates codes by exploiting the properties of context-free grammars since it is important to understand the strength of obfuscation provided by context-free grammar-based obfuscators. In addition, the possibility of automatically generated transformations is explored. Design/methodology/approach – As part of our empirical investigation, a development environment for obfuscating transformations is built. The tool is used to simulate a context-free obfuscator and to devise ways of reversing such transformations. Furthermore, a theoretical investigation of subset grammars and subset languages is carried out. Findings – It is concluded that context-free grammar-based obfuscators provide limited levels of protection. Nevertheless, their application is appropriate when combined with other obfuscating techniques. Research limitations/implications – The algorithms behave as expected on a limited number of test samples. Further work is required to increase their practicality and to establish their average reliability. Originality/value – This research shows how a frequency analysis attack can threaten the security of code scrambled by context-free grammar-based obfuscators.