Efficient file fuzz testing using automated analysis of binary file format

Fuzz testing is regarded as the most useful technique in finding serious security holes in a software system. It inserts unexpected data into the input of the software system and finds the system’s bugs or errors. However, one of the disadvantages that fuzz testing executed using binary files has is that it requires a large number of fault-inserted files to cover every test case, which could be up to 2 8 × FILESIZE files. In order to overcome this drawback, we propose a novel algorithm that efficiently reduces the number of fault-inserted files, yet still maintain the maximum test case coverage. The proposed approach enables the automatic analysis of fields of binary files by tracking and analyzing stack frames, assembly codes, and registers as the software system parses the files. We evaluate the efficacy of the new method by implementing a practical tool, the Binary File Analyzer and Fault Injector (BFAFI), which traces the program execution and analyzes the fields in binary file format. Our experiments demonstrate that the BFAFI reduced the total number of fault-inserted files with maximum test case coverage as well as detected approximately 14 times more exceptions than did the general fuzzer. Also, the BFAFI found 11 causes of exceptions; five of them were found only by BFAFI. Ten of the 11 causes of exceptions that we found were generated by a graphic rendering engine ( GDI32.dll); the other was generated by the system library ( kernel32.dll) in Windows XP SP2.