Towards a systematic approach to manual annotation of code smells

Code smells are structures in code that may indicate maintainability issues. They are challenging to define, and software engineers detect them differently. Mitigation of this problem could be an AI code smell detector. However, to develop it, we need a standardized benchmark dataset. Existing datasets suffer from (1) annotation subjectivity, (2) lack of ground-truth consensus among annotators, and (3) reproducibility issues. This paper aims to develop a systematic manual code smell annotation procedure that addresses these issues. We tailored the prescriptive natural language processing annotation methodology to code smell detection: (1) we cross-validate annotations to mitigate subjectivity, (2) we develop clear annotation guidelines to reach the ground-truth consensus, and (3) we follow literature recommendations for reproducibility and open-source our tools and dataset. We extracted the annotation guidelines from existing empirical code smell research. The annotators refined the guidelines and their understanding of the task through proof-of-concept annotation encompassing retrospective discussion and disagreement resolution and then performed full annotation. We confirmed that the ground-truth consensus was reached by measuring annotation consistency. Our contributions are the proposed annotation procedure, a novel code smell dataset of open-source C# projects, the annotators' experience report, and the open-sourced supporting tool. •Automated code maintainability analysis relies on datasets with limitations.•We propose a systematic procedure for manually annotating ambiguous code smells.•We propose an annotation model for the Large Class and Long Method smells.•We create a supporting tool, a medium-sized C# dataset, and annotation guidelines.•We assess annotation consistency using statistical tests and empirical comparison.