System and method for data organization in shingled magnetic recording drive

One embodiment described herein provides a system for managing metadata for a shingled magnetic recording (SMR) drive. During operation, the system stores a metadata file in a memory of a host of the SMR drive; subsequent to the SMR drive performing a write operation, the system generates a metadata update, and sends the generated metadata update to a buffer associated with the SMR drive and the memory of the host. In response to a compaction condition being met, the system launches, in the memory of the host, a metadata compaction operation to generate a compacted metadata file based on the previously stored metadata file and the metadata update, and sends the compacted metadata file to the SMR drive.