Organization and management of ATLAS nightly builds

The automated multi-platform software nightly build system is a major component in the ATLAS collaborative software organization, validation and code approval schemes. Code developers from ATLAS participating Institutes spread all around the world use about 30 branches of nightly releases for testing new packages, verification of patches to existing software, and migration to new platforms and compilers. The nightly releases lead up to, and are the basis of, stable software releases used for data processing worldwide. The ATLAS nightly builds are managed by the fully automated NICOS framework on the computing farm with 44 powerful multiprocessor nodes. The ATN test tool is embedded within the nightly system and provides results shortly after full compilations complete. Other test frameworks are synchronized with NICOS jobs and run larger scale validation jobs using the nightly releases. NICOS web pages dynamically provide information about the progress and results of the builds. For faster feedback, e-mail notifications about nightly releases problems are automatically distributed to the developers responsible.