Technical note: “Bit by bit”: a practical and general approach for evaluating model computational complexity vs. model performance

One of the main objectives of the scientific enterprise is the development of well-performing yet parsimonious models for all natural phenomena and systems. In the 21st century, scientists usually represent their models, hypotheses, and experimental observations using digital computers. Measuring performance and parsimony of computer models is therefore a key theoretical and practical challenge for 21st century science. "Performance" here refers to a model's ability to reduce predictive uncertainty about an object of interest. "Parsimony" (or complexity) comprises two aspects: descriptive complexity - the size of the model itself which can be measured by the disk space it occupies - and computational complexity - the model's effort to provide output. Descriptive complexity is related to inference quality and generality; computational complexity is often a practical and economic concern for limited computing resources.