An unethical optimization principle

If an artificial intelligence aims to maximize risk-adjusted return, then under mild conditions it is disproportionately likely to pick an unethical strategy unless the objective function allows sufficiently for this risk. Even if the proportion η of available unethical strategies is small, the probability p U of picking an unethical strategy can become large; indeed, unless returns are fat-tailed p U tends to unity as the strategy space becomes large. We define an unethical odds ratio, Υ (capital upsilon), that allows us to calculate p U from η , and we derive a simple formula for the limit of Υ as the strategy space becomes large. We discuss the estimation of Υ and p U in finite cases and how to deal with infinite strategy spaces. We show how the principle can be used to help detect unethical strategies and to estimate η . Finally we sketch some policy implications of this work.