Bandit Problems With Infinitely Many Arms

We consider a bandit problem consisting of a sequence of $n$ choices from an infinite number of Bernoulli arms, with $n \rightarrow \infty$. The objective is to minimize the long-run failure rate. The Bernoulli parameters are independent observations from a distribution $F$. We first assume $F$ to be the uniform distribution on (0, 1) and consider various extensions. In the uniform case we show that the best lower bound for the expected failure proportion is between $\sqrt{2}/\sqrt{n}$ and $2/\sqrt{n}$ and we exhibit classes of strategies that achieve the latter.