Rigorous Policy Pilots: Experimentation in the Administration of the Law

Rigorous tests are being used every day to develop effective medical treatments, drive consumer engagement, and, more generally, discover what works. But so far, rigorous policy piloting-temporarily introducing a change in law or policy in order to learn from it using welldesigned and well-implemented methods-has not been used widely because of the perception that policy experimentation is unfair and possibly illegal, difficult, and rare. This Essay draws upon case law and agency practice to show that, to the contrary, rigorous policy pilots are presumptively legal, feasible, and increasingly common, proceeding in several steps. First, it finds that many kinds of pilots, including those that vary internal agency processes, or which are opt-in are unlikely to be controversial. But a review of relevant cases suggests that courts are likely to uphold even pilots that treat like members of a population differently, including through randomization, when they advance learning. Further, it finds experimentation, by itself, to be unlikely to create special procedural or substantive hurdles. Second, it finds that agencies are engaging in a range of rigorous piloting activities to fill informational gaps in policy- and law-making, some of which simulate and others which effect policy variation on a temporary basis, and that developments such as the growth of open data are making such forms of information gathering easier. It draws from agency experience to develop a framework for proposing a policy pilot and identify steps that would further support the use of rigorous pilots. A companion online appendix applies this framework to propose several rigorous pilots that the United States Patent and Trademark Office ("USPTO"), building on its already strong tradition of piloting, to evolve its own policies and practices with respect to patent quality (through the robust vetting of applications in view of non-patent literature and team/time examination on demand) and inclusion in innovation (through automated error correction and addressing gender bias in examination).