Evaluation infrastructure for testing real-time content search

Systems and methods provide an experimentation system, or testing engine, for a real-time search infrastructure. An example method includes generating a snapshot of a production search environment and performing testing of a signal or index change in the snapshot. The change can be specified as parameters passed to the system. The method may include estimating an impact of the change based on the testing and determining, based on the estimate, whether the impact is positive. Responsive to determining the impact is positive the method may include generating a holdback environment of the production search environment, the holdback environment being a portion of the production search environment selected not to receive the change, pushing the change to the production search environment, and monitoring the change by running partial production traffic through the holdback environment.