On the Separation Question for Tree Languages

We show that the separation property fails for the classes Σ n of the Rabin-Mostowski index hierarchy of alternating automata on infinite trees. This extends our previous result (obtained with Szczepan Hummel) on the failure of the separation property for the class Σ 2 (i.e., for co-Büchi sets). The non-separation result is also adapted to the analogous classes induced by weak alternating automata. To prove our main result, we first consider the Rabin-Mostowski index hierarchy of deterministic automata on infinite words, for which we give a complete answer (generalizing previous results of Selivanov): the separation property holds for Π n and fails for Σ n -classes. The construction invented for words turns out to be useful for trees via a suitable game. It remains open if the separation property holds for all classes Π n of the index hierarchy for tree automata. To give a positive answer it would be enough to show the reduction property of the dual classes—a method well-known in descriptive set theory. We show that it cannot work here, because the reduction property fails for all classes in the index hierarchy.