Non-separable Lattices, Gabor Orthonormal Bases and Tilings

Let K ⊂ R d be a bounded set with positive Lebesgue measure. Let Λ = M ( Z 2 d ) be a lattice in R 2 d with density dens ( Λ ) = 1 . It is well-known that if M is a diagonal block matrix with diagonal matrices A and B , then G ( | K | - 1 / 2 χ K , Λ ) is an orthonormal basis for L 2 ( R d ) if and only if K tiles both by A ( Z d ) and B - t ( Z d ) . However, there has not been any intensive study when M is not a diagonal matrix. We investigate this problem for a large class of important cases of M . In particular, if M is any lower block triangular matrix with diagonal matrices A and B , we prove that if G ( | K | - 1 / 2 χ K , Λ ) is an orthonormal basis, then K can be written as a finite union of fundamental domains of A ( Z d ) and at the same time, as a finite union of fundamental domains of B - t ( Z d ) . If A t B is an integer matrix, then there is only one common fundamental domain, which means K tiles by a lattice and is spectral. However, surprisingly, we will also illustrate by an example that a union of more than one fundamental domain is also possible. We also provide a constructive way for forming a Gabor window function for a given upper triangular lattice. Our study is related to a Fuglede’s type problem in Gabor setting and we give a partial answer to this problem in the case of lattices.