Thermodynamic stability, compressibility matrices, and effects of mediated interactions in a strongly interacting Bose-Fermi mixture

We theoretically investigate the thermodynamic stability of a normal-state Bose-Fermi mixture, with a tunable Bose-Fermi pairing interaction -U-BF < 0 associated with a heteronuclear Feshbach resonance, as well as a weak repulsive Bose-Bose interaction U-BB >= 0. Including strong heteropairing fluctuations associated with the former interaction within the self-consistent T-matrix approximation, as well as the latter within the mean-field level, we calculate the compressibility matrix, to assess the stability of this system against density fluctuations. In the weak- and the intermediate-coupling regimes with respect to -U-BF, we show that an effective attractive interaction between bosons mediated by density fluctuations in the Fermi component makes the system unstable below a certain temperature T-clp (leading to density collapse). When U-BB = 0, T-clp is always higher than the Bose-Einstein condensation (BEC) temperature T-c. When U-BB > 0, the density collapse is suppressed, and the BEC transition becomes possible. It is also suppressed by the formation of tightly bound Bose-Fermi molecules when the heteropairing interaction -U-BF is strong; however, since the system may be viewed as a molecular Fermi gas in this case, the BEC transition does not also occur. Since quantum gases involving Bose atoms are known to be sensitive to interparticle correlations, our results would be useful for the study of many-body properties of a Bose-Fermi mixture in a stable manner, without facing the unwanted density collapse.