Ancestral polymorphisms and sex-biased migration shaped the demographic history of brown bears and polar bears

Recent studies have reported discordant gene trees in the evolution of brown bears and polar bears. Genealogical histories are different among independent nuclear loci and between biparentally inherited autosomal DNA (aDNA) and matrilineal mitochondrial DNA (mtDNA). Based on multi-locus genomic sequences from aDNA and mtDNA, we inferred the population demography of brown and polar bears and found that brown bears have 6 times (aDNA) or more than 14 times (mtDNA) larger population sizes than polar bears and that polar bear lineage is derived from within brown bear diversity. In brown bears, the effective population size ratio of mtDNA to aDNA was at least 0.62, which deviated from the expected value of 0.25, suggesting matriarchal population due to female philopatry and male-biased migration. These results emphasize that ancestral polymorphisms and sex-biased migration may have contributed to conflicting branching patterns in brown and polar bears across aDNA genes and mtDNA.