Does fin coloration signal social status in a dominance hierarchy of the livebearing fish Xiphophorus variatus?

•Males with YR fin coloration are the largest within populations.•X. variatus has a size-based hierarchy; males with YR coloration are at the top.•Aggression does not differ between males with and without YR fin color.•Smaller, lower ranking males do not reduce aggression towards YR males.•YR fin coloration likely has not evolved as a signal in agonistic interactions. In each population of the livebearing fish Xiphophorus variatus, only a small portion of the adult males develop bright yellow-red (YR) coloration on the dorsal and caudal fins. Here we characterized the dominance hierarchy in X. variatus and tested whether YR coloration is related to a male's position in the hierarchy and can therefore serve as a reliable cue to rival males. Populations varied considerably in the frequency of YR males. Across all populations, males with YR coloration were significantly larger than the rest of the males in the population. Observations of aggressive interactions among males in small groups in the laboratory revealed a sized-based dominance hierarchy with YR males at the top. Aggression was more common among males of a similar size and fighting increased as male body size differences decreased. However, despite the reliability of YR coloration as a signal of dominance status, males at lower social ranks did not avoid aggression with YR males and YR males did not experience fewer aggressive attacks compared to non-YR males. Our findings demonstrate that fin coloration is a reliable cue of a male's social status but rival males appear to not use this information to avoid potentially costly interactions with dominant males, suggesting that YR fin coloration has not evolved as a cue in agonistic interactions.