Tractate Yevamot - Chapter 13 - Mishnah 10

[If one was] of sound senses and the other deaf, and the yavam had intercourse with the woman of sound senses and then he also had intercourse with the deaf woman, or a brother of his had intercourse with the deaf woman, he does not disqualify the former [for him]. If the yavam had intercourse with the deaf woman, and then he also had intercourse with the woman of sound senses, or a brother of his had intercourse with the woman of sound senses, he disqualifies the deaf woman [for him].