Is your bird calm and docile when you handle him, but shows extreme aggression toward members of the opposite sex when they try to interact with him? It is possible that your bird has a gender bias. Some birds, although not all of them, do prefer men to women or vice versa -- and they will usually not be shy about letting their preference be known!
It could be something as simple as a fear of a man's deep voice, or an aversion to long hairstyles that women typically wear, but some birds would just rather be around members of one sex than they would the other.
My Sun Conure, Loco, tends to prefer my husband to me most days, while other days he refuses to leave my side. Even so, he allows my husband to give him "scritches" on his belly and under his wings, places that he won't let me touch.
While the reasons that birds can be gender biased are unknown, it's important not to take it personally if your bird prefers females to males or vice versa. Each and every bird has a unique, individualized personality, and forcing one to interact with a person that the bird has an aversion to can lead to behavior problems and bad bites if you aren't careful.
The best way to deal with a bird that has decided you aren't the "right type" for him is to try talking to him and interacting with him while he is in his cage for the safety of all parties involved. Once the bird realizes what a fantastic person you are, it will be much easier to move on to handling and more physical interaction.
Should this approach fail you, other options include changing the type of clothing you wear around the bird, the tone of your voice, or your hairstyle. Be creative and don't give up! Eventually, you should stumble onto the trait that the bird dislikes. Once that has been identified, it will make acclimating the bird to members of whichever sex he scorns a far more reasonable task.