Marine fish have figured out how to stay alive in the oceans for literally millions (billions?) of years. If they hadn't, quite simply they wouldn't be here, now. Each species developed a method for surviving, whether it was a really great defense mechanism (i.e. the Volitans Lionfish and its poisonous spines), schooling (the "safety in numbers" spreads the risk of an individual being eaten), the ability to hide from its pursuer (in rocks or corals), through symbiotic relations (the Clownfish wouldn't be around if it wasn't for anemones) or just the ability to be able to flee.
Taking a fish out of the ocean and putting it into a closed system, such as a home aquarium, greatly reduces a fish's ability to flee or hide from predation. At the same time, it also increases the competition for whatever food is available.
The chart below will give you an idea of which fish can and can not "normally" exist together in a closed space. In many cases it also indicates which will coexist with a certain amount of caution. Nothing is guaranteed. There will always be exceptions to any generalization, but the chart will give you a place to start when you are trying to figure out what will work in your aquarium.
Saltwater Aquarium Fish Compatibility Chart
Take a look at the below saltwater aquarium fish compatibility chart to check if your fish are compatible together in the same aquarium.
Why Aren't These Fish Compatible?
Different fish aren't compatible for several reasons, but it all comes down to competition. Everything in the ocean is competing for something in order to stay alive, eat and reproduce. The five basic types of competition are: Predator/Prey, Territory Protection, Mate & Status Protection, Spawn Protection and Opportunistic Feeding Behavior.
Predator or Prey
The recipients of this type of aggressive behavior are perceived as food organisms, such as a Dragon Moray Eel (Enchelycore pardalis) consuming an ornamental shrimp, or a Lionfish consuming a small Damselfish. This category is pretty much self explanatory. It is obvious that keeping larger predatory fish with anything small enough that they can perceive as food is not a wise idea. Many aquarists choose to keep these types of fish in a specific predatory tank community, with fish like large Groupers, Hawks, Snappers and other predatory species.
The recipient of this type of aggressive behavior are others of the same or similar species, such as a juvenile Angelfish and Jewelfish attacking others. When it comes to territorial aggression, most fish react in the same way, especially when you already have fish established in an aquarium and you add in a new tank mate later on. Generally, when you place all new fish into an aquarium at the same time, a few disputes will occur until territories are established. Once that is done, the fish usually settle down and life goes on. The problem of "harassment" will most likely occur whenever you put a new fish into an established aquarium community, and most often it doesn't seem to matter what type or species of fish it is.
Mate and Status Protection
The recipient of this type of aggressive behavior are other fish of the same species, such as a mated pair of Clownfish attacking others. It is interesting that most species exhibit this type of behavior. For example, if you put a mated pair of Angelfish, Butterflyfish, Boxfish, or just about any other type of mated species in a tank, and then add another male or female of the same species later, typically the same sexed fish will go after the same sexed fish that was newly introduced. Limiting a tank to one specific mated species is wise.
The recipient of this type of aggressive behavior are all other fish near a nest area, such as Damselfish protecting their spawn from other fish that may stray into their nesting area. It helps to keep fish that spawn in captivity in larger sized aquariums, and provide ample housing not only for the nesting species, but all the other tank inhabitants as well. Pelagic or "free-spawning" fish, such as the Yellow Tang, do not demonstrate this type of behavior.
Opportunistic Feeding Behavior
The recipient of this type of aggressive behavior are all other fish and invertebrates. A good example of this behavior are Triggerfishes, which will eat just about anything. Fish that have this type of behavior are best kept in a specific aggressive species tank community. Many times aquarists will keep Triggers with other Triggers, as well as Lions, Groupers, Hawks, Snappers, and other larger predatory species.
Before adding any new livestock to your tank, it is wise to learn about their particular behavior patterns in order to avoid competition which will end up in your losing some of your valuable fish and/or invertebrates.