An evolutionarily stable strategy (ESS) is a strategy that no other feasible alternative can better, given that sufficient members of the population adopt it. The best strategy for an individual depends upon the strategy or strategies which other members of the same population adopt. Since the same applies to all individuals in that particular population, a mutant gene cannot invade an ESS successfully.
The traditional way to illustrate this problem is the simulation of the encounter between two strategies, the hawk and the dove. When a hawk meets a hawk, it wins on half of the occasions, and it loses and suffers an injury on the other half. Hawks always beat doves. Doves always retreat against hawks. Whenever a dove meets another dove, there is always a display, and it wins on half of the occasions. Under these rules, populations of only hawks or doves are no ESS because a hawk can invade a population made up entirely of doves and a dove can invade a population of hawks only. Both would have an advantage and would spread in the population. A hawk in a population of doves would win all contests, and a dove in a population of hawks would never get injured because it wouldn’t fight.
However, it is possible for a mixture of hawks and doves to provide a stable situation when their numbers reach a certain proportion of the total population. For example, with payoffs as winner +50, injury -100, loser 0, display -10, a population consisting of hawks and doves (or individuals adopting hawk and dove strategies) is an ESS whenever 58,3% of the population are hawks and 41,7% doves; or when all individuals behave at random as hawks in 58,3 % of the encounters and doves in 41,7%. The percentages, or point of equilibrium, depend on costs and benefits (or the pay-off, which is equal to benefits minus costs).
Evolutionarily stable strategies are not artificial constructs. They exist in nature. The Oryx, Oryx gazella, have sharp pointed horns, which they never use in contests with rivals and only in defense against predators. They play the dove strategy. Up to 10% per year of Musk Ox, Ovibos moschatus, adult males die as a result of injuries sustained while fighting over females. They play the hawk strategy.
An ESS is a modified form of a Nash equilibrium. In most simple games, the ESSes and Nash equilibria coincide perfectly, but some games may have Nash equilibria that are not ESSes. Furthermore, even if a game has pure strategy Nash equilibria, it might be that none of those pure strategies are ESSes. We can prove both Nash equilibria and ESS mathematically (see references).
Peer-to-peer file sharing is a good example of an ESS in our modern society. BitTorrent peers use Tit for Tat strategy to optimize their download speed. Cooperation is achieved when upload bandwidth is exchanged for download bandwidth.
Evolutionary biology and sociobiology attempt to explain animal behavior and social structures (humans included), mainly in terms of evolutionarily stable strategies.