An evolutionarily stable strategy (ESS) is a strategy that no other feasible alternative can better, given that sufficient members of the population adopt it. The best strategy for an individual depends upon the strategy or strategies that other members of the same population adopt. Since the same applies to all individuals in that particular population, a mutant gene cannot invade an ESS successfully.
The traditional way to illustrate this problem is simulating the encounter between two strategies, hawk and dove. When a hawk meets a hawk, it wins on half of the occasions, and it loses and suffers an injury on the other half. Hawks always beat doves. Doves always retreat against hawks. Whenever a dove meets another dove, there is always a display, and it wins on half of the occasions. Under these rules, populations of only hawks or doves are no ESS because a hawk can invade a population made up entirely of doves and a dove can invade a population of hawks only. Both would have an advantage and would spread in the population. A hawk in a population of doves would win all contests, and a dove in a population of hawks would never get injured because it wouldn’t fight.
However, it is possible for a mixture of hawks and doves to provide a stable situation when their numbers reach a certain proportion of the total population. For example, with payoffs as winner +50, injury -100, loser 0, display -10, a population comprising hawks and doves (or individuals adopting a mixed strategy of alternating between playing hawk and dove strategies) is an ESS whenever 58,3% of the population are hawks and 41,7% doves; or when all individuals behave at random as hawks in 58,3% of the encounters and doves in 41,7%. The percentages (the point of equilibrium) depend on costs and benefits (or the pay-off, which is equal to benefits minus costs).
Evolutionarily stable strategies are not artificial constructs. They exist in nature. The Oryx, Oryx gazella, have sharp pointed horns, which they never use in contests with rivals and only in defense against predators. They play the dove strategy. Up to 10% per year of Musk Ox, Ovibos moschatus, adult males die because of injuries sustained while fighting over females. They play the hawk strategy.
An ESS is a modified form of a Nash equilibrium. In most simple games, the ESSes and Nash equilibria coincide perfectly, but some games may have Nash equilibria that are not ESSes. Furthermore, even if a game has pure strategy Nash equilibria, it might be that none of those pure strategies are ESSes. We can prove both Nash equilibria and ESS mathematically (see references).
Peer-to-peer file sharing is a good example of an ESS in our modern society. Bit Torrent peers use Tit-for-Tat strategy to optimize their download speed. They achieve cooperation exchanging upload bandwidth with download bandwidth.
Evolutionary biology and sociobiology attempt to explain animal behavior and social structures (humans included), primarily in terms of evolutionarily stable strategies.