We model and study the genetic evolution and conservation of a population of diploid hermaphroditic organisms, evolving continuously in time and subject to resource competition. In the absence of mutations, the population follows a three-type, nonlinear birth-and-death process, in which birth rates are designed to integrate Mendelian reproduction. We are interested in the long-term genetic behavior of the population (adaptive dynamics), and in particular we compute the fixation probability of a slightly nonneutral allele in the absence of mutations, which involves finding the unique subpolynomial solution of a nonlinear three-dimensional recurrence relationship. This equation is simplified to a one-dimensional relationship which is proved to admit exactly one bounded solution. Adding rare mutations and rescaling time, we study the successive mutation fixations in the population, which are given by the jumps of a limiting Markov process on the genotypes space. At this time scale, we prove that the fixation rate of deleterious mutations increases with the number of already fixed mutations, which creates a vicious circle called the extinction vortex.

A Markov chain model related to population genetics and its convergence to a diffusion process on the multi-dimensional bounded domain are treated. We discuss the case where natural selection is random and the different selection effects over successive generations are independent. Our model is a multi-allelic version of the haploid model of Karlin and Levikson. The asymptotic properties of the limiting diffusion are stated.