The invasion of alien species and genotypes is an increasing concern in contemporary ecology. A central question is, what life-history traits enable invasion amidst populations of wild species and conventional cultivars? In order to invade, the initially rare species must perform better than their resident competitors. We conducted a mathematical analysis and simulation of a two-species extension of the Maynard Smith and Slatkin model for population dynamics in discrete time to study the role of density dependence as different types of competition in the invasion of new species. The type of density dependence ranged from scramble to contest competition. This led to intrinsic dynamics of the species range from point equilibrium to cycles and chaos. The traits were treated either as free parameters or constrained by a trade-off resulting from a common fixed strength of density dependence or equilibrium density. Resident and intruder traits had up to ten-fold differences in all of the parameters investigated. Higher equilibrium density of the intruder allowed invasion. Under constrained equilibrium density, an intrinsically stable intruder could invade an unstable resident population. Scramble competition made a population more susceptible to invasion than contest competition (e.g., limitation by light or territory availability). This predicts that a population which is mainly limited by food (or nutrients in plants) is more likely to be invaded than a population limited by a hierarchical competition, such as light among plants. The intruder population may have an effect on the resident population's dynamics, which makes the traditional invasion analysis unable to predict invasion outcome.