We study the optimal conditions for nanoisland growth in ion beam assisted deposition (IBAD). This situation occurs when adatom islands remain small enough to prevent the onset of three-dimensional growth, while at the same time preventing ion-induced surface erosion. To this end, we develop a rate equation model of IBAD, which embodies continuous deposition of adatoms and creation of vacancies, recombination of vacancies at adatom island edges, as well as recombination of adatoms at vacancy island edges. These rate equations are solved by numerical simulations based on the particle coalescence method. To determine the optimal growth condition, we find the largest mean size of the vacancy islands leading to their survival. We show that at this onset between the rough and smooth layer-by-layer growth regimes there is a simple exponential relation between the largest size of the vacancy islands and the external control parameters of the growth. (c) 2007 Elsevier B.V. All rights reserved.