People perceive roughly constant surface colors despite large changes in illumination. The familiarity of colors of some natural objects might help achieve this feat through direct modulation of the objects' color appearance. Research on memory colors and color appearance has yielded controversial results and due to the employed methods has often confounded perceptual with semantic effects. We studied the effect of memory colors on color appearance by presenting photographs of fruit on a monitor under various simulated illuminations and by asking observers to make either achromatic or typical color settings without placing demands on short-term memory or semantic processing. In a control condition, we presented photographs of 3D fruit shapes without texture and 2D outline shapes. We found that (1) achromatic settings for fruit were systematically biased away from the gray point toward the opposite direction of a fruit's memory color; (2) the strength of the effect depended on the degree of naturalness of the stimuli; and (3) the effect was evident under all tested illuminations, being strongest for illuminations whose chromaticity was closest to the stimulus chromaticity. We conclude that the visual identity of an object has a measurable effect on color perception, and that this effect is robust under illuminant changes, indicating its potential significance as an additional mechanism for color constancy.