Eggs contain important compounds related to enhanced cognition, but it is not clear if egg consumption, as a whole, has a direct impact on memory decline in older adults. This study aimed to determine whether egg intake levels predict the rate of memory decline in healthy older adults after sociodemographic and dietary controls. We conducted a secondary analysis of data from 470 participants, age 50 and over, from the Biospsychosocial Religion and Health Study. Participants completed a food frequency questionnaire, which was used to calculate egg intake and divide participants into Low (<23 g/week, about half an egg), Intermediate (24–63 g/week, half to 1½ eggs) and High (≥63 g/week, about two or more eggs) tertiles. Participants were administered the California Verbal Learning Test – 2nd Edition (CVLT-II) Short Form in 2006–2007, and 294 of them were again tested in 2010–2011. Using linear mixed model analysis, no significant cross-sectional differences were observed in CVLT-II performance between egg intake levels after controlling for age, sex, race, education, body mass index, cardiovascular risk, depression and intake of meat, fish, dairy and fruits/vegetables. Longitudinally, the Intermediate egg group exhibited significantly slower rates of decline on the CVLT-II compared to the Low egg group. The High egg group also exhibited slower rates of decline, but not statistically significant. Thus, limited consumption of eggs (about 1 egg/week) was associated with slower memory decline in late life compared to consuming little to no eggs, but a dose-response effect was not clearly evident. This study may help explain discrepancies in previous research that did not control for other dietary intakes and risk factors.