Age-related hearing impairment (ARHI) is a common condition with complex etiology but a recognized genetic component. Heritability estimates for pure tone audiogram-determined hearing ability lie in the range 26–75%. The speech-in-noise (SIN) auditory test, however, may be better at encapsulating ARHI symptoms, particularly the diminished ability to segregate environmental sounds into comprehendible auditory streams. As heritability of SIN has not previously been reported, we explored the genetic and environmental contributions to ARHI determined by SIN in 2,076 twins (87.8% female) aged 18–87 (mean age 54.4). SIN was found to be significantly heritable (A, unadjusted for age = 40%; 95% confidence intervals, CI = 32%–47%). With age adjustment, heritability fell (A = 25%; 95% CI = 16–33%), and a relatively strong influence of environmental exposure unshared within twin siblings was identified (E = 75%). To explore the environmental aspects further, we assessed the influence of diet (through the Food Frequency Questionnaire, FFQ), smoking (through self-report and cotinine metabolite levels) and alcohol intake (through the FFQ). A negative influence of high cholesterol diet was observed after adjustment (p = .037). A protective effect of raised serum high-density lipoprotein (HDL) cholesterol levels was observed after adjustment (p = .004). This study is the first assessment of the genetic and environmental influence on SIN perception. The findings suggest SIN is less heritable than pure tone audiogram (PTA) ability and highly influenced by the environment unique to each twin. Furthermore, a possible role of dietary fat in the etiology of ARHI is highlighted.