Previous investigations of goethite revealed a substantial variation of color and diffuse reflectance spectra (DRS) in the extended visible range (350–2200 nm). To better understand the causes of this variability and to assess the potential of DRS as a mineralogical tool, we investigated the DRS of pure and Al-substituted goethite, α-Fe1−xAlxOOH with x from 0 to 0.33, and mean crystal lengths (MCL) from 170 to 1800 nm. The strongly overlapping ligand field bands were extracted by fitting the single-electron transitions 6A1 → 4T1, 6A1 → 4T2, 6A1 → (4E; 4A1), and 6A1 → 4E(4D) as functions of the ligand field splitting energy, 10 Dq, and the interelectronic repulsion parameters, Racah-B and -C. With x increasing from 0 to 0.33, 6A1 → 4T1 decreased from 10,590 to 10,150 cm−1 (944 to 958 nm), and 6A1 → 4T2 decreased from 15,310 to 14,880 cm−1 (653 to 672 nm), while 10 Dq increased from 15,770 to 16,220 cm−1. From the change of 10 Dq we calculated a decrease of the Fe-(O,OH) distances from 202.0 to 200.9 pm (−0.5%). This decrease is smaller than the average decrease of all (Al,Fe)-(O,OH) distances (−1.8%) calculated from the change of the unit-cell lengths (UCL). That is, there remains a substantial difference in size between the larger Fe- and the smaller Al-occupied octahedra in the solid solution which may indicate the existence of diaspore clusters within the goethite structure. The increasing strain in the crystal structure due to the size mismatch and limited contractibility of the oxygen cage around Fe may be the primary reason for Al substitution being restricted to x < 0.33. The bands 6A1 → (4E; 4A1) and 6A1 → 4E(4D) did not shift, indicating a constant covalency of the Fe-(O,OH) bonds with B = 628 cm−1 and C = 5.5B. Whereas variation of band energies could be explained in terms of the Fe-(O,OH) ligand field, the variation of color and band intensities was mainly determined by crystal size. Although our study confirmed the potential of DRS for mineralogical investigations, there is still a gap between the fundamental theory and the explanation of some spectral features.