The maternal brain undergoes adaptations to sensitive caregiving that are critical for infant well-being. We investigated structural alterations associated with neglectful caregiving and their effects on mother–child interactive behavior. High-resolution 3D volumetric images were obtained on 25 neglectful (NM) and 23 non-neglectful control (CM) mothers. Using voxel-based morphometry, we compared differences in gray and white matter (GM and WM, respectively) volume. Mothers completed an empathy scale and participated with their children in a play task (Emotional Availability Scale, EA). Neglectful mothers showed smaller GM volume in the right insula, anterior/middle cingulate (ACC/MCC), and right inferior frontal gyrus and less WM volume in bilateral frontal regions than did CM. A greater GM volume was observed in the right fusiform and cerebellum in NM than in CM. Regression analyses showed a negative effect of greater fusiform GM volume and a positive effect of greater right frontal WM volume on EA. Mediation analyses showed the role of emotional empathy in the positive effect of the insula and right inferior frontal gyrus and in the negative effect of the cerebellum on EA. Neglectful mothering involves alterations in emotional empathy-related areas and in frontal areas associated with poor mother–child interactive bonding, indicating how critical these areas are for sensitive caregiving.