We have carried out a study of the neutral hydrogen in the direction of the X-ray source 1E 161348–5055, a compact central object (CCO) located in the interior of the supernova remnant (SNR) RCW 103. The Hi 21 cm line observations were carried out using the Australia Telescope Compact Array, complemented with single dish data from the Parkes radio telescope to recover information at all spatial scales. We derive a distance to RCW 103 of 3.1 kpc, in agreement with previous distance measurements. We have also detected a small hole in the Hi emission which is positionally and kinematically coincident with the location of the CCO which confirms the association between the SNR and the CCO. This is the third case of a depression in Hi emission seemingly associated with CCOs in SNRs. The characteristic parameters of the holes such as their size, eccentricity and evacuated mass are similar in all three cases. We estimate the absorbing HI column density towards 1E 161348–5055 to be ∼6 × 1021 cm–2, a value compatible with a blackbody solution for the CCO X-ray emission. However, the implied temperature and luminosity are very high compared to most neutron stars. Moreover, the strong long-term variability in X-rays favours the hypothesis that 1E 161348–5055 is an accreting binary source rather than an isolated, cooling neutron star. An analysis of the continuum image obtained at 1.4 GHz from these observations shows no trace of a pulsar wind nebula around 1E 161348–5055, in spite of it being a young object.