X-ray reflectometry is a powerful tool for investigating rough surface and interface structures. Presently, X-ray reflectivity is based on Parratt formalism, accounting for the effect of roughness by the theory of Nevot–Croce. However, the calculated results showed a strange phenomenon in that the amplitude of the oscillation because of interference effects increases in the case of a specific roughness of the surface. We propose that the strange results originated from the currently used equation because of a serious error in which the Fresnel transmission coefficient in the reflectivity equation is increased at a rough interface, and the increase in the transmission coefficient completely overpowers any decrease in the value of the reflection coefficient because of lack of consideration in diffuse scattering. In the present study, we present a new improved formalism that corrects this error, and thereby derives an accurate analysis of X-ray reflectivity from a multilayer surface, taking into account the effect of roughness-induced diffuse scattering.