The systematic development of coarse-grained (CG) models via the Mori–Zwanzig projector operator formalism requires the explicit description of a deterministic drift term, a dissipative memory term and a random fluctuation term. The memory and fluctuating terms are related by the fluctuation–dissipation relation and are more challenging to sample and describe than the drift term due to complex dependence on space and time. This work proposes a rational basis for a Markovian data-driven approach to approximating the memory and fluctuating terms. We assumed a functional form for the memory kernel and under broad regularity hypothesis, we derived bounds for the error committed in replacing the original term with an approximation obtained by its asymptotic expansions. These error bounds depend on the characteristic time scale of the atomistic model, representing the decay of the autocorrelation function of the fluctuating force; and the characteristic time scale of the CG model, representing the decay of the autocorrelation function of the momenta of the beads. Using appropriate parameters to describe these time scales, we provide a quantitative meaning to the observation that the Markovian approximation improves as they separate. We then proceed to show how the leading-order term of such expansion can be identified with the Markovian approximation usually considered in the CG theory. We also show that, while the error of the approximation involving time can be controlled, the Markovian term usually considered in CG simulations may exhibit significant spatial variation. It follows that assuming a spatially constant memory term is an uncontrolled approximation which should be carefully checked. We complement our analysis with an application to the estimation of the memory in the CG model of a one-dimensional Lennard–Jones chain with different masses and interactions, showing that even for such a simple case, a non-negligible spatial dependence for the memory term exists.