The fractional derivatives include nonlocal information and thus their calculation requires huge storage and computational cost for long time simulations. We present an efficient and high-order accurate numerical formula to speed up the evaluation of the Caputo fractional derivative based on the L2-1σ formula proposed in [A. Alikhanov, J. Comput. Phys., 280 (2015), pp. 424-438], and employing the sum-of-exponentials approximation to the kernel function appeared in the Caputo fractional derivative. Both theoretically and numerically, we prove that while applied to solving time fractional diffusion equations, our scheme not only has unconditional stability and high accuracy but also reduces the storage and computational cost.