Fallow weeds can be abundant in rice paddies without any inputs and provide ecosystem services like those of cover crops, such as reducing nitrogen (N) leaching and capturing carbon. Therefore, allowing fallow weeds to grow is a potential alternative to cover crops in rice cropping systems. To evaluate the feasibility of this strategy, the effect of fallow weeds on grain yield in rice needs to be clarified. In this study, 2-year field experiments were conducted to compare N uptake, biomass production, yield components, and grain yield in rice with and without application of fallow weeds (500 g m−2, sun-dried). Results showed that the application of fallow weeds reduced aboveground N uptake and biomass production by 21–30% during the early growth period (from transplanting to mid-tillering) in rice. However, these reductions did not lead to reduced grain yield because they were compensated for or even exceeded by increased aboveground N uptake and biomass production during the middle and late growth periods (from panicle initiation to maturity). In addition, the application of fallow weeds increased spikelets per panicle in rice by 6–7%. These results provide preliminary evidence that fallow weeds may alter yield formation in rice and highlight the need for further investigations of the ecophysiological mechanism underlying the effect of fallow weeds on N uptake in rice.