Nitroethane (NE), 2-nitroethanol (NEOH) and 2-nitro-1-propanol (NPOH) were investigated in order to determine their inhibitory effects on in vitro ruminal fermentation and methane (CH4) production of a hay-rich substrate (alfalfa hay: maize meal = 4:1, w/w). The rumen liquor collected from cannulated Holstein dairy cows was incubated at 39 °C for 72 h. The addition of NE, NEOH and NPOH slowed down the fermentation process and notably decreased molar CH4 proportion by 96.8, 96.4 and 35.0%, respectively. The abundance of total methanogen and methanogens from the order Methanobacteriales were all decreased with NE, NEOH and NPOH supplementation. Meanwhile, the nitrocompound addition reduced mcrA gene expression, coenzyme F420 and F430 contents. The correlation analysis showed that CH4 production was correlated positively with the population abundance of total methanogens, Methanobacteriales, mcrA gene expression, coenzyme contents of F420 and F430. The nitrocompound addition decreased acetate concentration and increased propionate and butyrate concentrations in the culture fluid. In summary, both NE and NEOH addition presented nearly the same inhibitory effectiveness on in vitro CH4 production; they were more effective than NPOH. The results of the current study provide evidence that NE, NEOH and NPOH can dramatically decrease methanogen population, mcrA gene expression and the coenzyme content of F420 and F430 in ruminal methanogenesis.