The CoCrNiMox (x = 0, 0.1, and 0.2 in molar ratio) medium entropy alloys (MEAs) were fabricated by vacuum arc melting, followed by cold rolling and annealing treatments. The X-ray diffraction (XRD), electron back-scattered diffraction (EBSD), and transmission electron microscopy (TEM) were employed to characterize the microstructures. It has been shown that the CoCrNi MEA has a single FCC phase and the Mo-containing MEAs contain (Cr, Mo)-rich σ precipitates. In addition, the Mo addition caused significant grain refinement, due to the fact that the presence of σ phase exerts a strong pinning effect on the grain boundary migration. The hardness testing results indicate an increment in Vickers hardness from 187.5 ± 4.5 Hv of CoCrNi alloy to 309.5 ± 10.3 Hv of CoCrNiMo0.2 alloy. The yield strength and ultimate tensile strength also increase from 339 ± 2 to 644 ± 5 MPa and from 810 ± 5 to 1071 ± 17 MPa, respectively, but the elongation drops from 88.4 ± 4.0% to 29.5 ± 7.6%. The grain refinement and the precipitation of σ phase make synergistic contribution to the reinforcement of Mo-containing CoCrNi-based MEAs. The details and explanations in this study may guide the future design and research of the CoCrNi-based quaternary alloys with enhanced properties.