We carry out direct numerical simulations of turbulent Rayleigh–Bénard convection in a square box with rough conducting plates over the Rayleigh number range $10^7\leqslant Ra\leqslant 10^9$ and the Prandtl number range $0.01\leqslant Pr\leqslant 100$. In Zhang et al. (J. Fluid Mech., vol. 836, 2018, R2), it was reported that while the measured Nusselt number $Nu$ is enhanced at large roughness height $h$, the global heat transport is reduced at small $h$. The division between the two regimes yields a critical roughness height $h_c$, and we now focus on the effects of the Prandtl number ($Pr$) on $h_c$. Based on the variations of $h_c$, we identify three regimes for $h_c(Pr)$. For low $Pr$, thermal boundary layers become thinner with increasing $Pr$. This makes the boundary layers easier to be disrupted by rough elements, leading to the decrease of $h_c$ with increasing $Pr$. For moderate $Pr$, the corner-flow rolls become much more pronounced and suppress the global heat transport via the competition between the corner-flow rolls and the large-scale circulation (LSC). As a consequence, $h_c$ increases with increasing $Pr$ due to the intensification of the corner–LSC competition. For high $Pr$, the convective flow transitions to the plume-controlled regime. As the rough elements trigger much stronger and more frequent plume emissions, $h_c$ again decreases with increasing $Pr$.