A yield stress is added to Taylor's (1952, Proc. Royal Soc. A, 211, 225-239) model of a microscopic organism with a wavy cylindrical tail swimming through a viscous fluid. Viscoplastic slender-body theory is employed for the task, generalising existing results for Bingham fluid to the Herschel–Bulkley constitutive model. Numerical solutions are provided over a range of the two key parameters of the problem: the wave amplitude relative to the wavelength, and a Bingham number which describes the strength of the yield stress. Numerical solutions are supplemented with discussions of various limits of the problem in which analytical progress is possible. If the wave amplitude is sufficiently small, the yield stress of the material inevitably dominates the flow; the resulting ‘plastic locomotion’ results in swimming speeds that depend strongly on the swimming gait, and can, in some cases, even be negative. Conversely, when the yield stress is large, swimming becomes possible at the wave speed, with the swimmer sliding or burrowing along its centreline with a relatively high efficiency.