Third-harmonic Brillouin backscattering (3HBBS) instability is investigated in the interaction of a picosecond extraordinary laser pulse with a homogeneous transversely magnetized underdense plasma. Nonlinear coupled equations that describe the instability are derived and solved for a weakly coupled regime to find the maximum growth rate. The nonlinearity arises through the combined effect of relativistic mass increase, static magnetic field, and ponderomotive acceleration of plasma electrons. The growth rate is found to decrease as the static magnetic field increases. It also increases by increasing both plasma density and laser intensity. It is also established that the growth rate of 3HBBS instability in a magnetized plasma is lower than that of fundamental Brillouin backscattering instability.
