By developing a class of dynamic stochastic general equilibrium models with nominal rigidities and assuming a two-country currency union with sovereign risk, we show that there is not necessarily a trade-off between the prevention of default risk and stabilizing inflation. Under optimal monetary and fiscal policy, comprising a de facto inflation stabilization policy, the tax rate as an optimal fiscal policy tool plays an important role in stabilizing inflation, although not completely because of the distorted steady state. Changes in the tax rate to minimize welfare costs via stabilizing inflation then improve the fiscal surplus, and because of this and the incompletely stabilized inflation, the default rate does not increase as much.