This paper analyzes an abstract two-level algorithm for hybridizable discontinuous Galerkin (HDG) methods in a unified fashion. We use an extended version of the Xu-Zikatanov (X-Z) identity to derive a sharp estimate of the convergence rate of the algorithm, and show that the theoretical results also are applied to weak Galerkin (WG) methods. The main features of our analysis are twofold: one is that we only need the minimal regularity of the model problem; the other is that we do not require the triangulations to be quasi-uniform. Numerical experiments are provided to confirm the theoretical results.