Motivated by the near invariance of model spaces for the backward shift, we introduce a general notion of $(X,Y)$-invariant operators. The relations between this class of operators and the near invariance properties of their kernels are studied. Those lead to orthogonal decompositions for the kernels, which generalize well-known orthogonal decompositions of model spaces. Necessary and sufficient conditions for those kernels to be nearly X-invariant are established. This general approach can be applied to a wide class of operators defined as compressions of multiplication operators, in particular to Toeplitz operators and truncated Toeplitz operators, to study the invariance properties of their kernels (general Toeplitz kernels).