Let $(G,\unicode[STIX]{x1D6EC})$ be a self-similar $k$-graph with a possibly infinite vertex set $\unicode[STIX]{x1D6EC}^{0}$. We associate a universal C*-algebra ${\mathcal{O}}_{G,\unicode[STIX]{x1D6EC}}$ to $(G,\unicode[STIX]{x1D6EC})$. The main purpose of this paper is to investigate the ideal structures of ${\mathcal{O}}_{G,\unicode[STIX]{x1D6EC}}$. We prove that there exists a one-to-one correspondence between the set of all $G$-hereditary and $G$-saturated subsets of $\unicode[STIX]{x1D6EC}^{0}$ and the set of all gauge-invariant and diagonal-invariant ideals of ${\mathcal{O}}_{G,\unicode[STIX]{x1D6EC}}$. Under some conditions, we characterize all primitive ideals of ${\mathcal{O}}_{G,\unicode[STIX]{x1D6EC}}$. Moreover, we describe the Jacobson topology of some concrete examples, which includes the C*-algebra of the product of odometers. On the way to our main results, we study self-similar $P$-graph C*-algebras in depth.