Irregular cusps of orthogonal modular varieties

Abstract

Irregular cusps of an orthogonal modular variety are cusps where the lattice for Fourier expansion is strictly smaller than the lattice of translation. The presence of such a cusp affects the study of pluricanonical forms on the modular variety using modular forms. We study toroidal compactification over an irregular cusp, and clarify there the cusp form criterion for the calculation of Kodaira dimension. At the same time, we show that irregular cusps do not arise frequently: besides the cases when the group is neat or contains $-1$, we prove that the stable orthogonal groups of most (but not all) even lattices have no irregular cusp.

1 Introduction

Irregular cusps of a modular curve are cusps where the width of translation is strictly smaller than the width for Fourier expansion. It does not arise frequently, but does exist. At such a cusp, the vanishing order of cusp forms has to be considered carefully, especially when compared with that of pluricanonical forms (cf. [3, Sections 3.2 and 3.3]). In this article, we study and classify irregular cusps for orthogonal groups of signature $(2, b)$ , and clarify the effect of such cusps on the study of Kodaira dimension of orthogonal modular varieties.

Let L be a lattice of signature $(2, b)$ . Let ${\mathcal {D}}={\mathcal {D}_{L}}$ be the Hermitian symmetric domain attached to L, which is defined as either of the two connected components of the space

$$ \begin{align*} \{ {\mathbb{C}}\omega \in {{\mathbb P}}L_{{\mathbb{C}}} \: | \: (\omega, \omega)=0, \: (\omega, \bar{\omega})>0 \}. \end{align*} $$

We write ${\mathrm {O}^{+}(L)}$ for the subgroup of the orthogonal group $\mathrm {O}(L)$ that preserves the component ${\mathcal {D}}$ .

The domain ${\mathcal {D}}$ has zero-dimensional and one-dimensional cusps. For simplicity of exposition, we speak only of zero-dimensional cusps for the moment: in fact, the case of one-dimensional cusps can be reduced to that of adjacent zero-dimensional cusps (Proposition 6.3). A zero-dimensional cusp of ${\mathcal {D}}$ corresponds to a rank $1$ primitive isotropic sublattice I of L. Let ${U(I)_{\mathbb {Q}}}$ be the unipotent radical of the stabilizer of I in $\mathrm {O}^{+}(L_{{\mathbb {Q}}})$ . Then ${U(I)_{\mathbb {Q}}}$ is already abelian: it is a ${\mathbb {Q}}$ -vector space of dimension b (with a hyperbolic quadratic form). Let ${\Gamma }$ be a finite-index subgroup of ${\mathrm {O}^{+}(L)}$ . The cusp I is called an irregular cusp for ${\Gamma }$ if ${U(I)_{\mathbb {Q}}}\cap {\Gamma } \ne {U(I)_{\mathbb {Q}}}\cap {\langle \Gamma , -\mathrm {id} \rangle }$ . As we will explain, ${U(I)_{\mathbb {Z}}}={U(I)_{\mathbb {Q}}}\cap {\Gamma }$ is the lattice for Fourier expansion of ${\Gamma }$ -modular forms around I, while ${U(I)_{\mathbb {Z}}^{\prime }}={U(I)_{\mathbb {Q}}}\cap {\langle \Gamma , -\mathrm {id} \rangle }$ is the lattice of translation around I in the ${\Gamma }$ -action. We give several characterizations of irregularity (Proposition 3.1), including one suitable for explicit calculation.

Irregular cusps are rather rare: they do not exist when $-\mathrm {id}\in {\Gamma }$ or when ${\Gamma }$ is neat or when ${\Gamma }\subset \mathrm {SO}^{+}(L)$ with b odd. But they do exist, in infinitely many examples in every dimension (Section 4.5). Our particular interest is in the so-called stable orthogonal groups ${\widetilde {\mathrm {O}}^{+}(L)}$ of even lattices L, defined as the kernel of the reduction map ${\mathrm {O}^{+}(L)}\to \mathrm {O}(L^{\vee }/L)$ . This is the group that most frequently appears in the moduli problem related to orthogonal modular varieties. Our calculation concerning ${\widetilde {\mathrm {O}}^{+}(L)}$ can be summarized as follows.

Proposition 1.1 (Sections 4.1 and 4.5)

The stable orthogonal group ${\widetilde {\mathrm {O}}^{+}(L)}$ of an even lattice L has no irregular cusp unless $L^{\vee }/L \simeq {\mathbb {Z}}/8\oplus ({\mathbb {Z}}/2)^{\oplus a}$ or $L^{\vee }/L \simeq ({\mathbb {Z}}/4)^{\oplus 2}\oplus ({\mathbb {Z}}/2)^{\oplus a}$ as abelian groups. Conversely, if $L=U\oplus \langle -8 \rangle \oplus M$ or $L=U\oplus \langle -4 \rangle ^{\oplus 2} \oplus M$ with $M^{\vee }/M$ 2-elementary, then ${\widetilde {\mathrm {O}}^{+}(L)}$ has an irregular zero-dimensional cusp.

Consequently, we obtain classification for the following examples from moduli spaces (Section 4):

• • The modular group for $K3$ surfaces of degree $2d$ has an irregular cusp exactly when $d=4$ .

• • The modular group for irreducible symplectic manifolds of $K3^{[t+1]}$ -type with polarization of split type and degree $2d$ [10] has an irregular cusp exactly when $(t, d)=(1, 4), (2, 2), (4, 1)$ .

• • The modular group for O’Grady10 manifolds with polarization of split type and degree $2d$ [6], which is larger than ${\widetilde {\mathrm {O}}^{+}(L)}$ , has an irregular cusp exactly when $d=4$ .

• • Similarly, the modular group for deformation generalized Kummer varieties with polarization of split type and degree $2d$ [2] has an irregular cusp exactly when $(t, d)=(4, 1)$ .

• • We will also cover the groups considered in [4, 16, 18].

A subtle issue concerning irregular cusps, which is the main object of this article, is comparison of the vanishing order between cusp forms and pluricanonical forms. We take a toroidal compactification ${\mathcal {F}(\Gamma )^{\Sigma }}$ of the modular variety ${\mathcal {F}(\Gamma )}={\Gamma }\backslash {\mathcal {D}}$ . This is defined by choosing a finite collection $\Sigma =(\Sigma _{I})$ of suitable fans, one for each ${\Gamma }$ -equivalence class of rank $1$ primitive isotropic sublattices I of L. A ray $\sigma $ in $\Sigma _{I}$ corresponds to a boundary divisor $D(\sigma )$ of the torus embedding $\overline {{\mathcal {D}}/{U(I)_{\mathbb {Z}}}}$ , and thus determines a boundary divisor $\Delta (\sigma )$ of ${\mathcal {F}(\Gamma )^{\Sigma }}$ as the image of $D(\sigma )$ . The projection $\overline {{\mathcal {D}}/{U(I)_{\mathbb {Z}}}}\to {\mathcal {F}(\Gamma )^{\Sigma }}$ is ramified along $D(\sigma )$ (with index $2$ ) exactly when I is irregular and the ray $\sigma $ is irregular in the sense of Definition 3.2.

The vanishing order $\nu _{\sigma }(F)$ of a ${\Gamma }$ -modular form F at $D(\sigma )\subset \overline {{\mathcal {D}}/{U(I)_{\mathbb {Z}}}}$ can be measured by Fourier expansion (Section 8.2): this is done with ${U(I)_{\mathbb {Z}}}$ . On the other hand, the vanishing order of a pluricanonical form $\omega $ on ${\mathcal {F}(\Gamma )}$ should be measured at the level of $\Delta (\sigma ) \subset {\mathcal {F}(\Gamma )^{\Sigma }}$ : this is essentially done with ${U(I)_{\mathbb {Z}}^{\prime }}$ . When $\omega $ is m-canonical and corresponds to F (of weight $k=mb$ and character $\chi =\det ^{m}$ ), we have the relation (Proposition 8.7)

$$ \begin{align*} \nu_{\Delta(\sigma)}(\omega) = a_{\sigma} \cdot \nu_{\sigma}(F) - m, \end{align*} $$

where $a_{\sigma }=1$ if $\sigma $ is regular but $a_{\sigma }=1/2$ if $\sigma $ is irregular due to the boundary ramification. If we are involved only with modular forms of specific parity of weight k, namely k even for $\chi =1$ (e.g., [8, 13]) or $k \equiv b$ mod $2$ for $\chi =\det $ , we do not need to worry about irregular cusps because we can enlarge ${\Gamma }$ to ${\langle \Gamma , -\mathrm {id} \rangle }$ without any loss. However, if we use a modular form of weight in the remaining parity, we cannot add $-{\mathrm {id}}$ to ${\Gamma }$ , and have to be careful about the coefficient $a_{\sigma }=1/2$ at irregular rays $\sigma $ .

Gritsenko, Hulek, and Sankaran [7] gave a criterion, called the low weight cusp form trick, for ${\mathcal {F}(\Gamma )}$ to be of general type in terms of existence of a certain cusp form. It appears that irregular cusps are not covered in [7], essentially by assuming $-\mathrm {id}\in {\Gamma }$ , explicitly for one-dimensional cusps [7, p. 539] and implicitly for zero-dimensional cusps (see a remark below). In view of the coefficient $a_{\sigma }=1/2$ at irregular $\sigma $ , it seems that this criterion needs to be modified at such boundary divisors. The result is summarized as follows (compare with [7, Theorem 1.1]).

Theorem 1.2 (Theorem 8.9)

Let L be a lattice of signature $(2, b)$ with $b\geq 9$ , and let ${\Gamma }$ be a subgroup of ${\mathrm {O}^{+}(L)}$ of finite index. We take a ${\Gamma }$ -admissible collection $\Sigma =(\Sigma _{I})$ of fans so that $\Sigma _{I}$ is basic with respect to ${U(I)_{\mathbb {Q}}}\cap {\langle \Gamma , -\mathrm {id} \rangle }$ at every zero-dimensional cusp I. Assume that there exists a ${\Gamma }$ -cusp form F of weight $k<b$ and some character satisfying the following:

1. (1) F vanishes at the ramification divisor of ${\mathcal {D}}\to {\mathcal {F}(\Gamma )}$ .

2. (2) $\nu _{\sigma }(F)\geq 2$ at every irregular ray $\sigma $ at every irregular I.

Then ${\mathcal {F}(\Gamma )}$ is of general type.

The condition on $\Sigma $ is imposed in order to ensure that ${\mathcal {F}(\Gamma )^{\Sigma }}$ has canonical singularities [7, 13], and this can always be satisfied. When ${\Gamma }$ has no irregular cusp, the condition (2) is vacuous, and this is the criterion in [7]; the choice of $\Sigma $ does not matter with F and can be dropped (or hidden) from the criterion. Even when ${\Gamma }$ has an irregular cusp, if the weight k is even for $\chi =1$ or $k\equiv b$ mod $2$ for $\chi =\det $ , the condition (2) is still automatically satisfied by the cuspidality of F (Proposition 8.3). However, when ${\Gamma }$ has an irregular cusp and k belongs to the remaining parity, the condition (2) arises, and the choice of $\Sigma _{I}$ is then involved with F. Practically it would not be very easy to check (or achieve) $\nu _{\sigma }(F) \geq 2$ for specific F and $\Sigma _{I}$ . Probably the most plausible scenario would be to expect and check that the group ${\Gamma }$ in question has no irregular cusp. We could say that this is a small cost for using cusp forms of arbitrary weight.

By the examples discussed after Proposition 1.1, the general-type results in [2, 4, 10–7, 16, 18] are not affected. This is our essential purpose.

As a related remark, it should be remembered that in [1], subgroups of $\mathrm {O}^{+}(L_{{\mathbb {R}}})/\pm \mathrm {id}$ are considered, rather than of $\mathrm {O}^{+}(L_{{\mathbb {R}}})$ . This means that the given group ${{\Gamma }<{\mathrm {O}^{+}(L)}}$ is replaced by ${\langle \Gamma , -\mathrm {id} \rangle }/\pm {\mathrm {id}}$ . In this situation, it is not ${U(I)_{\mathbb {Z}}}={U(I)_{\mathbb {Q}}}\cap {\Gamma }$ but rather ${U(I)_{\mathbb {Z}}^{\prime }}={U(I)_{\mathbb {Q}}}\cap {\langle \Gamma , -\mathrm {id} \rangle }$ that is written as $U(F)_{{\mathbb {Z}}}$ in the notation of [1]. This is a subtle difference that may arise when working with [1] and that could cause overlooking of irregular cusps.

To conclude, irregular cusps are cusps where the lattice for Fourier expansion is smaller than the lattice of translation. It is the central element $-\mathrm {id}$ in the Lie group $\mathrm {O}^{+}(L_{{\mathbb {R}}})$ that is eventually responsible for the presence of such cusps. We need to be careful about such cusps when we use a cusp form of odd weight with $\chi =1$ or weight $k\not \equiv b$ mod $2$ with $\chi =\det $ for constructing a pluricanonical form on ${\mathcal {F}(\Gamma )^{\Sigma }}$ .

This article is organized as follows. In Section 2, we recall the structure of the stabilizer of a zero-dimensional cusp. In Section 3, we define and study irregular zero-dimensional cusps. In Section 4, we give examples of groups ${\Gamma }$ with/without irregular cusp. In Section 5, we recall the structure of the stabilizer of a one-dimensional cusp. In Section 6, we study irregular one-dimensional cusps. In Section 7, we study some basic properties of a toroidal compactification of ${\mathcal {F}(\Gamma )}$ . In Section 8, we prove Theorem 1.2. The main contents of this article are contained in Sections 3, 4, 6, and 8. Sections 2 and 5 are expository, but we tried to be rather self-contained because of the subtle nature of irregular cusps and for calculation of explicit examples in Section 4.

Throughout the article, a lattice usually means a free ${\mathbb {Z}}$ -module of finite rank endowed with a nondegenerate integral symmetric bilinear form $(\cdot , \cdot ) : L \times L \to {\mathbb {Z}}$ . In a few occasions, we use the word “lattice” just for a free ${\mathbb {Z}}$ -module of finite rank, but no confusion will likely to occur. The dual lattice ${\operatorname {Hom}}(L, {\mathbb {Z}})$ of L will be denoted by $L^{\vee }$ . A sublattice $I\subset L$ is called primitive when $L/I$ is free, and isotropic when $(I, I)\equiv 0$ . A lattice L is called even if $(l, l)\in 2{\mathbb {Z}}$ for every $l\in L$ , but this is not assumed except in Section 4. We write U for the even unimodular lattice of signature $(1, 1)$ given by the Gram matrix $\begin {pmatrix} 0 & 1 \\ 1 & 0 \end {pmatrix}$ .

2 Zero-dimensional cusps

Let L be a lattice of signature $(2, b)$ . We write $Q = Q_{L}$ for the isotropic quadric in ${{\mathbb P}}L_{{\mathbb {C}}}$ defined by $(\omega , \omega )=0$ . The Hermitian symmetric domain attached to L is the open set of Q

$$ \begin{align*} {\mathcal{D}} = {\mathcal{D}_{L}} = \{ {\mathbb{C}}\omega \in Q | (\omega, \bar{\omega})>0 \}^{+}, \end{align*} $$

where $+$ means the choice of a connected component. The domain ${\mathcal {D}}$ has two types of rational boundary components (cusps): zero-dimensional and one-dimensional cusps. They correspond to primitive isotropic sublattices of L of ranks $1$ and $2$ , respectively. In this section, we recall the structure of the stabilizer of a zero-dimensional cusp and partial toroidal compactification over it. Although the contents of this section are quite standard (cf. [7, 11, 12, 17]), we tried to be rather self-contained and explicit for two reasons: because of the subtle nature of irregular cusps (Section 3), and for the sake of calculation of explicit examples (Section 4).

2.1 Tube domain model

Throughout this section, we fix a rank $1$ primitive isotropic sublattice I of L. The zero-dimensional cusp corresponding to I is the point ${{\mathbb P}}I_{{\mathbb {C}}}$ of Q. We abbreviate $I^{\perp }=I^{\perp }\cap L$ and write

$$ \begin{align*} L(I) = (I^{\perp} / I) \otimes I. \end{align*} $$

Twisting by I, not choosing its generator, will be rather essential. The quadratic form on $I^{\perp }/I$ and an isomorphism $I\simeq {\mathbb {Z}}$ define a hyperbolic quadratic form on $L(I)$ . This is independent of the choice of $I\simeq {\mathbb {Z}}$ . We denote by ${\mathcal {C}_{I}}$ the positive cone in $L(I)_{{\mathbb {R}}}$ , namely a chosen connected component of $\{ w \in L(I)_{{\mathbb {R}}} \: | \: (w, w)>0 \}$ .

We write ${\mathcal {D}(I)}=Q - Q\cap {{\mathbb P}}I_{{\mathbb {C}}}^{\perp }$ . Then ${\mathcal {D}}$ is contained in ${\mathcal {D}(I)}$ . Indeed, if $[\omega ]\in {\mathcal {D}}\cap {{\mathbb P}}I_{{\mathbb {C}}}^{\perp }$ , the positive-definite plane $\langle \mathrm {Re}(\omega ), \mathrm {Im}(\omega ) \rangle $ would be contained in $I_{{\mathbb {R}}}^{\perp }/I_{{\mathbb {R}}}$ , which contradicts the hyperbolicity of $I_{{\mathbb {R}}}^{\perp }/I_{{\mathbb {R}}}$ . The linear projection ${{\mathbb P}}L_{{\mathbb {C}}}\dashrightarrow {{\mathbb P}}(L/I)_{{\mathbb {C}}}$ from the point ${{\mathbb P}}I_{{\mathbb {C}}}\in Q$ defines an isomorphism

$$ \begin{align*} {\mathcal{D}(I)} \: \stackrel{\simeq}{\to} \: {{\mathbb P}}(L/I)_{{\mathbb{C}}} - {{\mathbb P}}(I^{\perp}/I)_{{\mathbb{C}}}. \end{align*} $$

If we choose a rank $1$ sublattice $I'$ of L with $(I, I')\not \equiv 0$ , this defines the base point ${{\mathbb P}}(\langle I, I' \rangle _{{\mathbb {C}}}/I_{{\mathbb {C}}})$ of the affine space ${{\mathbb P}}(L/I)_{{\mathbb {C}}} - {{\mathbb P}}(I^{\perp }/I)_{{\mathbb {C}}}$ , and hence an isomorphism

$$ \begin{align*} {{\mathbb P}}(L/I)_{{\mathbb{C}}} - {{\mathbb P}}(I^{\perp}/I)_{{\mathbb{C}}} \simeq (I^{\perp}/I)_{{\mathbb{C}}}\otimes (I')_{{\mathbb{C}}}^{\vee} \simeq L(I)_{{\mathbb{C}}}. \end{align*} $$

The image of ${\mathcal {D}}\subset {\mathcal {D}(I)}$ by this series of isomorphisms is the tube domain in $L(I)_{{\mathbb {C}}}$ defined by

$$ \begin{align*} \mathcal{D}_{I} = \{ \: Z\in L(I)_{{\mathbb{C}}} \: | \: \mathrm{Im}(Z)\in {\mathcal{C}_{I}} \: \}. \end{align*} $$

In this way, we obtain the tube domain realization

(2.1) $$ \begin{align} {\mathcal{D}} \stackrel{\simeq}{\to} \mathcal{D}_{I} \subset L(I)_{{\mathbb{C}}} \end{align} $$

depending on the choice of $I'$ .

If we change $I'$ , the base point is changed, and the tube domain realization (2.1) is shifted by the corresponding translation of $L(I)_{{\mathbb {C}}}$ . For a given $I'$ , we can always find a (unique) isotropic line $\ne I_{{\mathbb {Q}}}$ from the hyperbolic plane $\langle I, I'\rangle _{{\mathbb {Q}}}$ . (Explicitly, if we take vectors $l\in I_{{\mathbb {Q}}}, l'\in I^{\prime }_{{\mathbb {Q}}}$ with $(l, l')=1$ , the vector $l'-2^{-1}(l', l')l$ generates this isotropic line.) This means that we can replace the given $I'$ to be isotropic without changing the base point. When $I'$ is isotropic, the inverse of (2.1) is given by

(2.2) $$ \begin{align} \mathcal{D}_{I} \to {\mathcal{D}}, \qquad z\otimes l\mapsto {\mathbb{C}}(l'+z-2^{-1}(z, z)l), \end{align} $$

where $z\in (I^{\perp }/I)_{{\mathbb {C}}}$ , $l\in I$ , and $l'\in I^{\prime }_{{\mathbb {Q}}}$ with $(l, l')=1$ , and we identify $(I^{\perp }/I)_{{\mathbb {C}}}\simeq \langle I, I' \rangle _{{\mathbb {C}}}^{\perp }\subset L_{{\mathbb {C}}}$ in the right side.

2.2 Stabilizer over ${\mathbb {Q}}$

Let ${\Gamma (I)_{\mathbb {Q}}}$ be the stabilizer of I in $\mathrm {O}^{+}(L_{{\mathbb {Q}}})$ . Note that we are not considering the stabilizer of $I_{{\mathbb {Q}}}$ , but of I. This is not restrictive when restricting to subgroups of ${\mathrm {O}^{+}(L)}$ . We put

$$ \begin{align*} {U(I)_{\mathbb{Q}}} = {\operatorname{Ker}}({\Gamma(I)_{\mathbb{Q}}} \to \mathrm{ O}(I_{{\mathbb{Q}}}^{\perp}/I_{{\mathbb{Q}}})\times \mathrm{GL}(I)). \end{align*} $$

This is the unipotent radical of ${\Gamma (I)_{\mathbb {Q}}}$ and can be explicitly described as follows. For a vector $m\otimes l$ of $L(I)_{{\mathbb {Q}}}$ , the Eichler transvection $E_{m\otimes l}\in {\Gamma (I)_{\mathbb {Q}}}$ is defined by (cf. [9, 17])

$$ \begin{align*} E_{m\otimes l}(v) = v - (\tilde{m}, v)l + (l, v)\tilde{m} - \frac{1}{2}(m, m)(l, v)l, \qquad v\in L_{{\mathbb{Q}}}, \end{align*} $$

where $\tilde {m}\in I^{\perp }_{{\mathbb {Q}}}=I^{\perp }_{{\mathbb {Q}}}\cap L_{{\mathbb {Q}}}$ is an arbitrary lift of $m\in (I^{\perp }/I)_{{\mathbb {Q}}}$ . This does not depend on the choice of $\tilde {m}$ . In particular, $E_{m\otimes l}(v) = v - (m, v)l$ when $v\in I^{\perp }_{{\mathbb {Q}}}$ . We have $E_{w}\circ E_{w'}=E_{w+w'}$ for $w, w'\in L(I)_{{\mathbb {Q}}}$ . Then we have the canonical isomorphism

$$ \begin{align*} L(I)_{{\mathbb{Q}}} \to {U(I)_{\mathbb{Q}}}, \qquad m\otimes l \mapsto E_{m\otimes l}. \end{align*} $$

We identify ${U(I)_{\mathbb {Q}}}$ with $L(I)_{{\mathbb {Q}}}$ in this way. We also identify $\textrm {O}(I_{{\mathbb {Q}}}^{\perp }/I_{{\mathbb {Q}}})\times \mathrm {GL}(I)$ with $\textrm {O}(L(I)_{{\mathbb {Q}}})\times \mathrm {GL}(I)$ by the canonical twisted isomorphism

$$ \begin{align*} \textrm{O}(I_{{\mathbb{Q}}}^{\perp}/I_{{\mathbb{Q}}})\times \mathrm{GL}(I) \to \textrm{O}(L(I)_{{\mathbb{Q}}})\times \mathrm{GL}(I), \qquad (\gamma_{1}, \gamma_{2})\mapsto (\gamma_{1}\otimes \gamma_{2}, \gamma_{2}). \end{align*} $$

We thus have the canonical exact sequence

(2.3) $$ \begin{align} 0 \to L(I)_{{\mathbb{Q}}}\to {\Gamma(I)_{\mathbb{Q}}} \stackrel{\pi}{\to} \mathrm{ O}^{+}(L(I)_{{\mathbb{Q}}})\times \mathrm{GL}(I) \to 1. \end{align} $$

If we choose a lift $(I^{\perp }/I)_{{\mathbb {Q}}}\hookrightarrow I^{\perp }_{{\mathbb {Q}}}$ of $(I^{\perp }/I)_{{\mathbb {Q}}}$ , or equivalently, a rank $1$ sublattice $I'$ of L with $(I, I')\not \equiv 0$ , the exact sequence (2.3) splits:

(2.4) $$ \begin{align} {\Gamma(I)_{\mathbb{Q}}} \simeq (\mathrm{O}^{+}(L(I)_{{\mathbb{Q}}})\times \mathrm{GL}(I))\ltimes L(I)_{{\mathbb{Q}}}. \end{align} $$

Here the lifted group $\mathrm {O}^{+}(L(I)_{{\mathbb {Q}}})$ acts on the lifted component $(I^{\perp }/I)_{{\mathbb {Q}}}\subset L_{{\mathbb {Q}}}$ through the natural isomorphism $\mathrm {O}(L(I)_{{\mathbb {Q}}})\simeq \mathrm { O}((I^{\perp }/I)_{{\mathbb {Q}}})$ , and $\mathrm {GL}(I)$ corresponds to $\{ \pm \mathrm {id}_{L} \}$ . Since $\gamma \circ E_{w} \circ \gamma ^{-1} =E_{\gamma w}$ for $\gamma \in {\Gamma (I)_{\mathbb {Q}}}$ , the adjoint action of ${\Gamma (I)_{\mathbb {Q}}}$ on ${U(I)_{\mathbb {Q}}}$ coincides with the natural action of ${\Gamma (I)_{\mathbb {Q}}}$ on $L(I)_{{\mathbb {Q}}}$ . Therefore, in the induced action of ${\mathrm {O}^{+}}(L(I)_{{\mathbb {Q}}})\times \mathrm {GL}(I)$ on ${U(I)_{\mathbb {Q}}}$ , $\mathrm {GL}(I)=\{ \pm 1 \}$ acts trivially, and ${\mathrm {O}^{+}}(L(I)_{{\mathbb {Q}}})$ acts by its natural action on $L(I)_{{\mathbb {Q}}}$ .

We take the tube domain realization ${\mathcal {D}}\to {\mathcal {D}}_{I}$ associated with (the same) $I'$ . Then the action of ${\Gamma (I)_{\mathbb {Q}}}$ on ${\mathcal {D}}$ is translated to the action of the right side of (2.4) on ${\mathcal {D}}_{I}$ . This is described as follows.

Lemma 2.1 In the action of the right side of (2.4) on ${\mathcal {D}}_{I}$ ,

1. (1) $E_{w}\in {U(I)_{\mathbb {Q}}}$ acts on ${\mathcal {D}}_{I}$ as the translation by $w\in L(I)_{{\mathbb {Q}}}$ on $L(I)_{{\mathbb {C}}}$ ;

2. (2) $\mathrm {O}^{+}(L(I)_{{\mathbb {Q}}})$ acts on ${\mathcal {D}}_{I}$ by its linear action on $L(I)_{{\mathbb {C}}}$ ;

3. (3) $\mathrm {GL}(I)=\{ \pm 1 \}$ acts on ${\mathcal {D}}_{I}$ trivially.

Proof This can be seen from direct calculation using (2.2).

2.3 Stabilizer over ${\mathbb {Z}}$

Now let ${\Gamma }$ be a subgroup of ${\mathrm {O}^{+}(L)}$ of finite index. We write

$$ \begin{align*} {\Gamma(I)_{\mathbb{Z}}} = {\Gamma(I)_{\mathbb{Q}}}\cap {\Gamma}, \quad {U(I)_{\mathbb{Z}}} = {U(I)_{\mathbb{Q}}}\cap {\Gamma}, \quad {\overline{\Gamma(I)}_{\mathbb{Z}}} = {\Gamma(I)_{\mathbb{Z}}}/{U(I)_{\mathbb{Z}}}. \end{align*} $$

Then ${U(I)_{\mathbb {Z}}}$ is a lattice on ${U(I)_{\mathbb {Q}}}$ . By definition, we have the exact sequence

(2.5) $$ \begin{align} 0 \to {U(I)_{\mathbb{Z}}} \to {\Gamma(I)_{\mathbb{Z}}} \to {\overline{\Gamma(I)}_{\mathbb{Z}}} \to 1. \end{align} $$

Although (2.3) splits, this does not mean that (2.5) splits. We write ${U(I)_{\mathbb {Q}/\mathbb {Z}}}={U(I)_{\mathbb {Q}}}/{U(I)_{\mathbb {Z}}}$ . This is the group of torsion points of the algebraic torus $T(I)=U(I)_{{\mathbb {C}}}/{U(I)_{\mathbb {Z}}}$ . We also put

$$ \begin{align*} {\overline{\Gamma(I)}_{\mathbb{Q}}} = \pi^{-1}(\mathrm{O}^{+}({U(I)_{\mathbb{Z}}})\times \mathrm{GL}(I)) / {U(I)_{\mathbb{Z}}}, \end{align*} $$

which makes sense because ${U(I)_{\mathbb {Z}}}$ is normal in $\pi ^{-1}(\mathrm {O}^{+}({U(I)_{\mathbb {Z}}})\times \mathrm {GL}(I))$ by definition. This group has the canonical exact sequence

(2.6) $$ \begin{align} 0 \to {U(I)_{\mathbb{Q}/\mathbb{Z}}} \to {\overline{\Gamma(I)}_{\mathbb{Q}}} \to \mathrm{ O}^{+}({U(I)_{\mathbb{Z}}})\times \mathrm{GL}(I) \to 1. \end{align} $$

Then ${\overline {\Gamma (I)}_{\mathbb {Z}}}$ is a subgroup of ${\overline {\Gamma (I)}_{\mathbb {Q}}}$ naturally. We have

(2.7) $$ \begin{align} {\overline{\Gamma(I)}_{\mathbb{Z}}}\cap {U(I)_{\mathbb{Q}/\mathbb{Z}}} = \{ 0 \} \end{align} $$

by the definition ${U(I)_{\mathbb {Z}}}={\Gamma (I)_{\mathbb {Z}}}\cap {U(I)_{\mathbb {Q}}}$ of ${U(I)_{\mathbb {Z}}}$ .

We choose a rank $1$ sublattice $I'\subset L$ with $(I, I')\not \equiv 0$ and accordingly take a tube domain realization of ${\mathcal {D}}$ and a splitting of ${\Gamma (I)_{\mathbb {Q}}}$ . Dividing by ${U(I)_{\mathbb {Z}}}$ and writing ${\mathcal {X}(I)}={\mathcal {D}}/{U(I)_{\mathbb {Z}}}$ , we obtain isomorphisms

$$ \begin{align*} {\mathcal{X}(I)} \simeq \mathcal{D}_{I}/{U(I)_{\mathbb{Z}}} \quad \subset \quad {\mathcal{D}(I)}/{U(I)_{\mathbb{Z}}} \simeq T(I), \end{align*} $$

(2.8) $$ \begin{align} {\overline{\Gamma(I)}_{\mathbb{Q}}} \: \simeq \: (\mathrm{O}^{+}({U(I)_{\mathbb{Z}}})\times \mathrm{ GL}(I))\ltimes {U(I)_{\mathbb{Q}/\mathbb{Z}}}, \end{align} $$

both depending on the choice of $I'$ . By Lemma 2.1, the natural action of ${\overline {\Gamma (I)}_{\mathbb {Q}}}$ on ${\mathcal {X}(I)}$ is translated to the standard action of $(\mathrm {O}^{+}({U(I)_{\mathbb {Z}}})\times \mathrm {GL}(I))\ltimes {U(I)_{\mathbb {Q}/\mathbb {Z}}}$ on $T(I)$ . Here $\mathrm {O}^{+}({U(I)_{\mathbb {Z}}})$ acts by torus automorphisms fixing the identity, $\mathrm {GL}(I)$ acts trivially, and ${U(I)_{\mathbb {Q}/\mathbb {Z}}}$ acts by translation.

By (2.7), the projection ${\overline {\Gamma (I)}_{\mathbb {Z}}}\to \mathrm {O}^{+}({U(I)_{\mathbb {Z}}})\times \mathrm {GL}(I)$ is injective. But this does not mean that ${\overline {\Gamma (I)}_{\mathbb {Z}}}$ as a subgroup of ${\overline {\Gamma (I)}_{\mathbb {Q}}}$ is contained in the lifted subgroup $\mathrm {O}^{+}({U(I)_{\mathbb {Z}}})\times \mathrm {GL}(I)$ in (2.8). Thus, the action of ${\overline {\Gamma (I)}_{\mathbb {Z}}}$ on ${\mathcal {X}(I)}$ may have translation component.

Remark 2.2 Let $I={\mathbb {Z}}l$ and $\Gamma (l)_{{\mathbb {Z}}}<{\Gamma (I)_{\mathbb {Z}}}$ be the kernel of ${\Gamma (I)_{\mathbb {Z}}}\to \mathrm {GL}(I)$ . In the case $-{\mathrm {id}}\in {\Gamma }$ , we have ${\Gamma (I)_{\mathbb {Z}}}=\Gamma (l)_{{\mathbb {Z}}}\times \{ \pm {\mathrm {id}} \}$ , so we may replace ${\Gamma (I)_{\mathbb {Z}}}$ by $\Gamma (l)_{{\mathbb {Z}}}$ when considering action on ${\mathcal {D}}$ , as was done in [13, Appendix]. (The last sentence of [13, Remark A.8] for ${\Gamma }={\widetilde {\mathrm {O}}^{+}(L)}$ should be understood under the condition $\Gamma (l)_{{\mathbb {Z}}}={\Gamma (I)_{\mathbb {Z}}}$ (e.g., $\mathrm {div}(I)>2$ ) or $A_{L}$ 2-elementary, or $\mathrm {div}(I)=1$ .)

2.4 Partial toroidal compactification

We recall partial toroidal compactification of ${\mathcal {X}(I)}={\mathcal {D}}/{U(I)_{\mathbb {Z}}}$ following [1]. We put a ${\mathbb {Q}}$ -structure on $U(I)_{{\mathbb {R}}}$ by ${U(I)_{\mathbb {Q}}}\simeq L(I)_{{\mathbb {Q}}}$ . We write ${\mathcal {C}_{I}^{+}}={\mathcal {C}_{I}}\cup \bigcup _{w}{\mathbb {R}}_{\geq 0}w$ , where w ranges over all isotropic vectors of $L(I)_{{\mathbb {Q}}}$ in the closure of ${\mathcal {C}_{I}}$ . A rational polyhedral cone decomposition (fan) $\Sigma =(\sigma _{\alpha })_{\alpha }$ in $U(I)_{{\mathbb {R}}}$ is called ${\Gamma (I)_{\mathbb {Z}}}$ -admissible [1] if the support of $\Sigma $ is ${\mathcal {C}_{I}^{+}}$ , $\Sigma $ is preserved under the adjoint (= natural) action of ${\Gamma (I)_{\mathbb {Z}}}$ on $U(I)_{{\mathbb {R}}}=L(I)_{{\mathbb {R}}}$ , and there are only finitely many cones up to the action of ${\Gamma (I)_{\mathbb {Z}}}$ . Isotropic rays in $\Sigma $ correspond to rational isotropic lines in $L(I)_{{\mathbb {Q}}}$ (hence independent of $\Sigma $ ), which in turn correspond to rank $2$ primitive isotropic sublattices J of L containing I.

The fan $\Sigma $ defines a torus embedding $T(I)\hookrightarrow T(I)^{\Sigma }$ of the torus $T(I)=U(I)_{{\mathbb {C}}}/{U(I)_{\mathbb {Z}}}$ . Each ray $\sigma $ of $\Sigma $ defines a sub-torus embedding $T(I)\hookrightarrow T(I)^{\sigma }\subset T(I)^{\Sigma }$ , isomorphic to $({\mathbb {C}}^{\times })^{b} \hookrightarrow {\mathbb {C}}\times ({\mathbb {C}}^{\times })^{b-1}$ , whose unique boundary divisor is the quotient torus defined by the quotient lattice ${U(I)_{\mathbb {Z}}}/({\mathbb {R}}\sigma \cap {U(I)_{\mathbb {Z}}})$ . The character group of this boundary torus is $\sigma ^{\perp }\cap U(I)_{{\mathbb {Z}}}^{\vee }$ . Here we regard $U(I)_{{\mathbb {Z}}}^{\vee }$ as a lattice on ${U(I)_{\mathbb {Q}}}$ by the quadratic form on ${U(I)_{\mathbb {Q}}}=L(I)_{{\mathbb {Q}}}$ , which gives the pairing between ${U(I)_{\mathbb {Z}}}$ and $U(I)_{{\mathbb {Z}}}^{\vee }$ .

We take a tube domain realization of ${\mathcal {D}}$ by choosing $I'\subset L$ with $(I, I')\not \equiv 0$ . Then let ${\mathcal {X}(I)^{\Sigma }}$ be the interior of the closure of ${\mathcal {X}(I)}\simeq \mathcal {D}_{I}/{U(I)_{\mathbb {Z}}}$ in $T(I)^{\Sigma }$ . This embedding ${\mathcal {X}(I)}\hookrightarrow {\mathcal {X}(I)^{\Sigma }}$ is the partial toroidal compactification over I defined by the fan $\Sigma $ . It is ${\overline {\Gamma (I)}_{\mathbb {Z}}}$ -equivariant, and does not depend on the choice of $I'$ . We can think of ${\mathcal {X}(I)^{\Sigma }}$ as giving a local chart for the boundary points of a full toroidal compactification lying over the I-cusp (see Section 7), like ${\mathcal {D}}$ gives a local chart for the interior points in ${\Gamma }\backslash {\mathcal {D}}$ .

3 Irregular zero-dimensional cusps

We now study irregular zero-dimensional cusps. Let ${\Gamma }$ be a finite-index subgroup of ${\mathrm {O}^{+}(L)}$ , and let I be a rank $1$ primitive isotropic sublattice of L. We keep the notation from Section 2. We will define irregularity in two stages: irregularity of a cusp (Section 3.1), and irregularity of a toroidal boundary divisor over (or adjacent to) an irregular cusp (Section 3.2). The first stage is concerned only with ${\Gamma }$ , but the second stage is also involved with a ${\Gamma (I)_{\mathbb {Z}}}$ -admissible fan.

3.1 Irregularity

We give several equivalent definitions of irregularity of a zero-dimensional cusp in the following form.

Proposition 3.1 The following conditions are equivalent.

1. (1) ${U(I)_{\mathbb {Z}}}\ne {U(I)_{\mathbb {Z}}^{\prime }}$ where ${U(I)_{\mathbb {Z}}^{\prime }}={U(I)_{\mathbb {Q}}}\cap {\langle \Gamma , -\mathrm {id} \rangle }$ .

2. (2) $-{\mathrm {id}}\not \in {\Gamma }$ and $-E_{w}\in {\Gamma (I)_{\mathbb {Z}}}$ for some $w\in L(I)_{{\mathbb {Q}}}$ .

3. (3) $-{\mathrm {id}}\not \in {\Gamma }$ and ${\overline {\Gamma (I)}_{\mathbb {Z}}}\to {\mathrm { O}^{+}}({U(I)_{\mathbb {Z}}})$ is not injective.

4. (4) ${\overline {\Gamma (I)}_{\mathbb {Z}}}$ contains an element which acts by a nonzero translation on ${{\mathcal {X}(I)}={\mathcal {D}}/{U(I)_{\mathbb {Z}}}}$ .

When these hold, we have ${U(I)_{\mathbb {Z}}^{\prime }}/{U(I)_{\mathbb {Z}}} = \langle E_{w} \rangle \simeq {\mathbb {Z}}/2$ and

$$ \begin{align*} {\operatorname{Ker}}({\overline{\Gamma(I)}_{\mathbb{Z}}}\to {\mathrm{O}^{+}}({U(I)_{\mathbb{Z}}})) = \langle -E_{w} \rangle \simeq {\mathbb{Z}}/2, \end{align*} $$

and the translation in $(4)$ is given by $[w]\in {U(I)_{\mathbb {Q}/\mathbb {Z}}}$ and is unique.

Definition 3.1 We say that the zero-dimensional cusp I is irregular for ${\Gamma }$ when these properties hold, and regular otherwise.

Proof $(1) \Rightarrow (2)$ : Since ${\Gamma }\ne {\langle \Gamma , -\mathrm {id} \rangle }$ , we have $-{\mathrm {id}}\not \in {\Gamma }$ . Let $E_{w}\in {U(I)_{\mathbb {Z}}^{\prime }}$ , but $E_{w}\not \in {U(I)_{\mathbb {Z}}}$ . Since ${\langle \Gamma , -\mathrm {id} \rangle }={\Gamma }\sqcup -{\Gamma }$ , we have $E_{w}\in -{\Gamma }$ , and so $-E_{w}\in {\Gamma }$ . Note that ${U(I)_{\mathbb {Z}}^{\prime }}/{U(I)_{\mathbb {Z}}}\simeq {\langle \Gamma , -\mathrm {id} \rangle }/{\Gamma }$ is of order $2$ , and so ${U(I)_{\mathbb {Z}}^{\prime }}/{U(I)_{\mathbb {Z}}} = \langle E_{w} \rangle $ .

$(2) \Rightarrow (1)$ : If $-E_{w}\in {\Gamma (I)_{\mathbb {Z}}}$ and $-{\mathrm {id}}\not \in {\Gamma }$ , then $E_{w}\not \in {U(I)_{\mathbb {Z}}}$ , but $E_{w}\in {U(I)_{\mathbb {Z}}^{\prime }}$ .

$(2) \Rightarrow (3)$ : Since $-E_{w}$ acts on $L(I)_{{\mathbb {Q}}}={U(I)_{\mathbb {Q}}}$ trivially, its image in ${\overline {\Gamma (I)}_{\mathbb {Z}}}$ is contained in the kernel of ${\overline {\Gamma (I)}_{\mathbb {Z}}}\to {\mathrm {O}^{+}}({U(I)_{\mathbb {Z}}})$ .

$(3) \Rightarrow (2)$ : Recall from (2.6) that the kernel of ${\overline {\Gamma (I)}_{\mathbb {Q}}}\to {\mathrm {O}^{+}}({U(I)_{\mathbb {Z}}})$ is

$$ \begin{align*} {\mathrm{GL}}(I)\times {U(I)_{\mathbb{Q}/\mathbb{Z}}} = {U(I)_{\mathbb{Q}/\mathbb{Z}}} \sqcup (-{\mathrm{ id}})\cdot {U(I)_{\mathbb{Q}/\mathbb{Z}}}. \end{align*} $$

Since ${\overline {\Gamma (I)}_{\mathbb {Z}}}\cap {U(I)_{\mathbb {Q}/\mathbb {Z}}} = \{ 0 \}$ by (2.7), a nontrivial element of the kernel of ${\overline {\Gamma (I)}_{\mathbb {Z}}}\to {\mathrm { O}^{+}}({U(I)_{\mathbb {Z}}})$ must be contained in $(-{\mathrm {id}})\cdot {U(I)_{\mathbb {Q}/\mathbb {Z}}}$ , hence, is the image of $-E_{w}$ for some $w\in L(I)_{{\mathbb {Q}}}$ . This also shows that the kernel is ${\mathbb {Z}}/2$ generated by $-E_{w}$ .

$(2) \Rightarrow (4)$ : The element $-E_{w}$ of ${\overline {\Gamma (I)}_{\mathbb {Z}}}$ acts on ${\mathcal {X}(I)}$ by the translation by $[w] \in {U(I)_{\mathbb {Q}/\mathbb {Z}}}$ . Since $-{\mathrm {id}}\not \in {\Gamma }$ , we have $E_{w}\not \in {U(I)_{\mathbb {Z}}}$ . This means that $[w]\ne 0 \in {U(I)_{\mathbb {Q}/\mathbb {Z}}}$ .

$(4) \Rightarrow (2), (3)$ : We choose a splitting of ${\overline {\Gamma (I)}_{\mathbb {Q}}}$ as in (2.8) and express an element of ${\overline {\Gamma (I)}_{\mathbb {Z}}}\subset {\overline {\Gamma (I)}_{\mathbb {Q}}}$ as $\gamma =(\gamma _{1}, \gamma _{2}, [w])$ accordingly, where $\gamma _{1}\in {\mathrm {O}^{+}}({U(I)_{\mathbb {Z}}})$ , $\gamma _{2}\in \mathrm {GL}(I)$ and $[w]\in {U(I)_{\mathbb {Q}/\mathbb {Z}}}$ . If $\gamma $ acts on ${\mathcal {X}(I)}$ by a nonzero translation, we must have $\gamma _{1}={\mathrm {id}}_{L(I)}$ and the translation is given by $[w]\in {U(I)_{\mathbb {Q}/\mathbb {Z}}}$ . Therefore, $\gamma $ is contained in the kernel of the projection to ${\mathrm {O}^{+}}({U(I)_{\mathbb {Z}}})$ . Since ${\overline {\Gamma (I)}_{\mathbb {Z}}}\cap {U(I)_{\mathbb {Q}/\mathbb {Z}}} = \{ 0 \}$ by (2.7) and $[w] \ne 0$ , we have $\gamma _{2}=-{\mathrm {id}}_{I}$ . Thus, $\gamma =-E_{w}$ . Finally, we have $-{\mathrm {id}}\not \in {\Gamma }$ , for otherwise $E_{w}=-\gamma $ would be contained in ${U(I)_{\mathbb {Z}}}$ and then $[w]=0\in {U(I)_{\mathbb {Q}/\mathbb {Z}}}$ .

Remark 3.2 Let $\Gamma (l)_{{\mathbb {Z}}}<{\Gamma (I)_{\mathbb {Z}}}$ be as in Remark 2.2. By the condition (3), I is irregular if and only if $-{\mathrm {id}}\not \in {\Gamma }$ , $\Gamma (l)_{{\mathbb {Z}}}\ne {\Gamma (I)_{\mathbb {Z}}}$ , and $\Gamma (l)_{{\mathbb {Z}}}$ and ${\Gamma (I)_{\mathbb {Z}}}$ have the same image in ${\mathrm { O}^{+}}({U(I)_{\mathbb {Z}}})$ . We do not use this characterization.

The condition (2) is useful for explicit calculation (Section 4). We give some immediate consequences.

Corollary 3.3 The group ${\Gamma }$ has no irregular cusp when $-{\mathrm {id}}\in {\Gamma }$ or when ${\Gamma }$ is neat or when ${\Gamma }<\mathrm {SO}^{+}(L)$ with b odd.

Proof The case $-{\mathrm {id}}\in {\Gamma }$ is obvious. When ${\Gamma }$ is neat, the subquotient ${\overline {\Gamma (I)}_{\mathbb {Z}}}$ is torsion-free, so it does not contain an element of finite order like $-E_{w}$ . When b is odd, $-E_{w}$ has determinant $(-1)^{b+2}=-1$ , so a subgroup of $\mathrm {SO}^{+}(L)$ never contains such an element.

Corollary 3.4 When b is even, ${\Gamma }$ has an irregular cusp if and only if ${\Gamma }\cap \mathrm {SO}^{+}(L)$ has an irregular cusp.

Proof When b is even, both $-{\mathrm {id}}$ and $-E_{w}$ are contained in $\mathrm {SO}^{+}(L)$ .

Corollary 3.5 If ${\Gamma }$ has an irregular cusp, any $\Gamma ' < {\mathrm {O}^{+}(L)}$ with $\Gamma ' \supset {\Gamma }$ and $-{\mathrm {id}}\not \in \Gamma '$ has an irregular cusp. Equivalently, if $-{\mathrm {id}}\not \in {\Gamma }$ and ${\Gamma }$ has no irregular cusp, any subgroup of ${\Gamma }$ of finite index has no irregular cusp.

The lattice ${U(I)_{\mathbb {Z}}^{\prime }}={U(I)_{\mathbb {Q}}}\cap {\langle \Gamma , -\mathrm {id} \rangle }$ is the projection image of

$$ \begin{align*} U(I)_{{\mathbb{Z}}}^{\star} = (\{ \pm {\mathrm{id}} \} \cdot {U(I)_{\mathbb{Q}}}) \cap {\Gamma} = {\operatorname{Ker}}({\Gamma(I)_{\mathbb{Z}}}\to {\mathrm{O}^{+}}({U(I)_{\mathbb{Z}}})) \end{align*} $$

in ${U(I)_{\mathbb {Q}}}$ . Thus, ${U(I)_{\mathbb {Z}}^{\prime }}$ is the lattice of translation in the ${\Gamma (I)_{\mathbb {Z}}}$ -action on the tube domain model. We have

(3.1) $$ \begin{align} U(I)_{{\mathbb{Z}}}^{\star}/{U(I)_{\mathbb{Z}}} = \begin{cases} \langle -{\mathrm{id}} \rangle \simeq {\mathbb{Z}}/2, & -{\mathrm{id}}\in {\Gamma}, \\ \{ 1 \}, & -{\mathrm{id}}\not\in {\Gamma}, I \; \textrm{regular}, \\ \langle -E_{w} \rangle \simeq {\mathbb{Z}}/2, & I \; \textrm{irregular}. \end{cases} \end{align} $$

This gives yet another characterization of irregularity: $-{\mathrm {id}}\not \in {\Gamma }$ and ${U(I)_{\mathbb {Z}}}\ne U(I)_{{\mathbb {Z}}}^{\star }$ .

As we will explain in Section 8.1, ${U(I)_{\mathbb {Z}}}$ is the lattice for Fourier expansion of ${\Gamma }$ -modular forms around I. Thus, irregular zero-dimensional cusps are those cusps whose lattice of translation is larger than the lattice for Fourier expansion.

Remark 3.6 In the case $b=1$ , we have an accidental isomorphism $\mathrm {SO}^{+}(2, 1)\simeq \mathrm {PSL}(2, {\mathbb {R}})$ which induces an isomorphism between the type IV domain here and the upper half-plane. However, $\mathrm {O}^{+}(2, 1)=\mathrm {SO}^{+}(2, 1) \times \{ \pm \mathrm {id} \}$ and $\mathrm {SL}(2, {\mathbb {R}})$ are different double covers of $\mathrm {SO}^{+}(2, 1)\simeq \mathrm {PSL}(2, {\mathbb {R}})$ . Therefore, although we have the perfect analogy

$$ \begin{align*} {U(I)_{\mathbb{Z}}} \leftrightarrow \left\{ \begin{pmatrix} 1 & \ast \\ 0 & 1 \end{pmatrix} \in {\Gamma} \right\}, \end{align*} $$

$$ \begin{align*} {U(I)_{\mathbb{Z}}^{\prime}} \leftrightarrow \left\{ \begin{pmatrix} 1 & \ast \\ 0 & 1 \end{pmatrix} \in \langle {\Gamma}, -1 \rangle \right\}, \end{align*} $$

subgroups of $\mathrm {O}^{+}(2, 1)$ which have irregular cusps never correspond to subgroups of $\mathrm {SL}(2, {\mathbb {R}})$ which have irregular cusps (in the classical sense [3]): they live in different covers of $\mathrm {SO}^{+}(2, 1)\simeq \mathrm {PSL}(2, {\mathbb {R}})$ . Subgroups of $\mathrm {SO}^{+}(2, 1)\simeq \mathrm {PSL}(2, {\mathbb {R}})$ have no irregular cusp anyway.

3.2 Irregular boundary divisors

Let $\Sigma =(\sigma _{\alpha })$ be a ${\Gamma (I)_{\mathbb {Z}}}$ -admissible fan in $U(I)_{{\mathbb {R}}}$ , and let ${\mathcal {X}(I)}\hookrightarrow {\mathcal {X}(I)^{\Sigma }}$ be the partial compactification defined in Section 2.4. For a ray $\sigma $ in $\Sigma $ , we denote by $D(\sigma )\subset {\mathcal {X}(I)^{\Sigma }}$ the corresponding boundary divisor. When I is irregular, these boundary divisors are divided into two types as follows.

Proposition 3.7 Let I be an irregular zero-dimensional cusp for ${\Gamma }$ . Let $-E_{w}\in {\Gamma (I)_{\mathbb {Z}}}$ . The following conditions for a ray $\sigma $ in $\Sigma $ are equivalent:

1. (1) $\sigma \cap {U(I)_{\mathbb {Z}}} \ne \sigma \cap {U(I)_{\mathbb {Z}}^{\prime }}$ .

2. (2) $-E_{w}$ acts trivially on the boundary divisor $D(\sigma )$ .

3. (3) $D(\sigma )$ is fixed by some nontrivial element of ${\overline {\Gamma (I)}_{\mathbb {Z}}}$ .

When these hold, the element in $(3)$ is given by $-E_{w}$ . In particular, it is unique, independent of $\sigma $ , and of order $2$ .

Definition 3.2 When these properties hold, we call $\sigma $ an irregular ray and $D(\sigma )$ an irregular boundary divisor. Otherwise, we call $\sigma $ regular. For the sake of completeness, we call any ray $\sigma $ regular when I is regular.

Proof $(1) \Leftrightarrow (2)$ : Recall from Lemma 2.1 that $-E_{w}$ acts on ${\mathcal {X}(I)}\subset T(I)$ as the translation by $[w]\in {U(I)_{\mathbb {Q}/\mathbb {Z}}}$ . A Zariski open set of $D(\sigma )$ is the quotient torus (or its analytic open set) associated with the quotient lattice ${U(I)_{\mathbb {Z}}}/\Lambda _{\sigma }$ where $\Lambda _{\sigma }= {\mathbb {R}}\sigma \cap {U(I)_{\mathbb {Z}}}$ . Hence, $-E_{w}$ acts on $D(\sigma )$ as the translation by the image of $[w]$ in ${U(I)_{\mathbb {Q}}}/({U(I)_{\mathbb {Z}}}+(\Lambda _{\sigma })_{{\mathbb {Q}}})$ . This is trivial if and only if $w\in {U(I)_{\mathbb {Z}}}+(\Lambda _{\sigma })_{{\mathbb {Q}}}$ , which in turn is equivalent to $\Lambda _{\sigma } \ne {\mathbb {R}}\sigma \cap {U(I)_{\mathbb {Z}}^{\prime }}$ . In this case, $-E_{w}$ acts by $-1$ on the normal torus $(\Lambda _{\sigma })_{{\mathbb {C}}}/\Lambda _{\sigma }\simeq {\mathbb {C}}^{\times }$ .

$(2) \Rightarrow (3)$ is obvious.

$(3) \Rightarrow (2)$ : Suppose that $\gamma \in {\overline {\Gamma (I)}_{\mathbb {Z}}}$ acts trivially on $D(\sigma )$ . Let $\gamma _{1}$ be the image of $\gamma $ in ${\mathrm {O}^{+}}({U(I)_{\mathbb {Z}}})$ . Then $\gamma _{1}$ must preserve $\sigma \cap {U(I)_{\mathbb {Z}}}$ and act trivially on ${U(I)_{\mathbb {Z}}}/\Lambda _{\sigma }$ . Hence, $\gamma _{1}$ acts trivially on $\Lambda _{\sigma }$ and $\Lambda _{\sigma }^{\perp }$ , and so $\gamma _{1}={\mathrm {id}}$ . This implies that $\gamma $ is contained in the kernel of ${\overline {\Gamma (I)}_{\mathbb {Z}}}\to {\mathrm { O}^{+}}({U(I)_{\mathbb {Z}}})$ , whence $\gamma =-E_{w}$ by Proposition 3.1. Therefore, $-E_{w}$ acts trivially on $D(\sigma )$ .

Corollary 3.8 When I is irregular, the quotient map ${\mathcal {X}(I)^{\Sigma }}\to {\mathcal {X}(I)^{\Sigma }}/{\overline {\Gamma (I)}_{\mathbb {Z}}}$ is ramified along the irregular boundary divisors with ramification index $2$ , caused by the common subgroup $\langle -E_{w} \rangle \simeq {\mathbb {Z}}/2$ of ${\overline {\Gamma (I)}_{\mathbb {Z}}}$ , and not ramified along other boundary divisors. When I is regular, ${\mathcal {X}(I)^{\Sigma }}\to {\mathcal {X}(I)^{\Sigma }}/{\overline {\Gamma (I)}_{\mathbb {Z}}}$ is not ramified along any boundary divisor.

Proof It remains to supplement the argument in the case I is regular. If $\gamma \in {\overline {\Gamma (I)}_{\mathbb {Z}}}$ fixes a boundary divisor, we see that $\gamma $ acts trivially on ${U(I)_{\mathbb {Z}}}$ by the same argument as $(3)\Rightarrow (2)$ above. When $-{\mathrm {id}}\not \in {\Gamma }$ , ${\overline {\Gamma (I)}_{\mathbb {Z}}}\to {\mathrm {O}^{+}}({U(I)_{\mathbb {Z}}})$ is injective by the condition (3) of Proposition 3.1, so we find that $\gamma = {\mathrm {id}}$ . When $-{\mathrm {id}}\in {\Gamma }$ , the kernel of ${\overline {\Gamma (I)}_{\mathbb {Z}}}\to {\mathrm { O}^{+}}({U(I)_{\mathbb {Z}}})$ is $\{ \pm {\mathrm {id}} \}$ , so $\gamma =\pm {\mathrm {id}}$ , which acts trivially on ${\mathcal {X}(I)}$ .

4 Examples

In this section, we study some examples of groups with/without irregular cusp. Logically, this section should be read after Section 6 where we complete the discussion of irregular one-dimensional cusps. But we encourage the reader to read this section just after Section 3 for the following two reasons. First, most of Sections 5 and 6 is designed for Sections 7 and 8, while the only result from Sections 5 and 6 we need in this section is Corollary 6.4, which just says that ${\Gamma }$ has no irregular one-dimensional cusp if it has no irregular zero-dimensional cusp. Second, it is Proposition 3.1(2), which is frequently used in this section, so we do not want to put this section too far from it.

We assume that the lattice L is even in this section. The quotient $A_{L}=L^{\vee }/L$ is called the discriminant group of L, equipped with a canonical quadratic form $A_{L}\to {\mathbb {Q}}/2{\mathbb {Z}}$ called the discriminant form. If I is a rank $1$ primitive isotropic sublattice of L, we write $\mathrm {div}(I)$ for the positive generator of the ideal $(I, L)\subset {\mathbb {Z}}$ . Then $I^{\ast }=\mathrm {div}(I)^{-1}I$ is primitive in $L^{\vee }$ , and we have a canonical isometry $I^{\perp }\cap L^{\vee }/I^{\ast } \simeq (I^{\perp }/I)^{\vee }$ .

4.1 Stable orthogonal groups

Let L be an even lattice of signature $(2, b)$ . Let ${\widetilde {\mathrm {O}}^{+}(L)}<{\mathrm {O}^{+}(L)}$ be the kernel of the reduction map ${\mathrm { O}^{+}(L)}\to \mathrm {O}(A_{L})$ , called the stable orthogonal group or the discriminant kernel. The following was asserted in [13, p. 901]. We supplement the proof for the sake of completeness.

Lemma 4.1 Let I be a rank $1$ primitive isotropic sublattice of L. For ${\Gamma }={\widetilde {\mathrm {O}}^{+}(L)}$ , we have ${U(I)_{\mathbb {Z}}}=L(I)$ .

Proof We take a generator l of I. The inclusion $L(I)\subset {U(I)_{\mathbb {Z}}}$ can be checked by testing the definition of $E_{m\otimes l}(v)$ for $v\in L^{\vee }$ and $m\in I^{\perp }/I$ , taking a lift of m from $I^{\perp }\cap L$ . Conversely, if $E_{m\otimes l}\in {\widetilde {\mathrm {O}}^{+}(L)}$ for a vector $m\in I^{\perp }_{{\mathbb {Q}}}/I_{{\mathbb {Q}}}$ , then $E_{m\otimes l}(v) = v - (m, v)l$ must be contained in $v+L$ for $v\in I^{\perp }\cap L^{\vee }$ . This implies that $(m, v)\in {\mathbb {Z}}$ for every $v\in I^{\perp }\cap L^{\vee }$ , and so $m\in (I^{\perp }/I)^{\vee \vee }=I^{\perp }/I$ .

We obtain a first example of regular cusps.

Lemma 4.2 If ${\Gamma }\supset {\widetilde {\mathrm {O}}^{+}(L)}$ and $\mathrm {div}(I)=1$ , then I is a regular cusp for ${\Gamma }$ .

Proof We take a generator l of I. Since $\mathrm {div}(I)=1$ , we can take an isotropic vector $l'\in L$ with $(l, l')=1$ . We can and do identify $I^{\perp }/I$ with $\langle l, l' \rangle ^{\perp }\cap L$ . We have the splitting $L=\langle l, l' \rangle \oplus (\langle l, l' \rangle ^{\perp }\cap L)$ . Suppose $-E_{m\otimes l}\in {\Gamma }$ for a vector $m\in \langle l, l' \rangle ^{\perp }\cap L_{{\mathbb {Q}}}$ . Then $E_{m\otimes l}$ preserves L, so we find that the vector $E_{m\otimes l}(l') = l' + m - \frac {1}{2}(m, m)l$ is contained in L. This implies that $m\in \langle l, l' \rangle ^{\perp }\cap L=I^{\perp }/I$ . Since ${\Gamma }\supset {\widetilde {\mathrm {O}}^{+}(L)}$ , we have ${U(I)_{\mathbb {Z}}}\supset L(I)$ by Lemma 4.1, and so $m\otimes l \in {U(I)_{\mathbb {Z}}}$ . This means that $E_{m\otimes l}\in {\Gamma }$ , and then $-{\mathrm {id}}\in {\Gamma }$ .

For ${\Gamma }={\widetilde {\mathrm {O}}^{+}(L)}$ , we have the following constraints for existence of irregular cusp.

Lemma 4.3 If I is an irregular zero-dimensional cusp for ${\Gamma }={\widetilde {\mathrm {O}}^{+}(L)}$ , then $\mathrm {div}(I)=2$ , $A_{L(I)}$ is 2-elementary, and ${U(I)_{\mathbb {Z}}^{\prime }}/{U(I)_{\mathbb {Z}}}\simeq {\mathbb {Z}}/2$ is a subgroup of $A_{L(I)}$ .

Proof Let $I={\mathbb {Z}}l$ and assume that $-E_{m\otimes l}\in {\widetilde {\mathrm {O}}^{+}(L)}$ for a vector m of $I^{\perp }_{{\mathbb {Q}}}/I_{{\mathbb {Q}}}$ . Then, for any $v\in I^{\perp }\cap L^{\vee }$ , the vector $-E_{m\otimes l}(v)=-v+(m, v)l$ must be contained in $v+L$ . This implies that

(4.1) $$ \begin{align} 2v \in (m, v)l + L. \end{align} $$

If we substitute $v=l/\mathrm {div}(I)$ , we find that $(2/\mathrm {div}(I))l \in L$ , and so $\mathrm {div}(I)=1$ or $2$ . The case $\mathrm {div}(I)=1$ is excluded by Lemma 4.2. Thus, $\mathrm {div}(I)=2$ .

If $[v]\in (I^{\perp }/I)^{\vee }$ denotes the image of $v\in I^{\perp }\cap L^{\vee }$ , then (4.1) means that $2[v]\in I^{\perp }/I$ . This shows that $A_{I^{\perp }/I}$ is 2-elementary. Finally, (4.1) implies that $(m, v)\in {\mathbb {Z}}$ for every $v\in I^{\perp }\cap L$ , and so $m\in (I^{\perp }/I)^{\vee }$ .

This determines the structure of $A_{L}$ when ${\widetilde {\mathrm {O}}^{+}(L)}$ has an irregular zero-dimensional cusp.

Proposition 4.4 If ${\widetilde {\mathrm {O}}^{+}(L)}$ has an irregular cusp, then $A_{L}\simeq {\mathbb {Z}}/8 \oplus ({\mathbb {Z}}/2)^{\oplus a}$ or $A_{L}\simeq ({\mathbb {Z}}/4)^{\oplus 2} \oplus ({\mathbb {Z}}/2)^{\oplus a}$ as abelian groups.

Proof Let $I={\mathbb {Z}}l$ be as in Lemma 4.3. Let $x=[l/2]\in A_{L}$ . Since $A_{L(I)}\simeq x^{\perp }/x$ is 2-elementary and both $\langle x \rangle $ and $A_{L}/x^{\perp }$ are isomorphic to ${\mathbb {Z}}/2$ , we see that $A_{L}$ must be isomorphic to either $({\mathbb {Z}}/2)^{\oplus a}$ or ${\mathbb {Z}}/4 \oplus ({\mathbb {Z}}/2)^{\oplus a}$ or ${\mathbb {Z}}/8 \oplus ({\mathbb {Z}}/2)^{\oplus a}$ or $({\mathbb {Z}}/4)^{\oplus 2} \oplus ({\mathbb {Z}}/2)^{\oplus a}$ as an abelian group. The first case $A_{L}\simeq ({\mathbb {Z}}/2)^{\oplus a}$ cannot occur because then $-{\mathrm {id}}\in {\widetilde {\mathrm {O}}^{+}(L)}$ . Let us show that the second case does not occur.

Suppose to the contrary that $A_{L}\simeq {\mathbb {Z}}/4 \oplus ({\mathbb {Z}}/2)^{\oplus a}$ as an abelian group. Then we have an orthogonal decomposition $A_{L}=A_0 \oplus A_1$ where $A_0\simeq {\mathbb {Z}}/4$ is generated by an element $x_0$ of norm $\varepsilon /4$ for some $\varepsilon \in ({\mathbb {Z}}/8)^{\times }$ , and $A_1=A_{0}^{\perp }$ is 2-elementary. The isotropic element $x\in A_{L}$ is either (i) contained in $A_1$ or (ii) of the from $2x_{0}+x_1$ with $x_1\ne 0 \in A_1$ . In the case (i), $x^{\perp }/x \supset A_0$ is not 2-elementary. In the case (ii), we can take an element $y_1\in A_1$ with $(x_1, y_1)=1/2$ by the nondegeneracy of $A_1$ . Then the element $y=x_0+y_1$ is contained in $x^{\perp }$ and $2y\ne x$ . Hence, $x^{\perp }/x$ is not 2-elementary again.

Remark 4.5 Further calculation shows that $x=[l/2]\in A_{L}$ is divisible by $4$ (and hence unique) in the case $A_{L}\simeq {\mathbb {Z}}/8 \oplus ({\mathbb {Z}}/2)^{\oplus a}$ , and divisible by $2$ in the case $A_{L}\simeq ({\mathbb {Z}}/4)^{\oplus 2} \oplus ({\mathbb {Z}}/2)^{\oplus a}$ .

Example 4.6 Let $L=2U \oplus mE_{8} \oplus \langle -2d \rangle $ . Then ${\widetilde {\mathrm {O}}^{+}(L)}$ has no irregular cusp when $d\ne 4$ . We show in Proposition 4.14 that ${\widetilde {\mathrm {O}}^{+}(L)}$ indeed has an irregular cusp when $d=4$ . When $m=2$ , ${\widetilde {\mathrm {O}}^{+}(L)}$ is the modular group for the moduli space of polarized $K3$ surfaces of degree $2d$ .

Example 4.7 Let $L=2U \oplus mE_{8} \oplus \langle -2t \rangle \oplus \langle -2d \rangle $ . Then ${\widetilde {\mathrm {O}}^{+}(L)}$ has no irregular cusp when $(t, d) \ne (4, 1), (2, 2), (1, 4)$ . We show in Section 4.5 that ${\widetilde {\mathrm {O}}^{+}(L)}$ indeed has an irregular cusp in these exceptional cases. When $m=2$ , ${\widetilde {\mathrm {O}}^{+}(L)}$ is the modular group for the moduli space of polarized irreducible symplectic manifolds of $K3^{[t-1]}$ -type with polarization of split type and degree $2d$ [10].

Example 4.8 When $L=U\oplus 2E_{8} \oplus M$ , where M is a certain lattice of signature $(1, 2)$ and discriminant $d\equiv 2$ mod $6$ , ${\widetilde {\mathrm {O}}^{+}(L)}$ is the modular group for the moduli space of special cubic fourfolds of discriminant d [18]. Since $A_{L}$ has length $\leq 3$ and order d, we find that ${\widetilde {\mathrm {O}}^{+}(L)}$ has no irregular cusp when $d \ne 8, 32$ .

Example 4.9 Similarly, when $L=U\oplus 2E_{8} \oplus M$ , where M is a certain lattice of signature $(1, 2)$ and discriminant $d\equiv 0, 2, 4$ mod $8$ , ${\widetilde {\mathrm {O}}^{+}(L)}$ is the modular group for the moduli space of special $K3^{[2]}$ -fourfolds of degree $2$ and discriminant d [16]. This group has no irregular cusp when $d\ne 32$ .

Example 4.10 When $L=U\oplus 2E_{8}\oplus \langle 2d \rangle $ , ${\widetilde {\mathrm {O}}^{+}(L)}$ is the modular group for the moduli space of $U\oplus \langle -2d \rangle $ -polarized $K3$ surfaces studied in [4]. This group has no irregular cusp when $d\ne 4$ .

4.2 O’Grady 10

In this subsection, we let L be an even lattice of the form $L= M \oplus \langle -2d \rangle $ with M of signature $(2, b-1)$ . We consider the group

$$ \begin{align*} {\Gamma} = \{ \: \gamma \in {\mathrm{O}^{+}(L)} \: | \: \gamma|_{A_{M}}=\pm{\mathrm{id}}, \: \gamma|_{A_{\langle -2d \rangle}}={\mathrm{id}} \; \}. \end{align*} $$

Then ${\Gamma }$ contains ${\widetilde {\mathrm {O}}^{+}(L)}$ with index $\leq 2$ , with ${\Gamma }={\widetilde {\mathrm {O}}^{+}(L)}$ if and only if $A_{M}$ is 2-elementary. We have $-{\mathrm {id}}\in {\Gamma }$ if and only if $d=1$ . When $M=2U\oplus 2E_{8} \oplus A_{2}$ , ${\Gamma }$ is the modular group for the moduli space of polarized O’Grady $10$ manifolds with polarization of split type and degree $2d$ [6].

Proposition 4.11 The group ${\Gamma }$ has no irregular cusp when $d\ne 2, 4$ .

Proof Assume that $I={\mathbb {Z}}l$ is an irregular cusp for ${\Gamma }$ and $-E_{m\otimes l}\in {\Gamma }$ for $m\in I^{\perp }_{{\mathbb {Q}}}/I_{{\mathbb {Q}}}$ . The case $d=1$ is excluded by $-{\mathrm {id}}\not \in {\Gamma }$ . We shall show that $d\mid 4$ . Since $E_{2m\otimes l}=(-E_{m\otimes l})\circ (-E_{m\otimes l}) \in {\widetilde {\mathrm {O}}^{+}(L)}$ , we see that $2m\otimes l \in L(I)$ by Lemma 4.1. Hence, we can take a lift $\tilde {m}$ of m from $I^{\perp }\cap \frac {1}{2}L$ . Let v be a generator of $\langle -2d \rangle ^{\vee }$ . The vector

$$ \begin{align*} -E_{m\otimes l}(v) = -v + (\tilde{m}, v)l - (l, v)\tilde{m} + \frac{1}{2}(m, m)(l, v)l \end{align*} $$

must be contained in $v+L$ , and hence

(4.2) $$ \begin{align} 2v \in (\tilde{m}, v)l - (l, v)\tilde{m} + \frac{1}{2}(m, m)(l, v)l + L. \end{align} $$

Since $(\tilde {m}, v)\in \frac {1}{2}{\mathbb {Z}}$ , $(l, v)\in {\mathbb {Z}}$ and $(m, m)\in \frac {1}{2}{\mathbb {Z}}$ , we find that $2v\in \frac {1}{4}L$ . Hence, $2d\mid 8$ .

The case $d=2$ does not occur when $|A_{M}|$ is square-free, because then $A_{L}$ is anisotropic and hence $\mathrm {div}(I)=1$ . In Proposition 4.14, we show that ${\Gamma }$ indeed has an irregular cusp when $d=4$ and M contains U.

4.3 Generalized Kummer

In this subsection, we let $L=M\oplus \langle -2d \rangle $ be as in Section 4.2 and consider the group

$$ \begin{align*} {\Gamma} = \{ \: \gamma \in {\mathrm{O}^{+}(L)} \: | \: \gamma|_{A_{M}}=\det (\gamma) {\mathrm{id}}, \: \gamma|_{A_{\langle -2d \rangle}}={\mathrm{id}} \; \}. \end{align*} $$

This is an index $\leq 2$ subgroup of the group considered in Section 4.2. When $M=2U \oplus \langle -2t \rangle $ with $t\geq 3$ , ${\Gamma }$ is the modular group for the moduli space of polarized deformation generalized Kummer varieties of $A^{[t]}$ -type with polarization of split type and degree $2d$ [2].

Proposition 4.12 The group ${\Gamma }$ has no irregular cusp when $d\nmid 4$ . Moreover, when b is even, ${\Gamma }$ has no irregular cusp unless $A_{L}$ is isomorphic to ${\mathbb {Z}}/8 \oplus ({\mathbb {Z}}/2)^{\oplus a}$ or $({\mathbb {Z}}/4)^{\oplus 2} \oplus ({\mathbb {Z}}/2)^{\oplus a}$ as abelian groups.

Proof The assertion $d\nmid 4$ follows from Corollary 3.5 and Proposition 4.11. Since ${\Gamma }\cap \mathrm {SO}^{+}(L) = {\widetilde {\mathrm {O}}^{+}(L)}\cap \mathrm {SO}^{+}(L)$ , Corollary 3.4 shows that when b is even, ${\Gamma }$ has an irregular cusp if and only if ${\widetilde {\mathrm {O}}^{+}(L)}$ has an irregular cusp. Then our assertion follows from Proposition 4.4.

This shows that when $M=2U \oplus \langle -2t \rangle $ , ${\Gamma }$ has no irregular cusp when $(t, d)\ne (4, 1), (2, 2), (1, 4)$ . In Section 4.5, we show that ${\Gamma }$ indeed has an irregular cusp in these exceptional cases.

4.4 Special cubic fourfolds

In this subsection, we let L be an even lattice of the form $L=M\oplus K$ with $|A_{K}|>1$ odd. (K may be either negative-definite or hyperbolic or of signature $(2, \ast )$ .) We consider the group

$$ \begin{align*} {\Gamma} = \{ \: \gamma \in {\mathrm{O}^{+}(L)} \: | \: \gamma|_{A_{M}}= \pm{\mathrm{id}}, \: \gamma|_{A_{K}}={\mathrm{id}} \; \}. \end{align*} $$

When $M=\langle 2n \rangle \oplus U \oplus 2E_{8}$ and $K=A_{2}$ , ${\Gamma }$ is the modular group for the moduli space of special cubic fourfolds of discriminant $6n$ [18].

Proposition 4.13 The group ${\Gamma }$ has no irregular cusp.

Proof Suppose to the contrary that $I={\mathbb {Z}}l$ is an irregular cusp and $-E_{m\otimes l}\in {\Gamma }$ . As in the proof of Proposition 4.11, we can take a lift of m from $I^{\perp }\cap \frac {1}{2}L$ . We take a vector $v\in K^{\vee }-K$ . Then $-E_{m\otimes l}(v)$ must be contained in $v+L$ . The same calculation as (4.2) tells us that $8v\in L$ . Therefore, $[v]\in A_{K}\subset A_{L}$ satisfies $8[v]=0$ , but this contradicts the assumption that $|A_{K}|$ is odd.

4.5 Examples of irregular cusps

In this subsection, we present two series of examples of irregular cusps, infinitely many in every dimension. We will denote by $e, f$ the standard basis of U.

As the first series of examples, we consider even lattices of the form $L=U\oplus \langle -8 \rangle \oplus M$ with M hyperbolic. We define the group ${\Gamma }$ by

$$ \begin{align*} {\Gamma} = \{ \: \gamma \in {\mathrm{O}^{+}(L)} \: | \: \gamma|_{A_{M}}=\pm{\mathrm{id}}, \: \gamma|_{A_{\langle -8 \rangle}}={\mathrm{id}} \; \}. \end{align*} $$

This is the group considered in Section 4.2 with $d=4$ . The group ${\Gamma }$ contains ${\widetilde {\mathrm {O}}^{+}(L)}$ with index $\leq 2$ , and we have ${\Gamma }={\widetilde {\mathrm {O}}^{+}(L)}$ if and only if $A_{M}$ is 2-elementary.

Proposition 4.14 The group ${\Gamma }$ has an irregular cusp.

Proof First, note that $-{\mathrm {id}}\not \in {\Gamma }$ by the condition $\gamma |_{A_{\langle -8 \rangle }}={\mathrm { id}}$ . Let v be a generator of $\langle -8 \rangle $ . We take the vectors

$$ \begin{align*} l = 2e + 2f + v, \quad m = e/2 - f/2, \end{align*} $$

and show that $-E_{m\otimes l}\in {\Gamma }$ . This amounts to checking the following:

$$ \begin{align*} E_{m\otimes l}(L) \subset L, \quad E_{m\otimes l}|_{A_{M}} = \pm{\mathrm{id}}, \quad E_{m\otimes l}(v/8) \in -v/8+L. \end{align*} $$

Since $M\perp \langle l, m \rangle $ , $E_{m\otimes l}$ acts trivially on M. By direct calculation, we see that

$$ \begin{align*} E_{m\otimes l}(e)=4e+f+v, \quad E_{m\otimes l}(f)=e, \quad E_{m\otimes l}(v)=-v-8e. \end{align*} $$

This proves our assertion.

As the second series of examples, we consider even lattices of the form $L=U\oplus \langle -4 \rangle ^{\oplus 2} \oplus M$ with M hyperbolic, and the group ${\Gamma }$ defined by

$$ \begin{align*} {\Gamma} = \{ \: \gamma \in {\mathrm{O}^{+}(L)} \: | \: \gamma|_{A_{M}}=\pm{\mathrm{id}}, \: \gamma|_{A_{\langle -4 \rangle^{\oplus 2}}}={\mathrm{id}} \; \}. \end{align*} $$

The group ${\Gamma }$ contains ${\widetilde {\mathrm {O}}^{+}(L)}$ with index $\leq 2$ , and ${\Gamma }={\widetilde {\mathrm {O}}^{+}(L)}$ if and only if $A_{M}$ is 2-elementary.

Proposition 4.15 The group ${\Gamma }$ has an irregular cusp.

Proof This is similar to the first example. We let $v_{1}, v_{2}$ be the standard basis of $\langle -4 \rangle ^{\oplus 2}$ and show that $-E_{m\otimes l}\in {\Gamma }$ for the vectors

$$ \begin{align*} l = 2e + 2f + v_{1} + v_{2}, \quad m = e + v_{1}/2. \end{align*} $$

The detail is left to the reader.

5 One-dimensional cusps

In this section, we recall, following [7, 11, 12, 17], the structure of the stabilizer of a one-dimensional cusp of ${\mathcal {D}}={\mathcal {D}_{L}}$ with its action on the Siegel domain model, and the canonical partial toroidal compactification over the cusp. This is a long preliminary for the next Section 6. Although this section is no more than expository, we need to keep the rather self-contained style of Section 2, for the same reasons as in Section 2 and for consistency.

Throughout this section, we fix a rank $2$ primitive isotropic sublattice J of L. The choice of the component ${\mathcal {D}}$ determines a connected component of ${{\mathbb P}}J_{{\mathbb {C}}} - {{\mathbb P}}J_{{\mathbb {R}}}$ , which is the cusp corresponding to J. This in turn determines an orientation of J. We abbreviate $J^{\perp }=J^{\perp }\cap L$ and write

$$ \begin{align*} L(J) = J^{\perp}/J, \end{align*} $$

which is a negative-definite lattice of rank $b-2$ . We will call an embedding $2U_{{\mathbb {Q}}}\hookrightarrow L_{{\mathbb {Q}}}$ a splitting for $J_{{\mathbb {Q}}}$ if it sends the standard two-dimensional isotropic subspace of $2U_{{\mathbb {Q}}}$ to $J_{{\mathbb {Q}}}$ . This defines a lift $L(J)_{{\mathbb {Q}}}\hookrightarrow J^{\perp }_{{\mathbb {Q}}}$ of $L(J)_{{\mathbb {Q}}}$ as $2U_{{\mathbb {Q}}}^{\perp }$ .

5.1 Siegel domain model

We consider the two-step linear projection

$$ \begin{align*} {{\mathbb P}}L_{{\mathbb{C}}} \dashrightarrow {{\mathbb P}}(L/J)_{{\mathbb{C}}} \dashrightarrow {{\mathbb P}}(L/J^{\perp})_{{\mathbb{C}}} \end{align*} $$

and restrict it to ${\mathcal {D}}\subset Q \subset {{\mathbb P}}L_{{\mathbb {C}}}$ . The center of the first projection ${{\mathbb P}}L_{{\mathbb {C}}} \dashrightarrow {{\mathbb P}}(L/J)_{{\mathbb {C}}}$ is the line ${{\mathbb P}}J_{{\mathbb {C}}}$ , and its fibers are planes containing ${{\mathbb P}}J_{{\mathbb {C}}}$ (minus ${{\mathbb P}}J_{{\mathbb {C}}}$ ). Since Q contains ${{\mathbb P}}J_{{\mathbb {C}}}$ , a plane containing ${{\mathbb P}}J_{{\mathbb {C}}}$ either

• • intersects with Q at two distinct lines (one is ${{\mathbb P}}J_{{\mathbb {C}}}$ ), or

• • intersects with Q at ${{\mathbb P}}J_{{\mathbb {C}}}$ with multiplicity $2$ , or

• • is contained in Q.

The first case occurs exactly when the plane is not contained in ${{\mathbb P}}J^{\perp }_{{\mathbb {C}}}$ . In that case, we can write the plane as ${{\mathbb P}}\langle J, v \rangle _{{\mathbb {C}}}$ with $(v, v)=0$ and $(v, J)\not \equiv 0$ . Then we have

$$ \begin{align*} {{\mathbb P}}\langle J, v \rangle_{{\mathbb{C}}} \cap Q = {{\mathbb P}}J_{{\mathbb{C}}} \cup {{\mathbb P}}\langle l, v \rangle_{{\mathbb{C}}}\,, \end{align*} $$

where ${\mathbb {C}}l=v^{\perp }\cap J_{{\mathbb {C}}}$ . This shows that the restriction of the first projection ${{\mathbb P}}L_{{\mathbb {C}}} \dashrightarrow {{\mathbb P}}(L/J)_{{\mathbb {C}}}$ to $Q-Q\cap {{\mathbb P}}J_{{\mathbb {C}}}^{\perp }$

$$ \begin{align*} \pi_{1} : Q - Q\cap {{\mathbb P}}J^{\perp}_{{\mathbb{C}}} \to {{\mathbb P}}(L/J)_{{\mathbb{C}}} - {{\mathbb P}}L(J)_{{\mathbb{C}}} \end{align*} $$

is an affine line bundle, with the fiber over the point ${{\mathbb P}}(\langle J, v \rangle _{{\mathbb {C}}}/J_{{\mathbb {C}}})$ being the affine line ${{\mathbb P}}\langle l, v \rangle _{{\mathbb {C}}} - [l]$ . The inequality $(\omega , \bar {\omega })>0$ defines a (shifted) upper half-plane in this affine line.

We identify $(L/J^{\perp })_{{\mathbb {C}}}=J^{\vee }_{{\mathbb {C}}}$ by the pairing. The second projection

$$ \begin{align*} \pi_{2} : {{\mathbb P}}(L/J)_{{\mathbb{C}}} - {{\mathbb P}}L(J)_{{\mathbb{C}}} \to {{\mathbb P}}(L/J^{\perp})_{{\mathbb{C}}} = {{\mathbb P}}J^{\vee}_{{\mathbb{C}}} \end{align*} $$

is an affine space bundle. It is (non-canonically) isomorphic to the vector bundle $L(J)_{{\mathbb {C}}} \otimes \mathcal {O}_{{{\mathbb P}}J^{\vee }_{{\mathbb {C}}}}(1)$ where, by abuse of notation, $\mathcal {O}_{{{\mathbb P}}J^{\vee }_{{\mathbb {C}}}}(1)$ stands for the line bundle corresponding to this sheaf (the dual of the tautological line bundle). To be more specific, if we choose a lift $L(J)_{{\mathbb {C}}}\hookrightarrow J_{{\mathbb {C}}}^{\perp }$ of $L(J)_{{\mathbb {C}}}$ , this determines a splitting $(L/J)_{{\mathbb {C}}}\simeq L(J)_{{\mathbb {C}}}\oplus (L/J^{\perp })_{{\mathbb {C}}}$ where $(L/J^{\perp })_{{\mathbb {C}}}$ is mapped to $L(J)_{{\mathbb {C}}}^{\perp }/J_{{\mathbb {C}}}$ . This splitting defines an isomorphism

$$ \begin{align*} L(J)_{{\mathbb{C}}} \otimes \mathcal{O}_{{{\mathbb P}}J^{\vee}_{{\mathbb{C}}}}(1) \: \simeq \: {{\mathbb P}}(L/J)_{{\mathbb{C}}} - {{\mathbb P}}L(J)_{{\mathbb{C}}} \end{align*} $$

over ${{\mathbb P}}J_{{\mathbb {C}}}^{\vee }={{\mathbb P}}(L/J^{\perp })_{{\mathbb {C}}}$ . At the fiber over each point $[v]$ of ${{\mathbb P}}(L/J^{\perp })_{{\mathbb {C}}}$ , this isomorphism is written as

(5.1) $$ \begin{align} {\operatorname{Hom}}({\mathbb{C}}v, L(J)_{{\mathbb{C}}}) \to {{\mathbb P}}({\mathbb{C}}v \oplus L(J)_{{\mathbb{C}}}) - {{\mathbb P}}L(J)_{{\mathbb{C}}}, \end{align} $$

where to a linear map ${\mathbb {C}}v\to L(J)_{{\mathbb {C}}}$ we associate its graph.

The orientation of J determines a connected component ${\mathbb {H}_{J}}$ of ${{\mathbb P}}J^{\vee }_{{\mathbb {C}}} - {{\mathbb P}}J^{\vee }_{{\mathbb {R}}}$ . We write ${\mathcal {V}_{J}} = \pi _{2}^{-1}({\mathbb {H}_{J}})$ and ${\mathcal {D}}(J) = \pi _{1}^{-1}({\mathcal {V}_{J}})$ . By definition, ${\mathcal {D}}(J)$ consists of points ${\mathbb {C}}\omega \in Q$ such that the map $(\cdot , \omega )\colon J_{{\mathbb {R}}}\to {\mathbb {C}}$ is an orientation-preserving ${\mathbb {R}}$ -isomorphism. We thus have the enlarged two-step fibration

$$ \begin{align*} {\mathcal{D}} \subset {\mathcal{D}}(J) \stackrel{\pi_{1}}{\to} {\mathcal{V}_{J}} \stackrel{\pi_{2}}{\to} {\mathbb{H}_{J}}. \end{align*} $$

This is the Siegel domain realization of ${\mathcal {D}}$ with respect to J. Here ${\mathcal {D}}(J) \to {\mathcal {V}_{J}}$ is an affine line bundle, inside which ${\mathcal {D}}\to {\mathcal {V}_{J}}$ is a fibration of upper half-planes. Over ${\mathbb {H}_{J}}\subset {{\mathbb P}}J_{{\mathbb {C}}}^{\vee }$ , we have the Hodge line bundle in $\underline {J_{{\mathbb {C}}}}=J_{{\mathbb {C}}} \otimes \mathcal {O}_{{\mathbb {H}_{J}}}$ defined by

$$ \begin{align*} F := \mathcal{O}_{{\mathbb{H}_{J}}}(-1)^{\perp} \subset \underline{J_{{\mathbb{C}}}}, \end{align*} $$

where we view $\mathcal {O}_{{\mathbb {H}_{J}}}(-1)$ as a sub-line bundle of $J_{{\mathbb {C}}}^{\vee } \otimes \mathcal {O}_{{\mathbb {H}_{J}}}$ naturally. Then $\mathcal {O}_{{\mathbb {H}_{J}}}(1)$ is naturally isomorphic to $\underline {J_{{\mathbb {C}}}}/F$ . To summarize, we have an isomorphism

(5.2) $$ \begin{align} {\mathcal{V}_{J}} \simeq L(J)_{{\mathbb{C}}}\otimes \mathcal{O}_{{\mathbb{H}_{J}}}(1) \simeq L(J)_{{\mathbb{C}}} \otimes (\underline{J_{{\mathbb{C}}}}/F). \end{align} $$

The relation with the tube domain model is as follows. We choose a rank $1$ primitive sublattice I of J. This corresponds to a zero-dimensional cusp in the closure of the one-dimensional cusp for J. The filtration $I\subset J \subset J^{\perp } \subset L$ determines the projections ${{\mathbb P}}(L/I)_{{\mathbb {C}}}\dashrightarrow {{\mathbb P}}(L/J)_{{\mathbb {C}}}\dashrightarrow {{\mathbb P}}(L/J^{\perp })_{{\mathbb {C}}}$ . Then the composition of this with the tube domain realization ${\mathcal {D}}\subset {\mathcal {D}(I)} \hookrightarrow {{\mathbb P}}(L/I)_{{\mathbb {C}}}$ is the Siegel domain realization above.

5.2 Stabilizer over ${\mathbb {Q}}$

Let ${\Gamma (J)_{\mathbb {Q}}}$ be the subgroup of the stabilizer of $J_{{\mathbb {Q}}}$ in $\mathrm {O}^{+}(L_{{\mathbb {Q}}})$ that acts on $J_{{\mathbb {Q}}}$ with determinant $1$ . The determinant $1$ condition is not restrictive when restricting to subgroups of ${\mathrm {O}^{+}(L)}$ . We write

$$ \begin{align*} {W(J)_{\mathbb{Q}}} = {\operatorname{Ker}}({\Gamma(J)_{\mathbb{Q}}} \to \mathrm{O}(L(J)_{{\mathbb{Q}}}) \times \mathrm{SL}(J_{{\mathbb{Q}}})), \end{align*} $$

$$ \begin{align*} {U(J)_{\mathbb{Q}}} = {\operatorname{Ker}}({\Gamma(J)_{\mathbb{Q}}} \to \mathrm{GL}(J_{{\mathbb{Q}}}^{\perp})), \end{align*} $$

$$ \begin{align*} {V(J)_{\mathbb{Q}}} = {W(J)_{\mathbb{Q}}}/{U(J)_{\mathbb{Q}}}. \end{align*} $$

By definition, we have the canonical exact sequences

(5.3) $$ \begin{align} 1 \to {W(J)_{\mathbb{Q}}} \to {\Gamma(J)_{\mathbb{Q}}} \to \mathrm{O}(L(J)_{{\mathbb{Q}}}) \times \mathrm{ SL}(J_{{\mathbb{Q}}}) \to 1, \end{align} $$

(5.4) $$ \begin{align} 0 \to {U(J)_{\mathbb{Q}}} \to {W(J)_{\mathbb{Q}}} \to {V(J)_{\mathbb{Q}}} \to 0. \end{align} $$

The group ${W(J)_{\mathbb {Q}}}$ is the unipotent radical of ${\Gamma (J)_{\mathbb {Q}}}$ . If we choose a splitting $L_{{\mathbb {Q}}}\simeq 2U_{{\mathbb {Q}}}\oplus L(J)_{{\mathbb {Q}}}$ for $J_{{\mathbb {Q}}}$ , the first exact sequence (5.3) splits (non-canonically):

(5.5) $$ \begin{align} {\Gamma(J)_{\mathbb{Q}}} \: \simeq \: (\mathrm{O}(L(J)_{{\mathbb{Q}}}) \times \mathrm{SL}(J_{{\mathbb{Q}}})) \ltimes {W(J)_{\mathbb{Q}}}. \end{align} $$

Here $\mathrm {SL}(J_{{\mathbb {Q}}})$ acts on the component $2U_{{\mathbb {Q}}}\simeq J_{{\mathbb {Q}}}\oplus J_{{\mathbb {Q}}}^{\vee }$ , and $\mathrm {O}(L(J)_{{\mathbb {Q}}})$ acts on the component $L(J)_{{\mathbb {Q}}}$ .

On the other hand, the second exact sequence (5.4) never splits. Indeed, ${W(J)_{\mathbb {Q}}}$ is a Heisenberg group as follows. We have a canonical $\wedge ^{2}J$ -valued symplectic form on $L(J)\otimes J$ as the tensor product of the quadratic form on $L(J)$ and the canonical symplectic form $J\times J \to \wedge ^{2}J$ on J. This gives a Heisenberg group structure on $\wedge ^{2}J_{{\mathbb {Q}}}\times (L(J)_{{\mathbb {Q}}}\otimes J_{{\mathbb {Q}}})$ . Explicitly, we take a bijection $L(J)_{{\mathbb {Q}}}\otimes J_{{\mathbb {Q}}} \simeq L(J)_{{\mathbb {Q}}} \times L(J)_{{\mathbb {Q}}}$ by choosing a positive basis of J, and define a product on $\wedge ^{2}J_{{\mathbb {Q}}}\times L(J)_{{\mathbb {Q}}} \times L(J)_{{\mathbb {Q}}}$ by

$$ \begin{align*} (\alpha, v_{1}, v_{2})\cdot (\beta, w_{1}, w_{2}) = (\alpha+\beta+(v_{2}, w_{1}), v_{1}+w_{1}, v_{2}+w_{2}). \end{align*} $$

The center is $\wedge ^{2}J_{{\mathbb {Q}}}\times \{ 0 \} \times \{ 0 \}$ .

Lemma 5.1 ${W(J)_{\mathbb {Q}}}$ is isomorphic to the Heisenberg group for $L(J)_{{\mathbb {Q}}}\otimes J_{{\mathbb {Q}}}$ with center ${U(J)_{\mathbb {Q}}}$ , and we have the canonical isomorphisms

$$ \begin{align*} \wedge^{2}J_{{\mathbb{Q}}} \to {U(J)_{\mathbb{Q}}}, \quad l\wedge l' \mapsto E_{l\otimes l'}, \end{align*} $$

$$ \begin{align*} L(J)_{{\mathbb{Q}}}\otimes J_{{\mathbb{Q}}} \to {V(J)_{\mathbb{Q}}}, \quad m\otimes l \mapsto E_{\tilde{m}\otimes l} \; \mathrm{mod} \; {U(J)_{\mathbb{Q}}}. \end{align*} $$

Proof This should be well-known (see, e.g., [12]), but we provide a proof in the present context for the convenience of the readers. We choose a rank $1$ primitive sublattice I of J and put $\bar {J}=(J/I)\otimes I \subset L(I)$ . Note that $\bar {J}\simeq \wedge ^{2}J$ naturally. We restrict the sequence (2.3) for ${\Gamma (I)_{\mathbb {Q}}}$ to ${W(J)_{\mathbb {Q}}}\subset {\Gamma (I)_{\mathbb {Q}}}$ . It is clear that ${W(J)_{\mathbb {Q}}} \cap {U(I)_{\mathbb {Q}}} = \bar {J}_{{\mathbb {Q}}}^{\perp }\cap L(I)_{{\mathbb {Q}}}$ , which contains ${U(J)_{\mathbb {Q}}}$ with

(5.6) $$ \begin{align} {U(J)_{\mathbb{Q}}} = (\bar{J}_{{\mathbb{Q}}}^{\perp})^{\perp} = \bar{J}_{{\mathbb{Q}}} \simeq \wedge^{2}J_{{\mathbb{Q}}} \quad \subset {U(I)_{\mathbb{Q}}}. \end{align} $$

The image of ${W(J)_{\mathbb {Q}}}\to {\mathrm {O}^{+}}(L(I)_{{\mathbb {Q}}})$ is the subgroup of the stabilizer of $\bar {J}_{{\mathbb {Q}}}$ that acts trivially on $\bar {J}_{{\mathbb {Q}}}$ and $\bar {J}_{{\mathbb {Q}}}^{\perp }/\bar {J}_{{\mathbb {Q}}}$ . This consists of Eichler transvections of $L(I)_{{\mathbb {Q}}}$ with respect to $\bar {J}_{{\mathbb {Q}}}$ , hence isomorphic to $(\bar {J}_{{\mathbb {Q}}}^{\perp }/\bar {J}_{{\mathbb {Q}}})\otimes \bar {J}_{{\mathbb {Q}}} \simeq L(J)_{{\mathbb {Q}}}\otimes (J/I)_{{\mathbb {Q}}}$ . In this way, we obtain the exact sequence

(5.7) $$ \begin{align} 0 \to \bar{J}_{{\mathbb{Q}}}^{\perp}\cap L(I)_{{\mathbb{Q}}} \to {W(J)_{\mathbb{Q}}} \to L(J)_{{\mathbb{Q}}}\otimes (J/I)_{{\mathbb{Q}}} \to 0. \end{align} $$

We choose lifts $L(J)_{{\mathbb {Q}}}\hookrightarrow J_{{\mathbb {Q}}}^{\perp }$ and $(J/I)_{{\mathbb {Q}}}\hookrightarrow J_{{\mathbb {Q}}}$ . This induces a section of (5.7) which consists of the Eichler transvections $E_{w}$ of $L_{{\mathbb {Q}}}$ with $w\in L(J)_{{\mathbb {Q}}}\otimes (J/I)_{{\mathbb {Q}}}$ . Together with the splitting $\bar {J}_{{\mathbb {Q}}}^{\perp }\cap L(I)_{{\mathbb {Q}}} \simeq \bar {J}_{{\mathbb {Q}}} \oplus (L(J)_{{\mathbb {Q}}}\otimes I_{{\mathbb {Q}}})$ , we obtain a bijection

$$ \begin{align*} {W(J)_{\mathbb{Q}}} \simeq \bar{J}_{{\mathbb{Q}}}\times (L(J)_{{\mathbb{Q}}}\otimes I_{{\mathbb{Q}}}) \times (L(J)_{{\mathbb{Q}}}\otimes (J/I)_{{\mathbb{Q}}}). \end{align*} $$

This gives an isomorphism with the Heisenberg group.

Note that ${U(J)_{\mathbb {Q}}}$ is not just the center of ${W(J)_{\mathbb {Q}}}$ , but also the center of ${\Gamma (J)_{\mathbb {Q}}}$ . This is the reason we put the determinant $1$ condition in the definition of ${\Gamma (J)_{\mathbb {Q}}}$ .

The action of ${\Gamma (J)_{\mathbb {Q}}}$ on the Siegel domain model can be described through the filtration ${U(J)_{\mathbb {Q}}}\subset {W(J)_{\mathbb {Q}}}\subset {\Gamma (J)_{\mathbb {Q}}}$ . By definition ${U(J)_{\mathbb {Q}}}$ acts on ${\mathcal {V}_{J}}\subset {{\mathbb P}}(J_{{\mathbb {C}}}^{\perp })^{\vee }$ trivially and ${W(J)_{\mathbb {Q}}}$ acts on ${\mathbb {H}_{J}}\subset {{\mathbb P}}J_{{\mathbb {C}}}^{\vee }$ trivially. We let $U(J)_{{\mathbb {C}}}\subset \mathrm {O}(L_{{\mathbb {C}}})$ be the group of Eichler transvections $E_{l\otimes l'}$ with $l, l'\in J_{{\mathbb {C}}}$ . Then $U(J)_{{\mathbb {C}}}\simeq \wedge ^{2}J_{{\mathbb {C}}}$ preserves ${\mathcal {D}}(J)$ and acts on ${\mathcal {V}_{J}}$ trivially. The following descriptions should be well-known, but we provide a proof in the present setting for the convenience of the readers.

Lemma 5.2 The following holds.

(1) ${\mathcal {D}}(J)\to {\mathcal {V}_{J}}$ is a principal $U(J)_{{\mathbb {C}}}$ -bundle.

(2) The group ${V(J)_{\mathbb {Q}}}\simeq L(J)_{{\mathbb {Q}}}\otimes J_{{\mathbb {Q}}}$ acts on ${\mathcal {V}_{J}}\to {\mathbb {H}_{J}}$ as the relative translation on the vector bundle $L(J)_{{\mathbb {C}}} \otimes (\underline {J_{{\mathbb {C}}}}/F)$ via an isomorphism (5.2).

(3) We choose a splitting for $J_{{\mathbb {Q}}}$ , which induces a splitting (5.5) of ${\Gamma (J)_{\mathbb {Q}}}$ and an isomorphism (5.2) for ${\mathcal {V}_{J}}$ . Then the lifted subgroup $\mathrm {O}(L(J)_{{\mathbb {Q}}})\times \mathrm {SL}(J_{{\mathbb {Q}}})$ of ${\Gamma (J)_{\mathbb {Q}}}$ acts on ${\mathcal {V}_{J}}\to {\mathbb {H}_{J}}$ by the equivariant action of $\mathrm {SL}(J_{{\mathbb {Q}}})$ on $\underline {J_{{\mathbb {C}}}}/F$ and the linear action of $\mathrm {O}(L(J)_{{\mathbb {Q}}})$ on $L(J)_{{\mathbb {C}}}$ .

Proof (1) Recall from Section 5.1 that a fiber of ${\mathcal {D}}(J)\to {\mathcal {V}_{J}}$ is an affine line ${{\mathbb P}}\langle l, v \rangle _{{\mathbb {C}}}-[l]$ where $v\in L_{{\mathbb {C}}}$ is an isotropic vector with $(v, J)\not \equiv 0$ and ${\mathbb {C}}l=v^{\perp }\cap J_{{\mathbb {C}}}$ . We take $l'\in J_{{\mathbb {C}}}$ with $(l', v)=1$ . Then $E_{\alpha l\wedge l'}\in U(J)_{{\mathbb {C}}}$ , $\alpha \in {\mathbb {C}}$ , sends a point ${\mathbb {C}}(v+\beta l)$ of ${{\mathbb P}}\langle l, v \rangle _{{\mathbb {C}}}-[l]$ to

$$ \begin{align*} {\mathbb{C}}(v+\beta l) \mapsto {\mathbb{C}}(v+(\alpha l', v)l + \beta l) = {\mathbb{C}}(v+(\alpha+\beta)l). \end{align*} $$

This shows that $U(J)_{{\mathbb {C}}}$ acts on each fiber of ${\mathcal {D}}(J)\to {\mathcal {V}_{J}}$ freely and transitively.

(2) We choose a splitting $L_{{\mathbb {Q}}}\simeq J_{{\mathbb {Q}}}\oplus L(J)_{{\mathbb {Q}}} \oplus (L/J^{\perp })_{{\mathbb {Q}}}$ for $J_{{\mathbb {Q}}}$ where the lift of $(L/J^{\perp })_{{\mathbb {Q}}}$ is perpendicular to the lift of $L(J)_{{\mathbb {Q}}}$ . Let $[v]$ be a point of ${\mathbb {H}_{J}}\subset {{\mathbb P}}(L/J^{\perp })_{{\mathbb {C}}}$ . By (5.1), the fiber of ${\mathcal {V}_{J}}\to {\mathbb {H}_{J}}$ over $[v]$ is the affine line

$$ \begin{align*} {{\mathbb P}}({\mathbb{C}}v\oplus L(J)_{{\mathbb{C}}}) - {{\mathbb P}}L(J)_{{\mathbb{C}}} \simeq {\operatorname{Hom}}({\mathbb{C}}v, L(J)_{{\mathbb{C}}}) \end{align*} $$

in ${{\mathbb P}}(L/J)_{{\mathbb {C}}}$ . Here the point corresponding to $f\in {\operatorname {Hom}}({\mathbb {C}}v, L(J)_{{\mathbb {C}}})$ is its graph ${\mathbb {C}}(v+f(v))$ . We take $E_{m\otimes l}\in {W(J)_{\mathbb {Q}}}/{U(J)_{\mathbb {Q}}}$ where $m\in L(J)_{{\mathbb {Q}}}$ and $l\in J_{{\mathbb {Q}}}$ . Then $E_{m\otimes l}$ sends ${\mathbb {C}}(v+f(v))$ to

$$ \begin{align*} {\mathbb{C}}(v+f(v)) & \mapsto {\mathbb{C}}(v + (l, v)m - 2^{-1}(m, m)(l, v)l + f(v) - (m, f(v))l) \\ & = {\mathbb{C}}(v + f(v) + (l, v)m) \: \: \in {{\mathbb P}}(L/J)_{{\mathbb{C}}}. \end{align*} $$

This means that $E_{m\otimes l}$ acts on ${\operatorname {Hom}}({\mathbb {C}}v, L(J)_{{\mathbb {C}}})$ by the translation $f\mapsto f + (l, \cdot )m$ . Finally, we notice that the ${\mathbb {R}}$ -isomorphism

$$ \begin{align*} L(J)_{{\mathbb{R}}}\otimes_{{\mathbb{R}}}J_{{\mathbb{R}}} \to L(J)_{{\mathbb{C}}}\otimes_{{\mathbb{C}}}(J_{{\mathbb{C}}}/F_{[v]}) = L(J)_{{\mathbb{C}}}\otimes_{{\mathbb{C}}}(J_{{\mathbb{C}}}/v^{\perp}) \simeq L(J)_{{\mathbb{C}}}\otimes_{{\mathbb{C}}}({\mathbb{C}}v)^{\vee} \end{align*} $$

sends $m\otimes l$ to $m\otimes (l, \cdot )$ . This proves the assertion (2).

The proof of (3) is straightforward and is left to the readers.

5.3 Stabilizer over ${\mathbb {Z}}$

Now let ${\Gamma }$ be a finite-index subgroup of ${\mathrm {O}^{+}(L)}$ . We write

$$ \begin{align*} {\Gamma(J)_{\mathbb{Z}}} = {\Gamma(J)_{\mathbb{Q}}}\cap {\Gamma}, \quad {W(J)_{\mathbb{Z}}} = {W(J)_{\mathbb{Q}}} \cap {\Gamma}, \quad {U(J)_{\mathbb{Z}}} = {U(J)_{\mathbb{Q}}} \cap {\Gamma}, \end{align*} $$

and consider the quotients

$$ \begin{align*} {\overline{\Gamma(J)}_{\mathbb{Z}}} = {\Gamma(J)_{\mathbb{Z}}}/{U(J)_{\mathbb{Z}}}, \quad {V(J)_{\mathbb{Z}}} = {W(J)_{\mathbb{Z}}}/{U(J)_{\mathbb{Z}}}, \quad \Gamma_{J} = {\Gamma(J)_{\mathbb{Z}}}/{W(J)_{\mathbb{Z}}}, \end{align*} $$

$$ \begin{align*} {\overline{\Gamma(J)}_{\mathbb{Q}}} = {\Gamma(J)_{\mathbb{Q}}}/{U(J)_{\mathbb{Z}}}, \; \; {W(J)_{\mathbb{Q}/\mathbb{Z}}} = {W(J)_{\mathbb{Q}}}/{U(J)_{\mathbb{Z}}}, \; \; {U(J)_{\mathbb{Q}/\mathbb{Z}}} = {U(J)_{\mathbb{Q}}}/{U(J)_{\mathbb{Z}}}. \end{align*} $$

By definition, we have the canonical exact sequences

(5.8) $$ \begin{align} 0 \to {V(J)_{\mathbb{Z}}} \to {\overline{\Gamma(J)}_{\mathbb{Z}}} \to \Gamma_{J} \to 1, \end{align} $$

(5.9) $$ \begin{align} 0 \to {W(J)_{\mathbb{Q}/\mathbb{Z}}} \to {\overline{\Gamma(J)}_{\mathbb{Q}}} \to \mathrm{ O}(L(J)_{{\mathbb{Q}}})\times \mathrm{SL}(J_{{\mathbb{Q}}}) \to 1, \end{align} $$

$$ \begin{align*} 0 \to {U(J)_{\mathbb{Q}/\mathbb{Z}}} \to {W(J)_{\mathbb{Q}/\mathbb{Z}}} \to {V(J)_{\mathbb{Q}}} \to 0. \end{align*} $$

Then (5.8) is canonically embedded in (5.9). We have

(5.10) $$ \begin{align} {V(J)_{\mathbb{Z}}}\cap {U(J)_{\mathbb{Q}/\mathbb{Z}}} = \{ 0 \} \end{align} $$

as subgroups of ${W(J)_{\mathbb {Q}/\mathbb {Z}}}$ because ${W(J)_{\mathbb {Z}}} \cap {U(J)_{\mathbb {Q}}}={U(J)_{\mathbb {Z}}}$ by definition. Note that ${U(J)_{\mathbb {Q}/\mathbb {Z}}}$ is the group of torsion points of the one-dimensional torus $T(J)=U(J)_{{\mathbb {C}}}/{U(J)_{\mathbb {Z}}}$ . If we choose a splitting for $J_{{\mathbb {Q}}}$ , the induced splitting (5.5) of ${\Gamma (J)_{\mathbb {Q}}}$ defines a splitting of (5.9):

(5.11) $$ \begin{align} {\overline{\Gamma(J)}_{\mathbb{Q}}} \: \simeq \: (\mathrm{O}(L(J)_{{\mathbb{Q}}}) \times \mathrm{ SL}(J_{{\mathbb{Q}}})) \ltimes {W(J)_{\mathbb{Q}/\mathbb{Z}}}. \end{align} $$

But this does not mean that (5.8) splits.

5.4 Partial toroidal compactification

We denote ${\mathcal {T}(J)}={\mathcal {D}}(J)/{U(J)_{\mathbb {Z}}}$ and ${\mathcal {X}(J)}={\mathcal {D}}/{U(J)_{\mathbb {Z}}}$ . By Lemma 5.2, ${\mathcal {T}(J)}\to {\mathcal {V}_{J}}$ is a principal $T(J)$ -bundle acted on equivariantly by ${\overline {\Gamma (J)}_{\mathbb {Q}}}$ . The projection ${\mathcal {X}(J)\to {\mathcal {V}_{J}}}$ is a punctured disk bundle in ${\mathcal {T}(J)}\to {\mathcal {V}_{J}}$ . By Lemma 5.2, the action of ${\overline {\Gamma (J)}_{\mathbb {Q}}}$ on ${\mathcal {V}_{J}}\to {\mathbb {H}_{J}}$ is described as follows.

Lemma 5.3 We choose a splitting for $J_{{\mathbb {Q}}}$ to give an isomorphism (5.2) for ${\mathcal {V}_{J}}$ and a splitting (5.11) of ${\overline {\Gamma (J)}_{\mathbb {Q}}}$ . We express an element $\gamma $ of ${\overline {\Gamma (J)}_{\mathbb {Q}}}$ as $\gamma =(\gamma _{1}, \gamma _{2}, \alpha )$ accordingly, where $\gamma _{1}\in \mathrm {O}(L(J))$ , $\gamma _{2}\in \mathrm {SL}(J)$ , and $\alpha \in {W(J)_{\mathbb {Q}/\mathbb {Z}}}$ . Then $\gamma $ acts on ${\mathcal {V}_{J}}\simeq L(J)_{{\mathbb {C}}}\otimes (\underline {J_{{\mathbb {C}}}}/F)$ as the equivariant action by $(\gamma _{1}, \gamma _{2})$ and the translation by $[\alpha ]\in {V(J)_{\mathbb {Q}}}\simeq L(J)_{{\mathbb {Q}}}\otimes J_{{\mathbb {Q}}}$ .

Thus, ${\mathcal {V}_{J}}/{V(J)_{\mathbb {Z}}}$ is a fibration of abelian varieties over ${\mathbb {H}_{J}}$ isogenous to the self-fiber product of the universal elliptic curve. The group $\Gamma _{J}$ acts on ${\mathcal {V}_{J}}/{V(J)_{\mathbb {Z}}}$ by the equivariant action plus some possible translation.

Now let $\overline {T(J)}\simeq {\mathbb {C}}$ be the canonical partial compactification of the torus $T(J)$ . We take the relative torus embedding

$$ \begin{align*} {\overline{\mathcal{T}(J)}}= ({\mathcal{T}(J)} \times \overline{T(J)})/T(J). \end{align*} $$

This is the line bundle associated with the principal $T(J)$ -bundle ${\mathcal {T}(J)}\to {\mathcal {V}_{J}}$ and the standard character of $T(J)$ . Let ${\overline {\mathcal {X}(J)}}$ be the interior of the closure of ${\mathcal {X}(J)}$ in ${\overline {\mathcal {T}(J)}}$ . This is the partial toroidal compactification of ${\mathcal {X}(J)}$ over the one-dimensional cusp J. Note that no choice of fan is required: this is canonical. The boundary divisor of ${\overline {\mathcal {X}(J)}}$ is canonically isomorphic to ${\mathcal {V}_{J}}$ .

The relation with a partial toroidal compactification over an adjacent zero-dimensional cusp $I \subset J$ is as follows. Recall that $\bar {J}=(J/I)\otimes I \simeq \wedge ^{2}J$ is an isotropic sublattice of $L(I)$ , oriented by the orientation of J. The ray $\sigma _{J}=(\bar {J}_{{\mathbb {R}}})_{\geq 0}$ is in the closure of the positive cone, and it is contained in any ${\Gamma (I)_{\mathbb {Z}}}$ -admissible fan $\Sigma $ . The torus embedding $T(I)\hookrightarrow T(I)^{\sigma _{J}}$ defined by $\sigma _{J}$ is a Zariski open set of $T(I)^{\Sigma }$ . By (5.6), we have $U(J)_{{\mathbb {R}}}=\bar {J}_{{\mathbb {R}}}\subset U(I)_{{\mathbb {R}}}$ and

(5.12) $$ \begin{align} {U(J)_{\mathbb{Z}}} = \bar{J}_{{\mathbb{R}}} \cap {U(I)_{\mathbb{Z}}} = {\mathbb{R}}\sigma_{J} \cap {U(I)_{\mathbb{Z}}}. \end{align} $$

Therefore, the inclusion ${\mathcal {D}}(J)\subset {\mathcal {D}(I)}$ induces the etale map

(5.13) $$ \begin{align} {\overline{\mathcal{T}(J)}}\to T(I)^{\sigma_{J}}\subset T(I)^{\Sigma}, \end{align} $$

which maps the boundary divisor of ${\overline {\mathcal {T}(J)}}$ to the unique boundary divisor of $T(I)^{\sigma _{J}}$ . We note that ${U(I)_{\mathbb {Z}}}\subset {\Gamma (J)_{\mathbb {Z}}}$ .

6 Irregular one-dimensional cusps

In this section, we define and study irregular one-dimensional cusps. For simplicity, we assume $b\geq 3$ so that $L(J)\ne \{ 0 \}$ . Let ${\Gamma }$ be a finite-index subgroup of ${\mathrm {O}^{+}(L)}$ , and let J be a rank $2$ primitive isotropic sublattice of L. We keep the notation from Section 5. Irregularity of the one-dimensional cusp J can be characterized as follows.

Proposition 6.1 The following conditions are equivalent.

1. (1) ${U(J)_{\mathbb {Z}}}\ne {U(J)_{\mathbb {Z}}^{\prime }}$ where ${U(J)_{\mathbb {Z}}^{\prime }}={U(J)_{\mathbb {Q}}}\cap {\langle \Gamma , -\mathrm {id} \rangle }$ .

2. (2) $-{\mathrm {id}}\not \in {\Gamma }$ and $-E_{w}\in {\Gamma (J)_{\mathbb {Z}}}$ for some $w\in \wedge ^{2}J_{{\mathbb {Q}}}$ .

3. (3) $-{\mathrm {id}}\not \in {\Gamma }$ and ${\overline {\Gamma (J)}_{\mathbb {Z}}}$ contains an element $\gamma $ of finite order whose image in $\mathrm {O}(L(J))\times \mathrm {SL}(J)$ is $(-{\mathrm {id}}_{L(J)}, -{\mathrm {id}}_{J})$ .

4. (4) ${\overline {\Gamma (J)}_{\mathbb {Z}}}$ contains an element $\gamma $ which acts trivially on ${\mathcal {V}_{J}}$ but nontrivially on ${\mathcal {X}(J)}$ .

When these hold, the element $\gamma $ of ${\overline {\Gamma (J)}_{\mathbb {Z}}}$ in (3), (4) is given by $-E_{w}$ in (2), has order $2$ , and is unique.

Definition 6.1 We say that the one-dimensional cusp J is irregular when these properties hold, and regular otherwise.

Proof The equivalence $(1) \Leftrightarrow (2)$ is similar to $(1) \Leftrightarrow (2)$ in Proposition 3.1. The quotient ${U(J)_{\mathbb {Z}}^{\prime }}/{U(J)_{\mathbb {Z}}}\simeq {\mathbb {Z}}/2$ is generated by $E_{w}$ in (2).

$(2) \Rightarrow (4)$ : Since $E_{w}$ for $w\in \wedge ^{2}J_{{\mathbb {Q}}}$ acts trivially on ${\mathcal {V}_{J}}$ by Lemma 5.2, so does $-E_{w}$ .

$(2) \Rightarrow (3)$ : The element $\gamma =[-E_{w}]$ of ${\overline {\Gamma (J)}_{\mathbb {Z}}}$ is of order $2$ and acts on J, $L(J)$ by $-1$ .

$(3) \Rightarrow (4)$ : By the description of the ${\overline {\Gamma (J)}_{\mathbb {Z}}}$ -action on ${\mathcal {V}_{J}}$ in Lemma 5.3, we find that the element $\gamma $ of (3) acts on ${\mathcal {V}_{J}}$ by some translation. Since $\gamma $ is of finite order by assumption, this translation must be trivial.

$(4) \Rightarrow (2), (3)$ : Suppose that $\gamma \in {\overline {\Gamma (J)}_{\mathbb {Z}}}$ acts trivially on ${\mathcal {V}_{J}}$ but nontrivially on ${\mathcal {X}(J)}$ . We take a splitting (5.11) of ${\overline {\Gamma (J)}_{\mathbb {Q}}}$ and express $\gamma = (\gamma _{1}, \gamma _{2}, \alpha )$ accordingly. Since $\gamma $ acts on $\mathbb {H}_{J}$ trivially, we must have $\gamma _{2}={\mathrm {id}}_{J}$ or $-{\mathrm {id}}_{J}$ . Then, since $\gamma $ acts on ${\mathcal {V}_{J}}\simeq L(J)_{{\mathbb {C}}}\otimes (\underline {J_{{\mathbb {C}}}}/F)$ trivially, we see from Lemma 5.3 that $(\gamma _{1}, \gamma _{2}) = ({\mathrm {id}}_{L(J)}, {\mathrm {id}}_{J})$ or $(-{\mathrm {id}}_{L(J)}, -{\mathrm { id}}_{J})$ , and the image of $\alpha \in {W(J)_{\mathbb {Q}/\mathbb {Z}}}$ in ${V(J)_{\mathbb {Q}}}$ must be $0$ , namely $\alpha \in {U(J)_{\mathbb {Q}/\mathbb {Z}}}$ . The case $(\gamma _{1}, \gamma _{2}) = ({\mathrm {id}}_{L(J)}, {\mathrm {id}}_{J})$ cannot occur, because then $\gamma \in {U(J)_{\mathbb {Q}/\mathbb {Z}}}\cap {V(J)_{\mathbb {Z}}}$ and so $\gamma =\mathrm {id}$ by (5.10). Therefore, $\gamma =(-{\mathrm {id}}_{L(J)}, -{\mathrm {id}}_{J}, E_{w})$ for some $w\in \wedge ^{2}J_{{\mathbb {Q}}}$ . Since $-{\mathrm {id}}_{L}=(-{\mathrm {id}}_{L(J)}, -{\mathrm {id}}_{J}, 0)$ with respect to this (and any) splitting, we find that $\gamma =-E_{w}$ . Thus, $-E_{w}\in {\Gamma }$ . Finally, we have $-{\mathrm {id}}\not \in {\Gamma }$ , for otherwise $E_{w}=-\gamma $ would be contained in ${U(J)_{\mathbb {Z}}}$ , which in turn implies that $\gamma $ acts trivially on ${\mathcal {X}(J)}$ .

As in the case of zero-dimensional cusps, $U(J)_{{\mathbb {Z}}}'$ is the projection image of $U(J)_{{\mathbb {Z}}}^{\star }=(\{ \pm {\mathrm {id}} \} \cdot {U(J)_{\mathbb {Q}}})\cap {\Gamma }$ in ${U(J)_{\mathbb {Q}}}$ , and we have $U(J)_{{\mathbb {Z}}}^{\star }/{U(J)_{\mathbb {Z}}}=\langle -E_{w} \rangle $ when J is irregular.

Since the boundary divisor of ${\overline {\mathcal {X}(J)}}$ is naturally isomorphic to ${\mathcal {V}_{J}}$ , the condition $(4)$ can be restated as follows.

Corollary 6.2 A one-dimensional cusp J is irregular if and only if ${\overline {\mathcal {X}(J)}}\to {\overline {\mathcal {X}(J)}}/{\overline {\Gamma (J)}_{\mathbb {Z}}}$ is ramified along the boundary divisor of ${\overline {\mathcal {X}(J)}}$ . In that case, the ramification index is $2$ , and the unique nontrivial element of ${\overline {\Gamma (J)}_{\mathbb {Z}}}$ fixing the boundary divisor is given by $-E_{w}$ .

By the condition (1), irregularity of a one-dimensional cusp reduces to that of an adjacent zero-dimensional cusp as follows.

Proposition 6.3 Let $I\subset J$ be a rank $1$ primitive sublattice, and let $\sigma _{J}\subset U(I)_{{\mathbb {R}}}$ be the isotropic ray corresponding to J. Then J is irregular if and only if I is irregular and $\sigma _{J}$ is an irregular ray.

Proof Recall from Definition 3.2 that the ray $\sigma _{J}$ is called irregular when ${\mathbb {R}}\sigma _{J}\cap {U(I)_{\mathbb {Z}}} \ne {\mathbb {R}}\sigma _{J} \cap {U(I)_{\mathbb {Z}}^{\prime }}$ . By (5.12), we have ${\mathbb {R}}\sigma _{J}\cap {U(I)_{\mathbb {Z}}} = {U(J)_{\mathbb {Z}}}$ , and similarly ${\mathbb {R}}\sigma _{J}\cap {U(I)_{\mathbb {Z}}^{\prime }} = {U(J)_{\mathbb {Z}}^{\prime }}$ . This proves our assertion.

Corollary 6.4 If ${\Gamma }$ has no irregular zero-dimensional cusp, it has no irregular one-dimensional cusp.

7 Toroidal compactification

In this section, we study singularities and ramification divisors in the boundary of a toroidal compactification of the modular variety. These are studied in [7, 13] under the condition $-{\mathrm {id}}\in {\Gamma }$ , and we explain what modification is necessary in the general case, especially at the irregular cusps.

Let L be a lattice of signature $(2, b)$ , and let ${\Gamma }$ be a subgroup of ${\mathrm {O}^{+}(L)}$ of finite index. The input data for constructing a toroidal compactification of ${\mathcal {F}(\Gamma )}={\Gamma }\backslash {\mathcal {D}}$ are a collection $\Sigma =(\Sigma _{I})_{I}$ of ${\Gamma (I)_{\mathbb {Z}}}$ -admissible fans (Section 2.4), one for each ${\Gamma }$ -equivalence class of rank $1$ primitive isotropic sublattices I of L. No choice is required for one-dimensional cusps. Thus, $\Sigma $ is a finite collection of independent fans.

The toroidal compactification associated with $\Sigma $ is defined as ([1, p. 163])

$$ \begin{align*} {\mathcal{F}(\Gamma)^{\Sigma}} = \left( {\mathcal{D}} \sqcup \bigsqcup_{I} \mathcal{X}(I)^{\Sigma_{I}} \sqcup \bigsqcup_{J}{\overline{\mathcal{X}(J)}} \right) / \sim, \end{align*} $$

where I (resp. J) ranges over all primitive isotropic sublattices of L of rank $1$ (resp. $2$ ), and $\sim $ is the equivalence relation generated by the following:

• • action of $\gamma \in {\Gamma }$ giving ${\mathcal {D}}\to {\mathcal {D}}$ , $\mathcal {X}(I)^{\Sigma _{I}}\to \mathcal {X}(\gamma I)^{\Sigma _{\gamma I}}$ , and ${\overline {\mathcal {X}(J)}} \to \overline {\mathcal {X}(\gamma J)}$ ;

• • the natural maps ${\mathcal {D}}\to \mathcal {X}(I)^{\Sigma _{I}}$ and ${\mathcal {D}}\to {\overline {\mathcal {X}(J)}}$ ;

• • the etale gluing maps ${\overline {\mathcal {X}(J)}}\to \mathcal {X}(I)^{\Sigma _{I}}$ for $I\subset J$ given by (5.13).

Theorem 7.1 [1]

The space ${\mathcal {F}(\Gamma )^{\Sigma }}$ is a compact Moishezon space containing ${\mathcal {F}(\Gamma )}$ as a Zariski open set, and we have a morphism from ${\mathcal {F}(\Gamma )^{\Sigma }}$ to the Baily–Borel compactification of ${\mathcal {F}(\Gamma )}$ . For each cusp I, J, the natural map

$$ \begin{align*} \mathcal{X}(I)^{\Sigma_{I}}/{\overline{\Gamma(I)}_{\mathbb{Z}}} \to {\mathcal{F}(\Gamma)^{\Sigma}}, \qquad {\overline{\mathcal{X}(J)}}/{\overline{\Gamma(J)}_{\mathbb{Z}}} \to {\mathcal{F}(\Gamma)^{\Sigma}} \end{align*} $$

is locally isomorphic in an open neighborhood of boundary points lying over that cusp.

Perhaps a word might be in order because, strictly speaking, the theory of [1] is applied to the image of ${\Gamma }$ in $\mathrm { O}^{+}(L_{{\mathbb {R}}})/\pm {\mathrm {id}}$ , which is ${\langle \Gamma , -\mathrm {id} \rangle }/\pm {\mathrm {id}}$ , rather than ${\Gamma }$ itself. Then ${U(I)_{\mathbb {Z}}}$ should be replaced by ${U(I)_{\mathbb {Z}}^{\prime }}$ , ${\mathcal {X}(I)}={\mathcal {D}}/{U(I)_{\mathbb {Z}}}$ by ${\mathcal {X}(I)'}={\mathcal {D}}/{U(I)_{\mathbb {Z}}^{\prime }}$ , ${\Gamma (I)_{\mathbb {Z}}}$ by $\Gamma '(I)_{{\mathbb {Z}}}=\langle {\Gamma (I)_{\mathbb {Z}}}, -{\mathrm {id}} \rangle / \pm {\mathrm {id}}$ , and similarly for one-dimensional cusps J. But since ${\mathcal {X}(I)'}={\mathcal {X}(I)}$ or ${\mathcal {X}(I)'}={\mathcal {X}(I)}/\langle -E_{w} \rangle $ with $-E_{w}\in {\Gamma (I)_{\mathbb {Z}}}$ (and similarly for J), we have naturally

$$ \begin{align*} \left( {\mathcal{D}} \sqcup \bigsqcup_{I} \mathcal{X}(I)^{\Sigma_{I}} \sqcup \bigsqcup_{J}{\overline{\mathcal{X}(J)}} \right) / \sim \; \; = \; \; \left( {\mathcal{D}} \sqcup \bigsqcup_{I} ({\mathcal{X}(I)'})^{\Sigma_{I}} \sqcup \bigsqcup_{J}\overline{\mathcal{X}(J)'} \right) / \sim', \end{align*} $$

where $\sim '$ is the equivalence relation similar to $\sim $ . The last statement of Theorem 7.1 [1, p. 175] is justified because we have

$$ \begin{align*} \mathcal{X}(I)^{\Sigma_{I}}/{\overline{\Gamma(I)}_{\mathbb{Z}}} = ({\mathcal{X}(I)'})^{\Sigma_{I}} / (\Gamma'(I)_{{\mathbb{Z}}}/{U(I)_{\mathbb{Z}}^{\prime}}) \end{align*} $$

(see also (7.1)), and similarly for J.

The reason we prefer to work with ${U(I)_{\mathbb {Z}}}$ rather than ${U(I)_{\mathbb {Z}}^{\prime }}$ is that Fourier expansion of ${\Gamma }$ -modular forms of arbitrary weight can be done with ${U(I)_{\mathbb {Z}}}$ (see Section 8).

If $D(\sigma )\subset \mathcal {X}(I)^{\Sigma _{I}}$ is the boundary divisor corresponding to a ray $\sigma \in \Sigma _{I}$ , general points of $D(\sigma )$ lie over the I-cusp if and only if $\sigma $ is positive-definite. When $\sigma =\sigma _{J}$ is isotropic corresponding to a one-dimensional cusp $J\supset I$ , $D(\sigma _{J})$ is glued with the boundary divisor of ${\overline {\mathcal {X}(J)}}$ , and its general points lie over the J-cusp. By combining the last statement of Theorem 7.1 with Corollaries 3.8 and 6.2, we obtain the following.

Proposition 7.2 (1) The projection $\mathcal {X}(I)^{\Sigma _{I}}\to {\mathcal {F}(\Gamma )^{\Sigma }}$ is ramified along irregular boundary divisors of $\mathcal {X}(I)^{\Sigma _{I}}$ with ramification index $2$ , and not ramified along other boundary divisors. If we take quotient by $U(I)_{{\mathbb {Z}}}^{\star }/{U(I)_{\mathbb {Z}}}$ , then $({\mathcal {D}}/U(I)_{{\mathbb {Z}}}^{\star })^{\Sigma _{I}}\to {\mathcal {F}(\Gamma )^{\Sigma }}$ is not ramified along the boundary divisors.

(2) The projection ${\overline {\mathcal {X}(J)}}\to {\mathcal {F}(\Gamma )^{\Sigma }}$ is ramified along the unique boundary divisor (with index $2$ ) if and only if J is irregular. If we take quotient by $U(J)_{{\mathbb {Z}}}^{\star }/{U(J)_{\mathbb {Z}}}$ , then $\overline {{\mathcal {D}}/U(J)_{{\mathbb {Z}}}^{\star }}\to {\mathcal {F}(\Gamma )^{\Sigma }}$ is not ramified along the boundary divisor.

Proof What remains is to show that (1) is still true even when a ray $\sigma =\sigma _{J}$ is isotropic. Since the map ${\overline {\mathcal {X}(J)}}\to {\mathcal {F}(\Gamma )^{\Sigma }}$ in (2) factorizes as ${\overline {\mathcal {X}(J)}}\to \mathcal {X}(I)^{\Sigma _{I}} \to {\mathcal {F}(\Gamma )^{\Sigma }}$ and the gluing map ${\overline {\mathcal {X}(J)}}\to \mathcal {X}(I)^{\Sigma _{I}}$ is etale, our assertion for $\mathcal {X}(I)^{\Sigma _{I}} \to {\mathcal {F}(\Gamma )^{\Sigma }}$ follows from (2) and Proposition 6.3.

When ${\Gamma }$ contains $-{\mathrm {id}}$ , Proposition 7.2 is proved in [7, 13]. In that case, we have no irregular cusp, so no ramification divisor in the boundary.

Remark 7.3 It appears that in some literatures, the “no ramification boundary divisor” property is used to claim that $\mathcal {F}(\Gamma ')^{\Sigma }\to {\mathcal {F}(\Gamma )^{\Sigma }}$ is not ramified along the boundary divisors for neat subgroups $\Gamma ' < {\Gamma }$ . This seems not true already in the case of modular curves: for example, $\Gamma (N) < \mathrm {SL}_{2}({\mathbb {Z}})$ . The point is that $U(I)_{{\mathbb {Z}},{\Gamma }}={U(I)_{\mathbb {Q}}}\cap {\Gamma }$ depends on ${\Gamma }$ , so $U(I)_{{\mathbb {Z}},{\Gamma }'}={U(I)_{\mathbb {Q}}}\cap {\Gamma }'$ is in general smaller than $U(I)_{{\mathbb {Z}},{\Gamma }}$ . If $\sigma $ is a ray in $\Sigma _{I}$ , assumed regular for simplicity, we have ramification index

$$ \begin{align*} [{\mathbb{R}}\sigma\cap U(I)_{{\mathbb{Z}},{\Gamma}} : {\mathbb{R}}\sigma\cap U(I)_{{\mathbb{Z}},{\Gamma}'}] \end{align*} $$

at the corresponding boundary divisor. It seems that so far, all argument using the above claim can be avoided: see the proof of Theorem 8.9.

Next, we study singularities. A fan $\Sigma _{I}=(\sigma _{\alpha })$ is called basic with respect to a lattice $\Lambda \subset {U(I)_{\mathbb {Q}}}$ if each cone $\sigma _{\alpha }$ is generated by a part of a basis of $\Lambda $ . The singularity theorem [7, 13] is still true, if we require the fan $\Sigma _{I}$ to be basic with respect to ${U(I)_{\mathbb {Z}}^{\prime }}$ , rather than ${U(I)_{\mathbb {Z}}}$ .

Proposition 7.4 (cf. [7, 13])

(1) We choose the fans $\Sigma =(\Sigma _{I})$ so that each $\Sigma _{I}$ is basic with respect to ${U(I)_{\mathbb {Z}}^{\prime }}$ . Then ${\mathcal {F}(\Gamma )^{\Sigma }}$ has canonical singularities at the boundary points lying over the zero-dimensional cusps.

(2) When $b\geq 9$ , ${\mathcal {F}(\Gamma )^{\Sigma }}$ has canonical singularities at the boundary points lying over the one-dimensional cusps.

Proof When ${\Gamma }$ contains $-{\mathrm {id}}$ , this is proved in [7, 13] for zero-dimensional cusps, and in [7] for one-dimensional cusps. We show that the general case is reduced to this case. We consider zero-dimensional cusps. The case of one-dimensional cusps is similar. It suffices to show that ${\mathcal {X}(I)}^{\Sigma _{I}}/{\overline {\Gamma (I)}_{\mathbb {Z}}}$ has canonical singularities.

Let $\Gamma ' = {\langle \Gamma , -\mathrm {id} \rangle }$ and $\Gamma '(I)_{{\mathbb {Z}}}=\Gamma ' \cap {\Gamma (I)_{\mathbb {Q}}}$ . Then ${U(I)_{\mathbb {Z}}^{\prime }}={U(I)_{\mathbb {Q}}}\cap \Gamma '$ and $\Gamma '(I)_{{\mathbb {Z}}} = \langle {\Gamma (I)_{\mathbb {Z}}}, -{\mathrm {id}} \rangle $ . Since the fan $\Sigma _{I}$ is also rational with respect to ${U(I)_{\mathbb {Z}}^{\prime }}$ , it defines a toroidal embedding $({\mathcal {D}}/{U(I)_{\mathbb {Z}}^{\prime }})^{\Sigma _{I}}$ of ${\mathcal {D}}/{U(I)_{\mathbb {Z}}^{\prime }}$ . This is the quotient of $({\mathcal {D}}/{U(I)_{\mathbb {Z}}})^{\Sigma _{I}}$ by the translation by ${U(I)_{\mathbb {Z}}^{\prime }}/{U(I)_{\mathbb {Z}}}$ (which is nontrivial exactly when I is irregular). Since ${U(I)_{\mathbb {Z}}^{\prime }}/{U(I)_{\mathbb {Z}}}\subset \Gamma '(I)_{{\mathbb {Z}}}/{U(I)_{\mathbb {Z}}}$ , we have

(7.1) $$ \begin{align} ({\mathcal{D}}/{U(I)_{\mathbb{Z}}})^{\Sigma_{I}}/{\overline{\Gamma(I)}_{\mathbb{Z}}} & = ({\mathcal{D}}/{U(I)_{\mathbb{Z}}})^{\Sigma_{I}}/(\Gamma'(I)_{{\mathbb{Z}}}/{U(I)_{\mathbb{Z}}}) \\ \nonumber & \simeq ({\mathcal{D}}/{U(I)_{\mathbb{Z}}^{\prime}})^{\Sigma_{I}}/(\Gamma'(I)_{{\mathbb{Z}}}/{U(I)_{\mathbb{Z}}^{\prime}}). \end{align} $$

Since $\Sigma _{I}$ is basic with respect to ${U(I)_{\mathbb {Z}}^{\prime }}$ and $-{\mathrm {id}}\in \Gamma '$ , we can apply the result of [13] to the last quotient to see that this has canonical singularities.

8 Modular forms and pluricanonical forms

Let L be a lattice of signature $(2, b)$ , and let ${\Gamma }$ be a subgroup of ${\mathrm {O}^{+}(L)}$ of finite index. For simplicity, we assume $b\geq 3$ . In this section, we compare the vanishing order of cusp forms and pluricanonical forms, and explain how the low weight cusp form trick of Gritsenko, Hulek, and Sankaran [7] is modified at irregular boundary divisors. We take this occasion to generalize “low weight” to “low slope,” for possible future use.

8.1 Modular forms

Let $\mathcal {L}=\mathcal {O}_{{{\mathbb P}}L_{{\mathbb {C}}}}(-1)|_{{\mathcal {D}}}$ be the restriction of the tautological line bundle to ${\mathcal {D}}\subset {{\mathbb P}}L_{{\mathbb {C}}}$ . Let $\chi $ be a character of ${\Gamma }$ . By our assumption $b\geq 3$ , $\chi ({\Gamma })\subset {\mathbb {C}}^{\times }$ is finite [14]. We assume that $\chi |_{{U(I)_{\mathbb {Z}}}}\equiv 1$ for every zero-dimensional cusp I. This holds, e.g., for $\chi =1, \det $ . A  ${\Gamma }$ -invariant section of the ${\Gamma }$ -linearized line bundle $\mathcal {L}^{\otimes k}\otimes \chi $ over ${\mathcal {D}}$ is called a modular form of weight k and character $\chi $ with respect to ${\Gamma }$ .

Let I be a rank $1$ primitive isotropic sublattice of L. We choose a generator $l_{I}$ of I. This defines a frame $s_{I}$ of $\mathcal {L}$ determined by the condition $(s_{I}([\omega ]), l_{I})=1$ , where we view $s_{I}([\omega ])\in \mathcal {L}_{[\omega ]}={\mathbb {C}}\omega \subset L_{{\mathbb {C}}}$ . The factor of automorphy with respect to $s_{I}$ is given by

(8.1) $$ \begin{align} j(\gamma, [\omega]) = \frac{(\gamma\omega, l_{I})}{(\omega, l_{I})} = \frac{(\omega, \gamma^{-1}l_{I})}{(\omega, l_{I})} , \qquad \gamma\in {\Gamma}, \; [\omega]\in {\mathcal{D}}. \end{align} $$

Let $1_{\chi }$ be a nonzero vector in the representation line of $\chi $ . Then $s_{I}^{\otimes k}\otimes 1_{\chi }$ is a frame of the line bundle $\mathcal {L}^{\otimes k}\otimes \chi $ , via which modular forms $F=f s_{I}^{\otimes k}\otimes 1_{\chi }$ of weight k and character $\chi $ are identified with holomorphic functions f on ${\mathcal {D}}$ satisfying

$$ \begin{align*} f(\gamma [\omega]) = \chi(\gamma)j(\gamma, [\omega])^{k}f([\omega]), \quad \gamma \in {\Gamma}, \; [\omega]\in {\mathcal{D}}. \end{align*} $$

Since $s_{I}^{\otimes k}\otimes 1_{\chi }$ is invariant under ${U(I)_{\mathbb {Z}}}$ by our assumption, f is ${U(I)_{\mathbb {Z}}}$ -invariant, hence descends to a function on ${\mathcal {D}}/{U(I)_{\mathbb {Z}}}$ . By the tube domain realization ${\mathcal {D}}\to {\mathcal {D}}_{I}\subset U(I)_{{\mathbb {C}}}$ (after a choice of $I'\subset L$ with $(I, I')\not \equiv 0$ ), f is identified with a function on ${\mathcal {D}}_{I}$ invariant under translation by the lattice ${U(I)_{\mathbb {Z}}}$ . Then it admits a Fourier expansion

(8.2) $$ \begin{align} f(Z) = \sum_{l\in U(I)_{{\mathbb{Z}}}^{\vee}} a(l) q^{l}, \quad q^{l}=\mathrm{exp}(2\pi i(l, Z)), \; Z\in \mathcal{D}_{I}. \end{align} $$

By the Koecher principle, we have $a(l)\ne 0$ only when $l\in \overline {{\mathcal {C}_{I}}}$ . The modular form F is called a cusp form if $a(l)=0$ for every $l\in U(I)_{{\mathbb {Z}}}^{\vee }$ with $(l, l)=0$ at every rank $1$ primitive isotropic sublattice I of L. ( $a(0)$ is the value of f at the zero-dimensional cusp for I, and $\sum _{\sigma \cap U(I)_{{\mathbb {Z}}}^{\vee }}a(l)q^{l}$ for an isotropic ray $\sigma =\sigma _{J}$ gives the restriction of f to the one-dimensional cusp for $J\supset I$ .)

Fourier expansion at an irregular cusp satisfies the following.

Lemma 8.1 Suppose that I is irregular and $-E_{w}\in {\Gamma (I)_{\mathbb {Z}}}$ . When the weight k satisfies $\chi (-E_{w})=(-1)^{k+1}$ , e.g., k odd for $\chi =1$ or $k\not \equiv b$ mod $2$ for $\chi =\det $ , then we have $a(l)=0$ for $l\in (U(I)_{{\mathbb {Z}}}^{\prime })^{\vee }$ . In particular, $a(0)=0$ in this case. When $\chi (-E_{w})=(-1)^{k}$ , we have $a(l)=0$ for $l\not \in (U(I)_{{\mathbb {Z}}}^{\prime })^{\vee }$ .

Proof Since $-E_{w}$ acts on I by $-1$ , the factor of automorphy of $-E_{w}$ on $\mathcal {L}$ is $-1$ by (8.1). Therefore, we find that $f(Z+w)=\chi (-E_{w})(-1)^{k}f(Z)$ . Thus, we have ${f(Z+w)=-f(Z)}$ when $\chi (-E_{w})=(-1)^{k+1}$ , while $f(Z+w)=f(Z)$ when ${\chi (-E_{w})=(-1)^{k}}$ .

On the other hand, since w generates ${U(I)_{\mathbb {Z}}^{\prime }}/{U(I)_{\mathbb {Z}}}\simeq {\mathbb {Z}}/2$ by Proposition 3.1, pairing with w defines an isomorphism $U(I)_{{\mathbb {Z}}}^{\vee }/({U(I)_{\mathbb {Z}}^{\prime }})^{\vee } \to \frac {1}{2}{\mathbb {Z}}/{\mathbb {Z}}$ . Thus, we have $(l, w)\in {\mathbb {Z}}$ for $l\in (U(I)_{{\mathbb {Z}}}^{\prime })^{\vee }$ , while $(l, w)\in 1/2+{\mathbb {Z}}$ for $l\in U(I)_{{\mathbb {Z}}}^{\vee }-(U(I)_{{\mathbb {Z}}}^{\prime })^{\vee }$ . Therefore, if we substitute $Z\to Z+w$ into $q^{l}=\mathrm {exp}(2\pi i(l, Z))$ , then $q^{l}\to q^{l}$ if $l\in (U(I)_{{\mathbb {Z}}}^{\prime })^{\vee }$ and $q^{l}\to -q^{l}$ if $l\in U(I)_{{\mathbb {Z}}}^{\vee }-(U(I)_{{\mathbb {Z}}}^{\prime })^{\vee }$ . This implies our assertion.

8.2 Vanishing order

In this subsection, we study the vanishing order of modular forms along boundary divisors. We will define two types of vanishing order: $\nu _{\sigma }(F)$ and $\nu _{\sigma , geom}(F)$ . $\nu _{\sigma }(F)$ is defined by Fourier expansion and is always an integer. On the other hand, $\nu _{\sigma , geom}(F)$ can be strictly half-integral, and measures the vanishing order at the level of ${\mathcal {F}(\Gamma )^{\Sigma }}$ .

Let I be a rank $1$ primitive isotropic sublattice of L. Let $\Sigma =\Sigma _{I}=(\sigma _{\alpha })$ be a ${\Gamma (I)_{\mathbb {Z}}}$ -admissible fan in $U(I)_{{\mathbb {R}}}$ , and let $\sigma $ be a ray in $\Sigma $ . Let $w_{\sigma }$ be the generator of $\sigma \cap {U(I)_{\mathbb {Z}}}$ . Let $f(Z)=\sum _{l\in U(I)_{{\mathbb {Z}}}^{\vee }}a(l)q^{l}$ be the Fourier expansion of a ${\Gamma }$ -modular form ${F=f s_{I}^{\otimes k}\otimes 1_{\chi }}$ around I. We define the vanishing order of F along $\sigma $ as

$$ \begin{align*} \nu_{\sigma}(F) = \min \{ \: (l, w_{\sigma}) \: | \: l\in U(I)_{{\mathbb{Z}}}^{\vee}, \: a(l)\ne 0 \: \}. \end{align*} $$

This is a nonnegative integer because $w_{\sigma }\in {\mathcal {C}_{I}}$ has nonnegative pairing with $\overline {{\mathcal {C}_{I}}}$ . Clearly, $\nu _{\sigma }(F)$ depends on ${U(I)_{\mathbb {Z}}}$ and hence on ${\Gamma }$ . If we shrink ${\Gamma }$ without changing F and $\sigma $ , then $\nu _{\sigma }(F)$ will be multiplied in general.

When $\sigma $ is positive-definite, we have $\sigma ^{\perp }\cap \overline {{\mathcal {C}_{I}}} = \{ 0 \}$ , and so $l=0$ is the only vector in $\overline {{\mathcal {C}_{I}}}$ with $(l, w_{\sigma })=0$ . Therefore, for such $\sigma $ , we have $\nu _{\sigma }(F)>0$ if and only if $a(0)=0$ . Similarly, when $\sigma $ is isotropic, we have $\sigma ^{\perp }\cap \overline {{\mathcal {C}_{I}}} = \sigma $ . Therefore, in this case, we have $\nu _{\sigma }(F)>0$ if and only if $a(l)=0$ for all $l\in \sigma \cap U(I)_{{\mathbb {Z}}}^{\vee }$ . Thus, F is a cusp form if and only if $\nu _{\sigma }(F)>0$ at every ray $\sigma $ at every zero-dimensional cusp I.

The following criterion is trivial but perhaps might be sometimes useful in view of Theorem 8.9. Compare with [1, 5] in related cases.

Corollary 8.2 Assume that the following holds: if $a(l)\ne 0$ , then $(l, w)\geq r$ for every $w\in {U(I)_{\mathbb {Z}}}\cap \overline {{\mathcal {C}_{I}}}$ . Then we have $\nu _{\sigma }(F)\geq r$ for every ray $\sigma \in \Sigma $ .

Proof Take w to be the generator of $\sigma \cap {U(I)_{\mathbb {Z}}}$ .

When $\sigma $ is irregular, $\nu _{\sigma }(F)$ belongs to the following parity.

Proposition 8.3 Suppose that the ray $\sigma $ is irregular and $-E_{w}\in {\Gamma (I)_{\mathbb {Z}}}$ . Then $\nu _{\sigma }(F)$ is odd when $\chi (-E_{w})=(-1)^{k+1}$ , and even when $\chi (-E_{w})=(-1)^{k}$ .

Proof Let $w_{\sigma }$ be the generator of $\sigma \cap {U(I)_{\mathbb {Z}}}$ . Since ${U(I)_{\mathbb {Z}}^{\prime }}=\langle {U(I)_{\mathbb {Z}}}, w_{\sigma }/2 \rangle $ , a vector l of $U(I)_{{\mathbb {Z}}}^{\vee }$ belongs to $(U(I)_{{\mathbb {Z}}}^{\prime })^{\vee }$ if and only if $(l, w_{\sigma })$ is even. Then our assertion follows from Lemma 8.1.

We also define the geometric vanishing order of F along $\sigma $ as

$$ \begin{align*} \nu_{\sigma, geom}(F) = \begin{cases} \nu_{\sigma}(F), & \sigma : \textrm{regular}, \\ \frac{1}{2}\nu_{\sigma}(F), & \sigma : \textrm{irregular}. \end{cases} \end{align*} $$

If $w_{\sigma }^{\prime }$ is the generator of $\sigma \cap {U(I)_{\mathbb {Z}}^{\prime }}$ , we can write uniformly as

(8.3) $$ \begin{align} \nu_{\sigma, geom}(F) = \min \{ \: (l, w_{\sigma}^{\prime}) \: | \: l\in U(I)_{{\mathbb{Z}}}^{\vee}, \: a(l)\ne 0 \: \}. \end{align} $$

Note that $\nu _{\sigma , geom}(F)$ is in $1/2+{\mathbb {Z}}$ when $\sigma $ is irregular and the weight k satisfies $\chi (-E_{w})=(-1)^{k+1}$ so that $\nu _{\sigma }(F)$ is odd.

Geometric interpretation of $\nu _{\sigma }(F)$ is as follows. Recall that the ray $\sigma $ corresponds to a boundary divisor $D(\sigma )$ of the partial compactification ${\mathcal {X}(I)^{\Sigma }}$ of ${\mathcal {X}(I)}={\mathcal {D}}/{U(I)_{\mathbb {Z}}}$ . The line bundle $\mathcal {L}^{\otimes k} \otimes \chi $ descends to a line bundle over ${\mathcal {X}(I)}$ , again denoted by $\mathcal {L}^{\otimes k} \otimes \chi $ . The point is that, since $s_{I}^{\otimes k}\otimes 1_{\chi }$ is ${U(I)_{\mathbb {Z}}}$ -invariant, it descends to a frame of $\mathcal {L}^{\otimes k} \otimes \chi $ over ${\mathcal {X}(I)}$ , and we use this frame to extend $\mathcal {L}^{\otimes k} \otimes \chi $ to a line bundle over ${\mathcal {X}(I)^{\Sigma }}$ , still denoted by the same notation. Namely, $s_{I}^{\otimes k}\otimes 1_{\chi }$ extends to a frame of the extended line bundle by definition. The property $l \in \overline {{\mathcal {C}_{I}}}=\overline {{\mathcal {C}_{I}}}^{\vee }$ in the Fourier expansion implies that a modular form F extends holomorphically over ${\mathcal {X}(I)^{\Sigma }}$ as a section of $\mathcal {L}^{\otimes k}\otimes \chi $ .

Proposition 8.4 $\nu _{\sigma }(F)$ is equal to the vanishing order of F as a section of $\mathcal {L}^{\otimes k} \otimes \chi $ over ${\mathcal {X}(I)^{\Sigma }}$ along the boundary divisor $D(\sigma )$ .

Proof Recall that $\sigma $ defines a sub-toroidal embedding $\mathcal {X}(I)^{\sigma }\subset {\mathcal {X}(I)^{\Sigma }}$ , the unique boundary divisor of which is a Zariski open set of $D(\sigma )$ and is the quotient torus (or its analytic open set) defined by the quotient lattice ${U(I)_{\mathbb {Z}}}/{\mathbb {Z}}w_{\sigma }$ . The character group of this boundary torus is $\sigma ^{\perp }\cap U(I)_{{\mathbb {Z}}}^{\vee }$ . We choose a vector $l_{\sigma } \in U(I)_{{\mathbb {Z}}}^{\vee }$ such that $(l_{\sigma }, w_{\sigma })=1$ and put $q=q^{l_{\sigma }}$ , which is a character of $T(I)$ . Then q extends holomorphically over $\mathcal {X}(I)^{\sigma }$ with $D(\sigma )=(q=0)$ . The Fourier expansion (8.2) can be arranged as $f=\sum _{m\geq 0}\varphi _{m}q^{m}$ where

$$ \begin{align*} \varphi_{m} = \sum_{l\in \sigma^{\perp}\cap U(I)_{{\mathbb{Z}}}^{\vee}} a(l+ml_{\sigma})q^{l}. \end{align*} $$

This is a Taylor expansion of f along the divisor $D(\sigma )$ . Since $(l+ml_{\sigma }, w_{\sigma })=m$ for $l\in \sigma ^{\perp }\cap U(I)_{{\mathbb {Z}}}^{\vee }$ , we find that

$$ \begin{align*} \nu_{\sigma}(F) = \min \{ \: m \: | \: \varphi_{m}\not\equiv 0 \: \}. \end{align*} $$

This proves our assertion.

We can also give a geometric interpretation of $\nu _{\sigma , geom}(F)$ when

(8.4) $$ \begin{align} s_{I}^{\otimes k}\otimes 1_{\chi} \; \textrm{is invariant under} \; U(I)_{{\mathbb{Z}}}^{\star} = (\{ \pm{\mathrm{id}} \} \cdot {U(I)_{\mathbb{Q}}})\cap {\Gamma}. \end{align} $$

This holds, e.g., when k is even with $\chi =1$ and when $k\equiv b$ mod $2$ with $\chi =\det $ . Recall that ${U(I)_{\mathbb {Z}}^{\prime }}$ is the image of $U(I)_{{\mathbb {Z}}}^{\star }$ in ${U(I)_{\mathbb {Q}}}$ . Under the condition (8.4), the function $f(Z)$ on the tube domain $\mathcal {D}_{I}$ is invariant under translation by ${U(I)_{\mathbb {Z}}^{\prime }}$ , so the index lattice in the Fourier expansion reduces to $({U(I)_{\mathbb {Z}}^{\prime }})^{\vee }\subset U(I)_{{\mathbb {Z}}}^{\vee }$ . In other words, $a(l)=0$ if $l\not \in ({U(I)_{\mathbb {Z}}^{\prime }})^{\vee }$ , so $\nu _{\sigma , geom}(F)$ is an integer. The frame $s_{I}^{\otimes k}\otimes 1_{\chi }$ descends to a frame of $\mathcal {L}^{\otimes k}\otimes \chi $ over

$$ \begin{align*} {\mathcal{X}(I)'} = {\mathcal{D}}/U(I)_{{\mathbb{Z}}}^{\star} = {\mathcal{D}}/{U(I)_{\mathbb{Z}}^{\prime}}, \end{align*} $$

using which we can extend $\mathcal {L}^{\otimes k}\otimes \chi $ to a line bundle over ${(\mathcal {X}(I)')^{\Sigma }}$ . The ray $\sigma $ corresponds to a boundary divisor $D(\sigma )'$ of ${(\mathcal {X}(I)')^{\Sigma }}$ . We have

• • $D(\sigma )'=D(\sigma )$ in ${\mathcal {X}(I)^{\Sigma }}={(\mathcal {X}(I)')^{\Sigma }}$ when I is regular.

• • $D(\sigma )'\simeq D(\sigma )$ with ${\mathcal {X}(I)^{\Sigma }}\to {(\mathcal {X}(I)')^{\Sigma }}$ doubly ramified along $D(\sigma )'$ when $\sigma $ is irregular.

• • $D(\sigma )'$ is the quotient of $D(\sigma )$ by ${U(I)_{\mathbb {Z}}^{\prime }}/{U(I)_{\mathbb {Z}}}\simeq {\mathbb {Z}}/2$ with ${\mathcal {X}(I)^{\Sigma }}\to {(\mathcal {X}(I)')^{\Sigma }}$ unramified along $D(\sigma )'$ when I is irregular but $\sigma $ is regular.

Then we see, either from Proposition 8.4 or by a similar argument, the following.

Proposition 8.5 When (8.4) holds, $\nu _{\sigma , geom}(F)$ is equal to the vanishing order of F as a section of $\mathcal {L}^{\otimes k}\otimes \chi $ over ${(\mathcal {X}(I)')^{\Sigma }}$ along the boundary divisor $D(\sigma )'$ .

The vanishing order at a one-dimensional cusp J is reduced to the case considered above. We choose a rank $1$ primitive sublattice $I\subset J$ and let $\sigma _{J}$ be the isotropic ray in $U(I)_{{\mathbb {R}}}$ corresponding to J. Then we define

$$ \begin{align*} \nu_{J}(F) = \nu_{\sigma_{J}}(F), \qquad \nu_{J, geom}(F) = \nu_{\sigma_{J}, geom}(F). \end{align*} $$

The Taylor expansion $f=\sum _{m}\varphi _{m}q^{m}$ in this case is nothing but the Fourier–Jacobi expansion, and $\varphi _{m}$ is essentially the mth Fourier–Jacobi coefficient. Thus, $\nu _{J}(F)$ is the minimal degree of nonzero Fourier–Jacobi coefficients.

We also have the following geometric interpretation of $\nu _{J}(F)$ . We use the ${U(J)_{\mathbb {Z}}}$ -invariant frame $s_{I}^{\otimes k}\otimes 1_{\chi }$ to extend $\mathcal {L}^{\otimes k}\otimes \chi $ to a line bundle over ${\overline {\mathcal {X}(J)}}$ . This is the pullback of the extended line bundle $\mathcal {L}^{\otimes k}\otimes \chi $ over ${\mathcal {X}(I)^{\Sigma }}$ by the etale gluing map ${\overline {\mathcal {X}(J)}}\to {\mathcal {X}(I)^{\Sigma }}$ . This extension does not depend on the choice of I up to isomorphism. Then $\nu _{J}(F)$ is the vanishing order of F as a section of the extended line bundle $\mathcal {L}^{\otimes k}\otimes \chi $ over ${\overline {\mathcal {X}(J)}}$ along the boundary divisor. Similarly, when $s_{I}^{\otimes k}\otimes 1_{\chi }$ is invariant under $U(J)_{{\mathbb {Z}}}^{\star }$ , $\nu _{J,geom}(F)$ equals to the vanishing order of F along the boundary divisor of $\overline {{\mathcal {D}}/U(J)_{{\mathbb {Z}}}^{\star }}=\overline {{\mathcal {D}}/{U(J)_{\mathbb {Z}}^{\prime }}}$ .

8.3 Pluricanonical forms

In this subsection, we compare the vanishing order of modular forms and pluricanonical forms along the boundary divisors. Recall that we have a canonical isomorphism

$$ \begin{align*} \mathcal{L}^{\otimes b}\otimes \det \simeq K_{\mathcal{D}} \end{align*} $$

over ${\mathcal {D}}$ , as a consequence of the isomorphism $K_{{{\mathbb P}}L_{{\mathbb {C}}}}\simeq \mathcal {O}_{{{\mathbb P}}L_{{\mathbb {C}}}}(-b-2)\otimes \det $ and the adjunction formula. Let I be a rank $1$ primitive isotropic sublattice of L. The above isomorphism descends to $\mathcal {L}^{\otimes b}\otimes \det \simeq K_{{\mathcal {X}(I)'}}$ over ${\mathcal {X}(I)'}={\mathcal {D}}/U(I)_{{\mathbb {Z}}}^{\star }$ . Both line bundles are extended over the partial compactification ${(\mathcal {X}(I)')^{\Sigma }}$ in the respective manner: $\mathcal {L}^{\otimes b}\otimes \det $ is extended by the frame $s_{I}^{\otimes b}\otimes 1_{\det }$ , while $K_{{\mathcal {X}(I)'}}$ is extended to $K_{{(\mathcal {X}(I)')^{\Sigma }}}$ .

Proposition 8.6 (cf. [15])

Over ${(\mathcal {X}(I)')^{\Sigma }}$ , the above isomorphism extends to

$$ \begin{align*} \mathcal{L}^{\otimes b}\otimes \det \: \simeq \: K_{{(\mathcal{X}(I)')^{\Sigma}}} \left( \sum_{\sigma}D(\sigma)'\right), \end{align*} $$

where $\sigma $ ranges over all rays in $\Sigma $ and $D(\sigma )'$ is the boundary divisor of ${(\mathcal {X}(I)')^{\Sigma }}$ corresponding to $\sigma $ .

Proof By the isomorphism $\mathcal {L}^{\otimes b}\otimes \det \simeq K_{{\mathcal {D}}}$ , the frame $s_{I}^{\otimes b}\otimes 1_{\det }$ of $\mathcal {L}^{\otimes b}\otimes \det $ corresponds to a flat canonical form $\omega _{I}$ on the tube domain ${\mathcal {D}}_{I}\subset U(I)_{{\mathbb {C}}}$ , because both extend over ${\mathcal {D}}(I)\simeq U(I)_{{\mathbb {C}}}$ and are $U(I)_{{\mathbb {C}}}$ -invariant. Let $\sigma $ be a ray in $\Sigma $ and $w_{\sigma }^{\prime }$ be the generator of $\sigma \cap {U(I)_{\mathbb {Z}}^{\prime }}$ . We take a vector $l_{\sigma }\in ({U(I)_{\mathbb {Z}}^{\prime }})^{\vee }$ with $(l_{\sigma }, w_{\sigma }^{\prime })=1$ and extend it to a basis of $({U(I)_{\mathbb {Z}}^{\prime }})^{\vee }$ . This defines a coordinate $Z_{1}=(l_{\sigma }, \cdot ), Z_{2}, \ldots , Z_{b}$ on $U(I)_{{\mathbb {C}}}$ . We have $\omega _{I}=dZ_{1}\wedge \cdots \wedge dZ_{b}$ up to constant. Then $q=q^{l_{\sigma }}, Z_{2}, \ldots , Z_{b}$ define a local coordinate around a point of $D(\sigma )'\subset {(\mathcal {X}(I)')^{\Sigma }}$ with $D(\sigma )'=(q=0)$ . Since we have

$$ \begin{align*} s_{I}^{\otimes b}\otimes 1_{\det} = dZ_{1} \wedge \cdots \wedge dZ_{b} = \frac{dq}{q}\wedge dZ_{2} \wedge \cdots \wedge dZ_{b} \end{align*} $$

around a point of $D(\sigma )'$ , this proves our assertion.

This is the situation at a local chart for the boundary. We pass to the global situation.

Proposition 8.7 Let F be a modular form of weight $mb$ and character $\det ^{m}$ with respect to ${\Gamma }$ and $\omega _{F}$ be the corresponding rational m-canonical form on ${\mathcal {F}(\Gamma )^{\Sigma }}$ . Let I be a zero-dimensional cusp, $\sigma $ be a ray in $\Sigma _{I}$ , and $\Delta (\sigma )$ be the corresponding boundary divisor of ${\mathcal {F}(\Gamma )^{\Sigma }}$ . Then the vanishing order $\nu _{\Delta (\sigma )}(\omega _{F})$ of $\omega _{F}$ along $\Delta (\sigma )$ is given by

$$ \begin{align*} \nu_{\Delta(\sigma)}(\omega_{F}) = \nu_{\sigma, geom}(F) - m = \begin{cases} \nu_{\sigma}(F)-m, & \sigma : \textrm{regular}, \\ \frac{1}{2}\nu_{\sigma}(F)-m, & \sigma : \textrm{irregular}. \end{cases} \end{align*} $$

Proof Let $\pi \colon (\mathcal {X}(I)')^{\Sigma _{I}} \to {\mathcal {F}(\Gamma )^{\Sigma }}$ be the projection. By Propositions 8.5 and 8.6, we have

$$ \begin{align*} \nu_{D(\sigma)'}(\pi^{\ast}\omega_{F}) = \nu_{\sigma, geom}(F) - m. \end{align*} $$

By Proposition 7.2(1), $\pi $ is not ramified along $D(\sigma )'$ , regardless of whether $\sigma $ is positive-definite or isotropic. This implies that $\nu _{D(\sigma )'}(\pi ^{\ast }\omega _{F})=\nu _{\Delta (\sigma )}(\omega _{F})$ .

When $\sigma =\sigma _{J}$ is isotropic, the above equality can be written as

$$ \begin{align*} \nu_{\Delta(\sigma_{J})}(\omega_{F}) = \nu_{J, geom}(F) - m, \end{align*} $$

where $\Delta (\sigma _{J})$ is the boundary divisor of ${\mathcal {F}(\Gamma )^{\Sigma }}$ over J.

By Gritsenko, Hulek, and Sankaran [7], every irreducible component of the ramification divisor of ${\mathcal {D}}\to {\mathcal {F}(\Gamma )}$ has ramification index $2$ (and is defined by a reflection). Since every boundary divisor of ${\mathcal {F}(\Gamma )^{\Sigma }}$ is of the form $\Delta (\sigma )$ for some ray $\sigma $ at some zero-dimensional cusp I, Proposition 8.7 implies the following.

Corollary 8.8 The m-canonical form $\omega _{F}$ extends holomorphically over the regular locus of ${\mathcal {F}(\Gamma )^{\Sigma }}$ if and only if the following hold:

1. (1) $\nu _{R}(F)\geq m$ at every irreducible component R of the ramification divisor of ${\mathcal {D}}\to {\mathcal {F}(\Gamma )}$ .

2. (2) $\nu _{\sigma }(F)\geq m$ at every regular ray $\sigma $ for every zero-dimensional cusp.

3. (3) $\nu _{\sigma }(F)\geq 2m$ at every irregular ray $\sigma $ for every irregular zero-dimensional cusp.

Note that extendability at the boundary divisors over the one-dimensional cusps is encoded in the conditions (2) and (3) at isotropic rays $\sigma $ for adjacent zero-dimensional cusps.

8.4 Low slope cusp form criterion

We now arrive at our principal purpose. Theorem 1.2 follows from the case $k<b$ in the following.

Theorem 8.9 Let L be a lattice of signature $(2, b)$ with $b\geq 9$ . Let ${\Gamma }$ be a subgroup of ${\mathrm {O}^{+}(L)}$ of finite index. We take a ${\Gamma }$ -admissible collection of fans $\Sigma =(\Sigma _{I})$ such that $\Sigma _{I}$ is basic with respect to ${U(I)_{\mathbb {Z}}^{\prime }}={U(I)_{\mathbb {Q}}}\cap {\langle \Gamma , -\mathrm {id} \rangle }$ at each zero-dimensional cusp I. Assume that we have a cusp form F of some weight k and character with respect to ${\Gamma }$ satisfying the following:

1. (1) At every irreducible component R of the ramification divisor of ${\mathcal {D}}\to {\mathcal {F}(\Gamma )}$ , we have $\nu _{R}(F)/k> 1/b$ .

2. (2) At every regular ray $\sigma $ of $\Sigma _{I}$ at every zero-dimensional cusp I, we have $\nu _{\sigma }(F)/k> 1/b$ .

3. (3) At every irregular ray $\sigma $ of $\Sigma _{I}$ at every irregular zero-dimensional cusp I, we have $\nu _{\sigma }(F)/k> 2/b$ .

Then ${\mathcal {F}(\Gamma )}$ is of general type.

Proof The following argument is a slight modification of the proof of [7, Theorem 1.1], avoiding the use of a neat cover.

Replacing F with its power, which does not change the slopes $\nu _{\ast }(F)/k$ , we may assume that the character $\chi $ is trivial. We first consider the case $b\nmid k$ . By further replacing F with its power $F^{2^{N}}$ , where N is determined by $[k/b]+2^{-N-1} \leq k/b < [k/b]+2^{-N}$ , we may assume that $k/b\geq [k/b]+1/2$ so that $[2k/b]=2[k/b]+1$ . We write $N_{0}=[k/b]+1$ . Then F has vanishing order $\geq N_{0}$ at the ramification divisors of ${\mathcal {D}}\to {\mathcal {F}(\Gamma )}$ and at the regular boundary divisors, and vanishing order $\geq 2N_{0}$ at the irregular boundary divisors. We denote by $M_{l}({\Gamma })$ the space of ${\Gamma }$ -modular forms of weight l with trivial character. For an even number m, we consider the subspace $V_{m}=F^{m}\cdot M_{(bN_{0}-k)m}({\Gamma })$ of $M_{bN_{0}m}({\Gamma })$ . Modular forms in $V_{m}$ have vanishing order $\geq mN_{0}$ at the interior ramification divisors and at the regular boundary divisors, and vanishing order $\geq 2mN_{0}$ at the irregular boundary divisors. Thus, the corresponding $mN_{0}$ -canonical forms extend holomorphically over the regular locus of ${\mathcal {F}(\Gamma )^{\Sigma }}$ by Corollary 8.8. By our choice of $\Sigma $ , ${\mathcal {F}(\Gamma )^{\Sigma }}$ has canonical singularities at the boundary points by Proposition 7.4, and the interior ${\mathcal {F}(\Gamma )}$ has canonical singularities by Gritsenko, Hulek, and Sankaran [7]. Therefore, these $mN_{0}$ -canonical forms extend holomorphically over a desingularization X of ${\mathcal {F}(\Gamma )^{\Sigma }}$ . Since $bN_{0}>k$ , we have

$$ \begin{align*} \dim V_{m} = \dim M_{(bN_{0}-k)m}({\Gamma}) \sim c\cdot m^{b} \qquad (m\to \infty) \end{align*} $$

for some $c>0$ , so we find that $K_{X}$ is big.

When $b\mid k$ , we replace F with the product of a sufficiently large power of F and a modular form of weight indivisible by b. This perturbs the slopes $\nu _{\ast }(F)/k$ only by $\varepsilon $ , so the inequalities in (1)–(3) still hold. Then the same argument works.

Remark 8.10 If we replace “ $>$ ” in the conditions (1)–(3) by “ $\geq $ ,” then the conclusion will be weakened to “ ${\mathcal {F}(\Gamma )}$ has nonnegative Kodaira dimension.” A power of F gives a nonzero pluricanonical form.

Geometric explanation of Theorem 8.9 is as follows. We have the ${\mathbb {Q}}$ -linear equivalence

$$ \begin{align*} K_{{\mathcal{F}(\Gamma)^{\Sigma}}} \sim_{{\mathbb{Q}}} b\mathcal{L} - B/2 - \Delta_{reg} - \Delta_{irr} \end{align*} $$

over ${\mathcal {F}(\Gamma )^{\Sigma }}$ , where B is the interior branch divisor and $\Delta _{reg}, \Delta _{irr}$ are the regular and irregular boundary divisors, respectively. The coefficients of B and $\Delta _{irr}$ will be multiplied by $2$ when pulled back to local charts. The existence of the cusp form F means that $b'\mathcal {L} - B/2 - \Delta _{reg} - \Delta _{irr}$ is ${\mathbb {Q}}$ -effective for some $b'<b$ , $b'\in {\mathbb {Q}}$ . (To be explicit, $b'=k/N_{0}$ in the case $b\nmid k$ in the proof.) Thus, we have

$$ \begin{align*} K_{{\mathcal{F}(\Gamma)^{\Sigma}}} \sim ({\mathbb{Q}}\textrm{-effective}) + (b-b')\mathcal{L} = ({\mathbb{Q}}\textrm{-effective}) + (\textrm{big}) = (\textrm{big}), \end{align*} $$

and the singularities do not impose obstruction.