ARITHMETICITY OF $\mathbb {C}$-FUCHSIAN SUBGROUPS OF SOME NONARITHMETIC LATTICES

Abstract

We study the arithmeticity of $\mathbb {C}$-Fuchsian subgroups of some nonarithmetic lattices constructed by Deraux et al. [‘New non-arithmetic complex hyperbolic lattices’, Invent. Math. 203 (2016), 681–771]. Our results give an answer to a question raised by Wells [Hybrid Subgroups of Complex Hyperbolic Isometries, Doctoral thesis, Arizona State University, 2019].

1 Introduction

Whether a group is arithmetic is a significant question for discrete subgroups and lattices in semisimple Lie groups. Margulis’ celebrated super-rigidity and arithmeticity theorems demonstrate that a lattice in a semisimple Lie group is arithmetic when its real rank is at least two. This means that nonarithmetic lattices only exist in real rank one, where the associated symmetric spaces are real hyperbolic spaces, complex hyperbolic spaces, quaternionic hyperbolic spaces and the octonionic hyperbolic plane. In the last two spaces, all lattices are arithmetic due to the work of Corlette and Gromov–Schoen. In real hyperbolic spaces, where the Lie group is $\mathrm {PO}(n,1)$ , Gromov and Piatetski-Shapiro showed that nonarithmetic lattices exist in $\mathrm {PO}(n,1)$ for all $n \geq 2$ . For the case of complex hyperbolic spaces, where the Lie group is $\mathrm {PU}(n,1)$ , the existence of nonarithmetic lattices has not been widely investigated.

Let H be a Hermitian matrix with signature $(2,1)$ on $\mathbb {C}^{3}$ . The projective unitary group $\mathrm {PU}(2,1)$ of H acts as the holomorphic isometry group on the complex hyperbolic plane $\mathbf {H}_{\mathbb {C}}^2$ . The first nonarithmetic lattices in $\mathrm {PU}(2,1)$ were constructed by Mostow [9]. These lattices are the equilateral triangle groups $\mathcal {S}(p,\tau )$ , which are generated by a complex reflection $R_{1}$ with order p and an order three isometry J with $\mathrm {tr}(R_{1}J)=\tau $ . The equilateral triangle groups with some given values of $\tau $ are called sporadic triangle groups (see [3]). In [1], the authors gave a conjectural list of sporadic triangle groups and proved, by computer experimentation, that only finitely many of these sporadic triangle groups are lattices. Following this, in [2, 3], they showed which sporadic triangle groups are lattices and found new nonarithmetic lattices in $\mathrm {PU}(2,1)$ , by using a systematic approach to produce their fundamental domains.

Suppose that $\Gamma $ is a discrete subgroup of $\mathrm {PU}(2,1)$ . The Fuchsian subgroups of $\Gamma $ are defined as the intersection of $\Gamma $ with Lie subgroups isomorphic to $\mathrm {PSL}(2,\mathbb {R})$ . A Fuchsian subgroup is $\mathbb {C}$ -Fuchsian if it stabilises a complex line. In [12], Stover showed that a complex hyperbolic lattice, which contains a complex reflection, must contain a $\mathbb {C}$ -Fuchsian subgroup. Let the complex line $L_{j}$ be the fixed point set of the complex reflection $R_{j}$ for $j=1,2,3$ . In his doctoral thesis [16], Wells studied the $\mathbb {C}$ -Fuchsian subgroup stabilising the complex line $L_{1}$ of the complex hyperbolic lattice $\mathcal {S}(p,\tau )$ for $p=3,4,5,6,8,12$ and $\tau =-(1+i\sqrt {7})/2$ . He gave the generators of the $\mathbb {C}$ -Fuchsian subgroup and proved that this group is a lattice in $\mathrm {SU}(1,1)$ . Additionally, he asked the following question.

Question 1.1. Let $R_{i}$ be the complex reflection of order p so that $R_{i}$ fixes the complex line $L_{i}$ for $i=1,2,3$ . Assume that $R_{1}, R_{2}, R_{3}$ are the generators for $\mathcal {S}(p,\tau ),$ where ${p=3,4,5,6,8,12}$ and $\tau =-(1+i\sqrt {7})/2$ . Set

$$ \begin{align*}G_{1}=\langle (R_{1}R_{2})^2, (R_{1}R_{3})^2, (R_{1}R_{2}R_{3}R_{2}^{-1})^{3}, (R_{1}R_{3}^{-1}R_{2}R_{3})^{3} \rangle.\end{align*} $$

Then $G_{1}$ is a $\mathbb {C}$ -Fuchsian subgroup stabilising $L_{1}$ . Is the group $G_{1}|_{L_{1}}$ arithmetic?

Takeuchi [14] studied and characterised arithmetic Fuchsian groups of finite covolume. Subsequently, in [15], applying these results to triangle groups in $\mathrm {SL}(2,\mathbb {R})$ , he gave a necessary and sufficient condition for a triangle group to be arithmetic and derived a complete list of all arithmetic triangle groups. Maclachlan and Reid in [7] generalised Takeuchi’s methods to Kleinian groups and obtained a similar characterisation of arithmetic Kleinian groups. More effective criteria for arithmetic Fuchsian groups and Kleinian groups can be found in [4, 6, 8].

Note that $\mathcal {S}(p,\tau )$ in Question 1.1 is arithmetic when $p=3$ . Thus, $G_{1}|_{L_{1}}$ is arithmetic for $p=3$ . In the present paper, our main goal is to study the arithmeticity of the $\mathbb {C}$ -Fuchsian subgroups of nonarithmetic lattices. Our general procedure to prove the arithmeticity or nonarithmeticity of each group is as follows. Firstly, we explore a transformation interchanging the original Hermitian form into a more familiar Hermitian form. Then we transform the generators of each group into elements in $\mathrm {SU}(1,1)$ and these elements can be turned into elements in $\mathrm {SL}(2,\mathbb {R})$ since there exists a bijection between $\mathrm {SU}(1,1)$ and $\mathrm {SL}(2,\mathbb {R})$ . Finally, we check the arithmeticity of each group according to the criteria for the arithmeticity of a Fuchsian group. We obtain the following result.

Theorem 1.2. The group $G_{1}|_{L_{1}}$ is nonarithmetic for $p=4,5,6,8,12$ .

Recently, Sun [13] also considered the $\mathbb {C}$ -Fuchsian subgroups of some complex hyperbolic lattices $\mathcal {S}(p,\tau )$ appearing in [2, 3]. For each $\mathbb {C}$ -Fuchsian subgroup, she forced all pyramids of the side representatives to have the same base $L_{1}$ , and obtained a polygon in $L_{1}$ which is a fundamental domain of the $\mathbb {C}$ -Fuchsian subgroup. Applying the Poincaré polygon theorem, she gave the following presentation for each $\mathbb {C}$ -Fuchsian subgroup.

Theorem 1.3 [13].

Let $R_{1}, R_{2}, R_{3}$ be three complex reflections of order p so that $R_{i}$ fixes a complex line $L_{i}$ for $i=1,2,3$ . Suppose that $R_{1}, R_{2}, R_{3}$ are the generators for $\mathcal {S}(p,\tau )$ . Then there exist $\mathbb {C}$ -Fuchsian subgroups fixing the complex line $L_{1}$ that have the following structure according to $(\tau ,p).$

1. (i) $\tau =-1+i\sqrt {2},p=3,4,6$ : $\Gamma _{1}=\langle g_{1},g_{2},g_{3},g_{4},g_{5}\rangle ,$ where

   $$ \begin{align*} g_{1}&=(R_{1}R_{3}^{-1}R_{2}R_{3})^{2}, \quad g_{2}=(R_{1}R_{3})^{3}, \quad g_{3}=(R_{1}R_{2})^{3}, \\ g_{4}& =(R_{1}R_{2}R_{3}R_{2}^{-1})^{2}(R_{1}R_{2})^{3},\\ g_{5}&=(R_{1}R_{2}R_{3}R_{2}R_{3}^{-1}R_{2}^{-1})^{3}(R_{1}R_{2}R_{3}R_{2}^{-1})^{2}(R_{1}R_{2})^{3}. \end{align*} $$

2. (ii) $\tau =-{1+i\sqrt {7}}/{2},p=3,4,5,6,8,12$ : $\Gamma _{2}=\langle g_{1},g_{2},g_{3}\rangle ,$ where

   $$ \begin{align*}g_{1}=(R_{1}R_{2})^{2}, \quad g_{2}=R_{2}R_{3}R_{2}^{-1}R_{1}JR_{1}J, \quad g_{3}=(R_{1}R_{3})^{2}.\end{align*} $$

3. (iii) $\tau ={1+\sqrt {5}}/{2},p=3,4,5,10$ : $\Gamma _{3}=\langle g_{1},g_{2},g_{3}\rangle ,$ where

   $$ \begin{align*} g_{1} & =R_{1}R_{3}^{-1}R_{2}^{-1}R_{3}R_{2}R_{3}, \quad g_{2}=R_{1}R_{3}R_{1}R_{2}R_{1}^{-1}R_{3}^{-1}, \\ g_{3} & =(R_{1}R_{3}^{-1}R_{2}R_{3})^{3}R_{1}R_{3}^{-1}R_{2}^{-1}R_{3}R_{2}R_{3}. \end{align*} $$

A natural question is whether these $\mathbb {C}$ -Fuchsian subgroups are arithmetic. Notice that the groups $\mathcal {S}(3,-(1+i\sqrt {7})/2)$ and $\mathcal {S}(p,(1+\sqrt {5})/2)$ for $p=3,4,5,10$ are arithmetic. Hence, their $\mathbb {C}$ -Fuchsian subgroups are also arithmetic. For the remaining cases, we prove the following theorem.

Theorem 1.4. The $\mathbb {C}$ -Fuchsian subgroups $\Gamma _{1}$ and $\Gamma _{2}$ have the following properties.

• • The group $\Gamma _{1}|_{L_{1}}$ is nonarithmetic for $p=3,4,6$ .

• • The group $\Gamma _{2}|_{L_{1}}$ is nonarithmetic for $p=4,5,6,8,12$ .

2 Preliminaries

In this section, we recall some basic material on the complex hyperbolic plane, equilateral triangle groups and the arithmeticity of Fuchsian groups.

2.1 The complex hyperbolic plane

Let $\mathbb {C}^{2,1}$ be the complex vector space of dimension three equipped with a Hermitian form of signature $(2,1)$ . For H a Hermitian matrix of signature $(2,1)$ , the Hermitian form is defined as $\langle \textbf {z},\textbf {w}\rangle =\textbf {w}^{\ast }H\textbf {z}$ . Consider the subsets

$$ \begin{align*} \begin{aligned} V_{-}&=\{\textbf{z} \in \mathbb{C}^{2,1} \mid \langle \textbf{z},\textbf{z}\rangle<0 \},\\ V_{0}&=\{\textbf{z} \in \mathbb{C}^{2,1}-\{0\} \mid \langle \textbf{z},\textbf{z}\rangle=0 \},\\ V_{+}&=\{\textbf{z} \in \mathbb{C}^{2,1} \mid \langle \textbf{z},\textbf{z}\rangle>0 \}. \end{aligned} \end{align*} $$

Let $\mathbb {P}:\mathbb {C}^{2,1}-\{0\}\rightarrow \mathbb {C}P^{2}$ denote the projection map. Then the complex hyperbolic plane is $\mathbf {H}_{\mathbb {C}}^2=\mathbb {P}(V_{-})$ and its boundary is defined to be $\partial \mathbf {H}_{\mathbb {C}}^2=\mathbb {P}(V_{0})$ . Let $\rho (z,w)$ be the distance between two points $z,w\in \mathbf {H}_{\mathbb {C}}^2$ . The Bergman metric on $\mathbf {H}_{\mathbb {C}}^2$ is given by

$$ \begin{align*} \cosh^{2}\bigg( \frac{\rho(z,w)}{2} \bigg)=\frac{\langle \textbf{z},\textbf{w}\rangle \langle \textbf{w},\textbf{z}\rangle}{\langle \textbf{z},\textbf{z}\rangle \langle \textbf{w},\textbf{w}\rangle} , \end{align*} $$

where $\textbf {z},\textbf {w}\in \mathbb {C}^{2,1}$ are lifts of $z,w$ . Note that the Bergman metric is independent of the lifts of z and w.

A matrix $A \in \textrm U(2,1)$ is unitary if $\langle A\textbf {z}, A\textbf {w}\rangle =\langle \textbf {z}, \textbf {w} \rangle $ for $\textbf {z},\textbf {w} \in \mathbb {C}^{2,1}$ . A unitary matrix preserves the Bergman metric. The holomorphic isometry group of $\textbf {H}_{\mathbb {C}}^{2}$ is

$$ \begin{align*}\textrm{PU}(2,1)=\textrm{U}(2,1)/\{\textrm{e}^{{i}\theta} \textrm{I} \mid 0\leq\theta<2\pi\},\end{align*} $$

where $\mathrm {I}$ is the identity matrix in $\textrm U(2,1)$ .

Let $\textbf {n}$ be a vector in $V_{+}$ and let ${\mathbf {n}}^{\bot }$ be the orthogonal complement of $\mathbf {n}$ with respect to H. Then the intersection of the projective line $\mathbb {P}({\mathbf {n}}^{\bot })$ with $\mathbf {H}_{\mathbb {C}}^2$ is a complex line L. The vector $\textbf {n}$ is called the polar vector to the complex line L.

2.2 Equilateral triangle groups

Let $p\in \mathbb {Z}$ . Equilateral triangle groups $\mathcal {S}(p,\tau )$ are generated by three reflections $R_{1},R_{2},R_{3}$ of order p ( $p\geq 2$ ) with the property that there is a regular elliptic element J of order three such that these reflections satisfy the relationships $R_{2}=JR_{1}J^{-1}$ and $R_{3}=JR_{2}J^{-1}$ . They can be parametrised by the order p of the generators and the complex parameter $\tau =\mathrm {tr}(R_{1}J)$ .

For $i=1,2,3$ , the fixed point set of $R_{i}$ is the complex line $L_{i}$ . Let ${\mathbf {n}}_{i}$ be the polar vector to $L_{i}$ and set $u=e^{{2\pi i}/{3p}}$ . By the trace formula of $\mathrm {tr}(R_{1}J)$ , the parameter $\tau $ can be written as

$$ \begin{align*}\tau=\mathrm{tr}(R_{1}J)=(u^{2}-\bar{u})\frac{\langle {\mathbf{n}}_{j+1}, {\mathbf{n}}_{j}\rangle}{\|{\mathbf{n}}_{j+1}\| \, \|{\mathbf{n}}_{j}\|}.\end{align*} $$

Let vectors ${\mathbf {n}}_{1},{\mathbf {n}}_{2},{\mathbf {n}}_{3}$ be a basis of $\mathbb {C}^{3}$ . We write

$$ \begin{align*} {\mathbf{n}}_{1}= \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \quad {\mathbf{n}}_{2}= \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}, \quad {\mathbf{n}}_{3}= \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}. \end{align*} $$

We obtain matrices for the Hermitian form H and the permutation isometry J, given explicitly by

$$ \begin{align*} H= \begin{bmatrix} \alpha & \beta & \bar{\beta} \\ \bar{\beta} & \alpha & \beta \\ \beta & \bar{\beta} & \alpha \end{bmatrix}, \quad J= \begin{bmatrix} 0 & 0 & 1 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix}, \end{align*} $$

where $\alpha =2-u^{3}-\bar {u}^{3}$ and $\beta =(\bar {u}^{2}-u)\tau $ . For H to have signature $(2,1)$ , its determinant must be negative, namely,

$$ \begin{align*} \alpha^{3}+2{\textrm{Re}}(\beta^{3})-3\alpha|\beta|^{2}<0. \end{align*} $$

According to the formula of the complex reflection,

$$ \begin{align*} R_{1}(\textbf{z})=e^{-{i\phi}/{3}}\textbf{z}+(e^{{2i\phi}/{3}}-e^{-{i\phi}/{3}})\frac{\langle \textbf{z}, {\mathbf{n}}_{1}\rangle}{\langle {\mathbf{n}}_{1}, {\mathbf{n}}_{1}\rangle}{\mathbf{n}_{\mathbf{1}}}, \end{align*} $$

where $\phi =2\pi /p$ , and we have a representation of $R_{1}$ in $\mathrm {SU}(2,1)$ given by

$$ \begin{align*} R_{1}= \begin{bmatrix} u^{2} & \tau & -u\bar{\tau} \\ 0 & \bar{u} & 0 \\ 0 & 0 & \bar{u} \end{bmatrix}. \end{align*} $$

The corresponding matrices of $R_{2},R_{3}$ can be obtained from the relationships

$$ \begin{align*}R_{2}=JR_{1}J^{-1}, \quad R_{3}=JR_{2}J^{-1}.\end{align*} $$

It is difficult to determine the values of p and $\tau $ such that the equilateral triangle group is a lattice or discrete. A necessary condition for a group in $\mathrm {PU}(2,1)$ to be discrete is that all its elliptic elements have finite order. For an equilateral triangle group, assume that $R_{1}J$ and $R_{1}R_{2}$ are elliptic. If this assumption holds, then the equilateral triangle group is a Mostow lattice or a subgroup of a Mostow lattice, or a sporadic triangle group (see [10]). Following this, the conjectural list of lattices among sporadic triangle groups is given and proved in detail (see [1–3]). The values of p and $\tau $ for a sporadic triangle group $\mathcal {S}(p,\tau )$ to be a lattice are listed in Table 1.

Table 1 Values of $p, \tau $ such that $\mathcal {S}(p, \tau )$ are lattices.

2.3 Arithmetic Fuchsian groups

To state the criteria for the arithmeticity of Fuchsian groups, we recall the notion of the trace field and the invariant trace field. Let $\Gamma $ be a finitely generated group of $\mathrm {SL}(2,\mathbb {R})$ . The trace field of $\Gamma $ is the field generated over $\mathbb {Q}$ by the traces of the elements in $\Gamma $ , and is denoted by $\mathbb {Q}(\mathrm {tr}(\Gamma ))$ . We write tr $(\gamma )$ as the trace of an element $\gamma $ in $\Gamma $ and set $\mathrm {tr}(\Gamma )=\{\mathrm {tr}(\gamma ) \mid \gamma \in \Gamma \}$ .

The subgroup $\Gamma ^{(2)}$ of $\Gamma $ is generated by the set $\{ \gamma ^{2} \mid \gamma \in \Gamma \}$ . Since $\Gamma $ is finitely generated, $\Gamma ^{(2)}$ is a normal subgroup of finite index. The invariant trace field of $\Gamma $  is

$$ \begin{align*}k\Gamma=\mathbb{Q}(\mathrm{tr}(\Gamma^{(2)})),\end{align*} $$

which is an invariant of commensurability class of $\Gamma $ (see [11]). The following two propositions provide an easy computation for the invariant trace field $k\Gamma $ .

Proposition 2.1 [6].

Let $\gamma _{1},\ldots ,\gamma _{n}$ be in $\mathrm {SL}(2,\mathbb {C})$ such that $\mathrm {tr}(\gamma _{i})\neq 0$ for $i=1,\ldots ,n$ . Let $\Gamma $ be $\langle \gamma _{1},\ldots ,\gamma _{n}\rangle $ and let $\Gamma ^{SQ}$ be $\langle \gamma _{1}^{2},\ldots ,\gamma _{n}^{2}\rangle $ . Then $\mathbb {Q}(\mathrm {tr}(\Gamma ^{(2)}))=\mathbb {Q}(\mathrm {tr}(\Gamma ^{SQ}))$ .

Proposition 2.2 [8].

Let $\Gamma $ be generated by $\gamma _{1},\gamma _{2},\ldots ,\gamma _{n}$ , where $\gamma _{i}\in \mathrm {SL}(2,\mathbb {C})$ for $i=1,\ldots ,n$ and $\gamma \in \Gamma $ . Then $\mathrm {tr}(\gamma )$ is an integer polynomial in $\{\mathrm {tr}(\delta ) \mid \delta \in Q\}$ , where

$$ \begin{align*}Q=\{\gamma_{i_{1}}\cdots \gamma_{i_{r}} \mid r\geq1 \ \mathrm{and} \ 1\leq i_{1}<\cdots<i_{r}\leq n\}.\end{align*} $$

Next, we state results about the arithmeticity of Fuchsian groups.

Theorem 2.3 [6].

A finitely generated subgroup $\Gamma $ of $\mathrm {SL}(2,\mathbb {R})$ is arithmetic if and only if $\Gamma ^{(2)}$ is derived from a quaternion algebra.

Theorem 2.4 [14].

Let $\Gamma $ be a Fuchsian group of finite covolume. Then $\Gamma $ is a Fuchsian group derived from a quaternion algebra if and only if $\Gamma $ satisfies the following conditions.

1. (i) If $k_{1}$ is the field $\mathbb {Q}(\mathrm {tr}(\Gamma ))$ , then $k_{1}$ is an algebraic number field of finite degree and $\mathrm {tr}(\Gamma )$ is contained in the ring $O_{k_{1}}$ of integers of $k_{1}$ .

2. (ii) If $\varphi $ is any isomorphism of $k_{1}=\mathbb {Q}(\mathrm {tr}(\Gamma ))$ into $\mathbb {C}$ such that $\varphi $ is not the identity, then $\varphi (\mathrm {tr}(\Gamma ))$ is bounded in $\mathbb {C}$ .

Proposition 2.5 [14].

Let $\Gamma $ be a Fuchsian group of finite covolume. Assume that $\Gamma $ satisfies conditions (i) and (ii) of Theorem 2.4. Then $k_{1}=\mathbb {Q}(\mathrm {tr}(\Gamma ))$ is totally real. Moreover, if $\varphi $ is any isomorphism of $k_{1}$ into $\mathbb {R}$ such that $\varphi $ is not the identity, then $\varphi (\mathrm {tr}(\Gamma ))$ is contained in the interval $[-2,2]$ .

From Theorems 2.3 and 2.4 and Proposition 2.5, we have the following corollary.

Corollary 2.6. Suppose that $\Gamma $ is an arithmetic Fuchsian group of finite covolume. If $\varphi $ is any isomorphism from $k\Gamma $ to $\mathbb {R}$ such that $\varphi $ is not the identity, then $\varphi (\mathrm {tr}(\Gamma ^{(2)}))$ is contained in the interval $[-2,2]$ .

It is more convenient to determine the arithmeticity of a noncocompact Fuchsian group by the following theorem.

Theorem 2.7 [6].

A finitely generated noncocompact lattice $\Gamma $ of $\mathrm {SL}(2,\mathbb {R})$ is derived from a quaternion algebra if and only if the following conditions hold.

1. (i) $\mathrm {tr}(\gamma )$ is an integer for all $\gamma \in \Gamma $ .

2. (ii) $\mathbb {Q}(\mathrm {tr}(\Gamma ))=\mathbb {Q}.$

3 Proof of Theorem 1.2

In this section, we give a proof of Theorem 1.2. The group $G_{1}$ is the subgroup stabilising $L_{1}$ , and hence it is naturally identified with a subgroup of $\mathrm {SU}(1,1)$ . Since there is a bijection between $\mathrm {SU}(1,1)$ and $\mathrm {SL}(2,\mathbb {R})$ , this group in $\mathrm {SU}(1,1)$ can be transformed into the corresponding group in $\mathrm {SL}(2,\mathbb {R})$ , which is denoted by $G_{11}$ . The procedure of obtaining $G_{11}$ can be seen below. After that, we determine its nonarithmeticity by using Theorem 2.3, Corollary 2.6 and Theorem 2.7.

The $\mathbb {C}$ -Fuchsian subgroup $G_{1}$ is generated by $x_{1},x_{2},x_{3},x_{4}$ , where

$$ \begin{align*} x_{1}=(R_{1}R_{2})^{2},\quad x_{2}=(R_{1}R_{3})^{2}, \quad x_{3}=(R_{1}R_{2}R_{3}R_{2}^{-1})^{3}, \quad x_{4}=(R_{1}R_{3}^{-1}R_{2}R_{3})^{3}. \end{align*} $$

Let $R_{1}, R_{2}, R_{3}, H, \textbf {n}_{1}, u,\alpha $ and $\beta $ be defined as in Section 2.2. Then every point in $L_{1}$ has the form $[z,-(\alpha z+\bar {\beta })/\beta ,1]^{t} \in \mathbf {H}_{\mathbb {C}}^2$ for a complex parameter z.

We start by choosing a suitable transformation P. Let $\textbf {v}_{1}=[1/\sqrt {\alpha },0,0]^{t}$ , $\textbf {v}_{2}=[0,-\bar {\beta }/\beta ,1]^{t}$ , $\textbf {v}_{3}=[a,-(a\alpha +\bar {\beta })/\beta ,1]^{t}$ , where $a=(\bar {\beta }^{3}\, {-}\, 2\alpha |\beta |^{2}+\beta ^{3})/ (\alpha ^{2}\beta\, {-}\, \alpha \bar {\beta }^{2})$ . The vector $\textbf {v}_{1}$ is orthogonal to $L_{1}$ and the vector $\textbf {v}_{3}$ satisfies $\langle \textbf {v}_{3},\textbf {v}_{2}\rangle =0$ and $\textbf {v}_{3}\in L_{1}$ . Normalising these vectors to have unit norm, we take

$$ \begin{align*} \widetilde{\textbf{v}_{1}}=\frac{\textbf{v}_{1}}{\sqrt{\langle \textbf{v}_{1},\textbf{v}_{1}\rangle}}, \quad \widetilde{\textbf{v}_{2}}=\frac{\textbf{v}_{2}}{\sqrt{\langle \textbf{v}_{2},\textbf{v}_{2}\rangle}}, \quad \widetilde{\textbf{v}_{3}}=\frac{\textbf{v}_{3}}{i\sqrt{-\langle \textbf{v}_{3},\textbf{v}_{3}\rangle}}. \end{align*} $$

Let P denote the matrix $[\widetilde {\textbf {v}_{1}},\widetilde {\textbf {v}_{2}},\widetilde {\textbf {v}_{3}}]$ . Then

$$ \begin{align*} P^{\ast}HP= \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & -1 \end{bmatrix} =H_{1}. \end{align*} $$

For $i=1,2,3,4$ , the element $x_{i}$ preserves the Hermitian form H, so $y_{i}=P^{-1}x_{i}P$ preserves the Hermitian form $H_{1}$ , and, by a straightforward calculation,

$$ \begin{align*} y_i = \begin{bmatrix} a_{i} & 0 & 0 \\ 0 & b_{i} & c_{i} \\ 0 & d_{i} & e_{i} \end{bmatrix}, \end{align*} $$

where $a_{i},b_{i},c_{i},d_{i},e_{i}\in \mathbb {C}$ .

We now work in the Hermitian form $H_{1}$ . Let $\widetilde {L_{1}}=P^{-1}(L_{1})$ be the corresponding complex line. Then the polar vector of $\widetilde {L_{1}}$ is $P^{-1}(\textbf {n}_{1})=[\sqrt {\alpha },0,0]^{t}$ with respect to $H_{1}$ , and hence each point in $\widetilde {L_{1}}$ has the form $[0,z,1]^{t} \in \mathbf {H}_{\mathbb {C}}^2$ for a complex parameter z. Consider the action of $y_{i}$ on $\widetilde {L_{1}}$ , namely,

$$ \begin{align*} y_{i}: \begin{bmatrix} 0 \\ z \\ 1 \end{bmatrix} \mapsto \begin{bmatrix} 0 \\ \dfrac{b_{i}z+c_{i}}{d_{i}z+e_{i}} \\ 1 \end{bmatrix}. \end{align*} $$

If $u_{i}$ is the element in $\mathrm {SU}(1,1)$ corresponding to the action of $y_{i}$ on $\widetilde {L_{1}}$ , then

$$ \begin{align*} u_{i}=\sqrt{a_{i}} \begin{bmatrix} b_{i} & c_{i} \\ d_{i} & e_{i} \end{bmatrix}. \end{align*} $$

Let $w_i = \sigma (u_i) \in \mathrm {SL}(2,\mathbb {R})$ , where $\sigma $ is the bijection between $\mathrm {SU}(1,1)$ and $\mathrm {SL}(2,\mathbb {R})$ given by

$$ \begin{align*} \sigma: \mathrm{SU}(1,1) &\rightarrow \mathrm{SL}(2,\mathbb{R}) \\ \begin{bmatrix} A & B \\ C & D \end{bmatrix}&\mapsto \dfrac{1}{2}\begin{bmatrix} A+B+C+D & -i(A-B+C-D) \\ -i(-A-B+C+D) & A-B-C+D \end{bmatrix}. \end{align*} $$

Then we obtain a Fuchsian group $G_{11}= \langle w_{1},w_{2},w_{3},w_{4}\rangle $ , which is isomorphic to $G_{1}|_{L_{1}}$ . Set

$$ \begin{align*}Q=\{w_{i_{1}}\cdots w_{i_{r}}:r\geq1 \ \mathrm{and} \ 1\leq i_{1}<\cdots<i_{r}\leq4\}.\end{align*} $$

We call $w_{i}$ for $i=1,\ldots ,4$ the corresponding matrices and $G_{11}$ the corresponding Fuchsian group. One computes that $\mathrm {tr}(w_{i})\neq 0$ ; therefore $\mathbb {Q}(\mathrm {tr}(G_{11}^{(2)}))=\mathbb {Q}(\mathrm {tr}(G_{11}^{SQ}))$ from Proposition 2.1.

For $p=4$ , Proposition 2.2 gives $\mathbb {Q}(\mathrm {tr}(G_{11}^{(2)}))=\mathbb {Q}(\mathrm {tr}(G_{11}^{SQ})) =\mathbb {Q}(\sqrt {7})$ . Consider the isomorphism $\varphi _{1}$ from $\mathbb {Q}(\sqrt {7})$ to $\mathbb {R}$ given by $\varphi _{1}: a+b\sqrt {7} \mapsto a-b\sqrt {7}$ . A direct computation yields

$$ \begin{align*}\varphi_{1}(\mathrm{tr}(w_{2}^{2}w_{3}^{2}w_{4}^{2}))=-90+32\sqrt{7} \notin [-2,2].\end{align*} $$

By Corollary 2.6, $G_{11}$ is nonarithmetic for $p=4$ .

For $p=5$ ,

$$ \begin{align*} \mathbb{Q}(\mathrm{tr}(G_{11}^{(2)}))=\mathbb{Q}( \mathrm{tr}(G_{11}^{SQ})) =\mathbb{Q}\Big(\sqrt{5}, \sqrt{7}\times \sqrt{10-2\sqrt{5}}\Big). \end{align*} $$

Consider the isomorphism $\varphi _{2}$ from $\mathbb {Q}(\sqrt {5}, \sqrt {7}\times \sqrt {10-2\sqrt {5}}) $ to $\mathbb {R}$ given by

$$ \begin{align*} \varphi_{2}:\ & a+b\sqrt{5}+c\sqrt{7}\times\sqrt{10-2\sqrt{5}}+d\sqrt{35}\times\sqrt{10-2\sqrt{5}} \\ & \mapsto a-b\sqrt{5}-c\sqrt{7}\times\sqrt{10+2\sqrt{5}} +d\sqrt{35}\times\sqrt{10+2\sqrt{5}}. \end{align*} $$

We calculate that

$$ \begin{align*} \varphi_{2}(\mathrm{tr}(w_{1}^{2}w_{2}^{2}w_{3}^{2}w_{4}^{2})) & =\frac{839}{2}-185\sqrt{5}-\frac{535\sqrt{7}\times\sqrt{10+2\sqrt{5}}}{8}\\ &\quad +\frac{241\sqrt{35}\times\sqrt{10+2\sqrt{5}}}{8} \notin [-2,2]. \end{align*} $$

By Corollary 2.6, $G_{11}$ is nonarithmetic for $p=5$ .

Similarly, in the case when $p=6$ , $\mathbb {Q}(\mathrm {tr}(G_{11}^{(2)}))=\mathbb {Q}(\mathrm {tr}(G_{11}^{SQ})) =\mathbb {Q}(\sqrt {21})$ . Consider the isomorphism $\varphi _{3}$ from $\mathbb {Q}(\sqrt {21})$ to $\mathbb {R}$ given by $\varphi _{3}:a+b\sqrt {21} \mapsto a-b\sqrt {21}$ . A direct computation yields

$$ \begin{align*}\varphi_{3}(\mathrm{tr}(w_{1}^{2}w_{3}^{2}w_{4}^{2}))=-212+45\sqrt{21} \notin [-2,2].\end{align*} $$

By Corollary 2.6, $G_{11}$ is nonarithmetic for $p=6$ .

In the same manner, for $p=8$ , we obtain $\mathbb {Q}(\mathrm {tr}(G_{11}^{(2)}))=\mathbb {Q}(\mathrm {tr}(G_{11}^{SQ})) =\mathbb {Q}(\sqrt {2},\sqrt {7})$ . Consider the isomorphism $\varphi _{4}$ from $\mathbb {Q}(\sqrt {2},\sqrt {7})$ to $\mathbb {R}$ given by

$$ \begin{align*} \varphi_{4}:a+b\sqrt{2}+c\sqrt{7}+d\sqrt{14} \mapsto a-b\sqrt{2}-c\sqrt{7}+d\sqrt{14}. \end{align*} $$

A straightforward calculation yields

$$ \begin{align*} \varphi_{4}(\mathrm{tr}((w_{1}w_{3})^{2}))=10-4\sqrt{2}-2\sqrt{7}+3\sqrt{14}\notin [-2,2]. \end{align*} $$

By Corollary 2.6, $G_{11}$ is nonarithmetic for $p=8$ .

For the case when $p=12$ , one can deduce that $\mathbb {Q}(\mathrm {tr}(G_{11}^{(2)}))= \mathbb {Q}(\mathrm {tr}(G_{11}^{SQ})) = \mathbb {Q}(\sqrt {3},\sqrt {7})$ . Consider the isomorphism $\varphi _{5}$ from $\mathbb {Q}(\sqrt {3},\sqrt {7})$ to $\mathbb {R}$ given by

$$ \begin{align*} \varphi_{5}:a+b\sqrt{3}+c\sqrt{7}+d\sqrt{21} \mapsto a-b\sqrt{3}+c\sqrt{7}-d\sqrt{21}. \end{align*} $$

A simple calculation yields

$$ \begin{align*} \varphi_{5}(\mathrm{tr}(w_{1}^{2}w_{2}^{2}w_{3}^{2}w_{4}^{2}))=34-\frac{37\sqrt{3}}{2}+\frac{25\sqrt{7}}{2}-7\sqrt{21}\notin [-2,2]. \end{align*} $$

It follows, from Corollary 2.6, that $G_{11}$ is nonarithmetic for $p=12$ .

4 Proof of Theorem 1.4

In this section, we prove Theorem 1.4 using methods analogous to those used in the proof of Theorem 1.2. Let $R_{1}, R_{2}, R_{3}, H, \textbf {n}_{1}, u,\alpha $ and $\beta $ be defined as in Section 2.2.

$\tau =-1+i\sqrt {2}$

The $\mathbb {C}$ -Fuchsian subgroup $\Gamma _{1}$ stabilises $L_{1}$ and is generated by $g_{1},g_{2},g_{3},g_{4},g_{5}$ , where

$$ \begin{align*} g_{1}=(R_{1}R_{3}^{-1}R_{2}R_{3})^{2},& \quad g_{2} =(R_{1}R_{3})^{3}, \quad g_{3}=(R_{1}R_{2})^{3}, \quad g_{4}=(R_{1}R_{2}R_{3}R_{2}^{-1})^{2}(R_{1}R_{2})^{3},\\ g_{5} & =(R_{1}R_{2}R_{3}R_{2}R_{3}^{-1}R_{2}^{-1})^{3}(R_{1}R_{2}R_{3}R_{2}^{-1})^{2}(R_{1}R_{2})^{3}. \end{align*} $$

A fundamental domain of $\Gamma _{1}$ in $L_{1}$ is a decagon with some vertices in $L_{1}$ and others on the boundary of $L_{1}$ (see [13]). It follows that $\Gamma _{1}$ is a noncocompact lattice.

By the procedure described in Section 3, we have a corresponding Fuchsian group $\Gamma _{11}=\langle t_{1},t_{2},t_{3},t_{4},t_{5}\rangle $ that is isomorphic to $\Gamma _{1}|_{L_{1}}$ . Set

$$ \begin{align*}Q_{1}=\{t_{i_{1}}\cdots t_{i_{r}} \mid r\geq1 \ \mathrm{and} \ 1\leq i_{1}<\cdots<i_{r}\leq5\}.\end{align*} $$

(1) The cases $p=\textit{3, 4}$ . Since $\Gamma _{11}^{(2)}$ is a normal subgroup of finite index, it is a noncocompact lattice. It follows, from Proposition 2.2, that tr $(\gamma )$ is an integer polynomial in $\{\mathrm {tr}(\delta ) \mid \delta \in Q_{1}\}$ . Observing that $Q_{1}$ is a finite set, we check that the trace of each element in $Q_{1}$ is an algebraic integer. This implies that tr $(\gamma )$ is an algebraic integer for $\gamma \in \Gamma _{11}$ . Thus, the traces of elements in $\Gamma _{11}^{(2)}$ are algebraic integers.

For $i=1,\ldots ,5$ , note that tr $(t_{i})\neq 0$ and set $s_{i}=t_{i}^{2}$ and $\Gamma _{11}^{SQ}=\langle s_{1},s_{2},s_{3},s_{4},s_{5}\rangle $ . According to Propositions 2.1 and 2.2 and a direct computation, for $p=3$ ,

$$ \begin{align*} \mathbb{Q}(\mathrm{tr}(\Gamma_{11}^{(2)}))=\mathbb{Q}(\mathrm{tr}(\Gamma_{11}^{SQ}))=\mathbb{Q}(\sqrt{6})\neq\mathbb{Q}. \end{align*} $$

Similarly, in the case when $p=4$ ,

$$ \begin{align*} \mathbb{Q}(\mathrm{tr}(\Gamma_{11}^{(2)}))=\mathbb{Q}(\mathrm{tr}(\Gamma_{11}^{SQ}))=\mathbb{Q}(\sqrt{2})\neq\mathbb{Q}. \end{align*} $$

Therefore, $\Gamma _{11}^{(2)}$ is not derived from a quaternion algebra by Theorem 2.7. Thus, $\Gamma _{11}$ is nonarithmetic for $p=3,4$ by Theorem 2.3.

(2) The case $p=\textit{6}$ . Again, $\Gamma _{11}^{(2)}$ is a noncocompact lattice since $\Gamma _{11}^{(2)}$ is a normal subgroup of finite index. By Proposition 2.2, tr $(\gamma )$ is an integer polynomial in $\{\mathrm {tr}(\delta ) \mid \delta \in Q_{1}\}$ . Since $Q_{1}$ is finite, we can check that the trace of each element in $Q_{1}$ is an algebraic integer. Thus, tr $(\gamma )$ is an algebraic integer for $\gamma \in \Gamma _{11}$ and the traces of elements in $\Gamma _{11}^{(2)}$ are also algebraic integers.

For the trace field of $\Gamma _{11}^{(2)}$ , the case is a little different from the previous ones where $\mathrm {tr}(t_{2})=\mathrm {tr}(t_{3})=0$ . Consider

$$ \begin{align*}\widetilde{\Gamma_{11}}=\langle t_{1},t_{1}^{-1}t_{2},t_{1}^{-1}t_{3},t_{4},t_{5}\rangle.\end{align*} $$

In fact, $\widetilde {\Gamma _{11}}=\Gamma _{11}$ , but the traces of generators of $\widetilde {\Gamma _{11}}$ are not equal to $0$ . By a computation,

$$ \begin{align*} \mathbb{Q}(\mathrm{tr}(\Gamma_{11}^{(2)}))=\mathbb{Q}(\mathrm{tr}(\widetilde{\Gamma_{11}}^{(2)}))=\mathbb{Q}(\mathrm{tr}(\widetilde{\Gamma_{11}}^{SQ})) =\mathbb{Q}(\sqrt{6})\neq\mathbb{Q}. \end{align*} $$

Therefore, $\Gamma _{11}$ is a nonarithmetic lattice for $p=6$ .

$\tau =-({1+i\sqrt {7})}/{2}$

The $\mathbb {C}$ -Fuchsian subgroup $\Gamma _{2}$ stabilises $L_{1}$ and is generated by $g_{1},g_{2},g_{3}$ , where

$$ \begin{align*}g_{1}=(R_{1}R_{2})^{2}, \quad g_{2}=R_{2}R_{3}R_{2}^{-1}R_{1}JR_{1}J, \quad g_{3}=(R_{1}R_{3})^{2}.\end{align*} $$

By the procedure described in Section 3, we construct a corresponding Fuchsian group $\Gamma _{21}=\langle t_{1},t_{2},t_{3}\rangle $ that is isomorphic to $\Gamma _{2}|_{L_{1}}$ . Set

$$ \begin{align*}Q_{2}=\lbrace \mathrm{tr}(t_{1}),\mathrm{tr}(t_{2}),\mathrm{tr}(t_{3}),\mathrm{tr}(t_{1}t_{2}),\mathrm{tr}(t_{1}t_{3}),\mathrm{tr}(t_{2}t_{3}),\mathrm{tr}(t_{1}t_{2}t_{3})\rbrace.\end{align*} $$

A fundamental domain of $\Gamma _{2}$ in $L_{1}$ is a hexagon (see [13]).

(1) The cases $p=\textit{4, 6}$ . Since some vertices of the hexagon are on the boundary of $L_{1}$ , it follows that $\Gamma _{21}$ is a noncocompact lattice, and the same is true for $\Gamma _{21}^{(2)}$ .

It follows, from Proposition 2.2, that tr $(\gamma )$ is an integer polynomial in $Q_{2}$ and one can check that every element in $Q_{2}$ is an algebraic integer. Thus, tr $(\gamma )$ is an algebraic integer for $\gamma \in \Gamma _{21}$ and the traces of elements in $\Gamma _{21}^{(2)}$ are also algebraic integers. Note that tr $(t_{2})=0$ . We consider the group $\widetilde {\Gamma _{21}}=\langle t_{1},t_{1}^{-1}t_{2},t_{3}\rangle $ . In fact, $\widetilde {\Gamma _{21}}=\Gamma _{21}$ , but the traces of generators of $\widetilde {\Gamma _{21}}$ are not equal to $0$ . By computation, for $p=4$ ,

$$ \begin{align*} \mathbb{Q}(\mathrm{tr}(\Gamma_{21}^{(2)}))=\mathbb{Q}(\mathrm{tr}(\widetilde{\Gamma_{21}}^{(2)}))=\mathbb{Q}(\mathrm{tr}(\widetilde{\Gamma_{21}}^{SQ})) =\mathbb{Q}(\sqrt{7})\neq \mathbb{Q}.\end{align*} $$

Similarly, in the case when $p=6$ ,

$$ \begin{align*} \mathbb{Q}(\mathrm{tr}(\Gamma_{21}^{(2)}))=\mathbb{Q}(\mathrm{tr}(\widetilde{\Gamma_{21}}^{(2)}))=\mathbb{Q}(\mathrm{tr}(\widetilde{\Gamma_{21}}^{SQ})) =\mathbb{Q}(\sqrt{21})\neq \mathbb{Q}. \end{align*} $$

Consequently, $\Gamma _{21}^{(2)}$ is not derived from a quaternion algebra by Theorem 2.7. Thus, $\Gamma _{21}$ is nonarithmetic for $p=4, 6$ .

(2) The cases $p=\textit{5, 8, 12}$ . As all vertices of the hexagon lie in $L_{1}$ , it follows that $\Gamma _{21}$ is a cocompact lattice.

It follows, from Proposition 2.2, that tr $(\gamma )$ is an integer polynomial in the set $Q_{2}$ and one can check that every element in $Q_{2}$ is an algebraic integer. Hence, tr $(\gamma )$ is an algebraic integer for $\gamma \in \Gamma _{21}$ and the traces of elements in $\Gamma _{21}^{(2)}$ are also algebraic integers.

Observe that $\mathrm {tr}(t_{2})= 0$ and $\mathrm {tr}(t_{1}^{-1}t_{2})\neq 0$ . Consider $\widetilde {\Gamma _{21}}=\langle t_{1},t_{1}^{-1}t_{2},t_{3}\rangle $ . In fact, ${\Gamma _{21}=\widetilde {\Gamma _{21}}}$ . Let $s_{1}=t_{1}^{2}$ , $s_{2}=(t_{1}^{-1}t_{2})^{2}$ , $s_{3}=t_{3}^{2}$ and $\widetilde {\Gamma _{21}}^{SQ}=\langle s_{1},s_{2},s_{3}\rangle $ .

For $p=5$ ,

$$ \begin{align*} \mathbb{Q}(\mathrm{tr}(\Gamma_{21}^{(2)}))=\mathbb{Q}(\mathrm{tr}(\widetilde{\Gamma_{21}}^{(2)}))=\mathbb{Q}(\mathrm{tr}(\widetilde{\Gamma_{21}}^{SQ})) =\mathbb{Q}\Big(\sqrt{5}, \sqrt{7}\times \sqrt{10-2\sqrt{5}}\Big), \end{align*} $$

which is totally real.

Consider the isomorphism $\varphi _{2}$ from $\mathbb {Q}(\sqrt {5}, \sqrt {7}\times \sqrt {10-2\sqrt {5}})$ to $\mathbb {R}$ given by

$$ \begin{align*} \varphi_{2} & : a+b\sqrt{5}+c\sqrt{7}\times\sqrt{10-2\sqrt{5}}+d\sqrt{35}\times\sqrt{10-2\sqrt{5}} \\ & \quad \mapsto a-b\sqrt{5}-c\sqrt{7}\times\sqrt{10+2\sqrt{5}}+d\sqrt{35}\times\sqrt{10+2\sqrt{5}}. \end{align*} $$

By a direct calculation,

$$ \begin{align*} \varphi_{2}(\mathrm{tr}(s_{2}^{2}))&=497-\frac{433\sqrt{5}}{2}-\frac{627\sqrt{7}\times\sqrt{10+2\sqrt{5}}}{8}+\frac{285\sqrt{35}\times\sqrt{10+2\sqrt{5}}}{8} \\ &\notin [-2,2]. \end{align*} $$

It follows, from Corollary 2.6, that $\Gamma _{21}$ is nonarithmetic for $p=5$ .

For $p=8$ ,

$$ \begin{align*} \mathbb{Q}(\mathrm{tr}(\Gamma_{21}^{(2)}))=\mathbb{Q}(\mathrm{tr}(\widetilde{\Gamma_{21}}^{(2)}))=\mathbb{Q}(\mathrm{tr}(\widetilde{\Gamma_{21}}^{SQ})) =\mathbb{Q}(\sqrt{2}, \sqrt{7}). \end{align*} $$

Consider the isomorphism $\varphi _{4}$ from $\mathbb {Q}(\sqrt {2},\sqrt {7})$ to $\mathbb {R}$ given by

$$ \begin{align*} \varphi_{4}:a+b\sqrt{2}+c\sqrt{7}+d\sqrt{14} \mapsto a-b\sqrt{2}-c\sqrt{7}+d\sqrt{14}. \end{align*} $$

A direct computation yields

$$ \begin{align*} \varphi_{4}(\mathrm{tr}(s_{2}^{2}))=284-164\sqrt{2}-88\sqrt{7}+76\sqrt{14}\notin [-2,2]. \end{align*} $$

By Corollary 2.6, $\Gamma _{21}$ is not arithmetic for $p=8$ .

In the case when $p=12$ ,

$$ \begin{align*} \mathbb{Q}(\mathrm{tr}(\Gamma_{21}^{(2)}))=\mathbb{Q} (\mathrm{tr}(\widetilde{\Gamma_{21}}^{(2)}))=\mathbb{Q}(\mathrm{tr}(\widetilde{\Gamma_{21}}^{SQ})) =\mathbb{Q}(\sqrt{3}, \sqrt{7}). \end{align*} $$

Consider the isomorphism $\varphi _{5}$ from $\mathbb {Q}(\sqrt {3}, \sqrt {7})$ to $\mathbb {R}$ given by

$$ \begin{align*} \varphi_{5}:a+b\sqrt{3}+c\sqrt{7}+d\sqrt{21} \mapsto a-b\sqrt{3}+c\sqrt{7}-d\sqrt{21}. \end{align*} $$

One computes that

$$ \begin{align*} \varphi_{5}(\mathrm{tr}(s_{2}^{2}))=\frac{213}{2}-60\sqrt{3}+40\sqrt{7}-\frac{45\sqrt{21}}{2}\notin [-2,2]. \end{align*} $$

It follows, from Corollary 2.6, that $\Gamma _{21}$ is nonarithmetic for $p=12$ .