ON THE AUTOMORPHISMS OF THE NONSPLIT CARTAN MODULAR CURVES OF PRIME LEVEL

Abstract

We study the automorphisms of the nonsplit Cartan modular curves $X_{\text{ns}}(p)$ of prime level $p$. We prove that if $p\geqslant 29$ all the automorphisms preserve the cusps. Furthermore, if $p\equiv 1~\text{mod}~12$ and $p\neq 13$, the automorphism group is generated by the modular involution given by the normalizer of a nonsplit Cartan subgroup of $\text{GL}_{2}(\mathbb{F}_{p})$. We also prove that for every $p\geqslant 29$ the existence of an exceptional rational automorphism would give rise to an exceptional rational point on the modular curve $X_{\text{ns}}^{+}(p)$ associated to the normalizer of a nonsplit Cartan subgroup of $\text{GL}_{2}(\mathbb{F}_{p})$.

Introduction

Modular curves can be constructed as compactifications of certain quotients of the upper half complex plane. When the genus exceeds 1, it is a classical question to ask if all of their automorphisms are induced by an automorphism of the upper half complex plane. Here we make some progress toward answering this question for nonsplit Cartan modular curves.

Let $N$ be a positive integer and let $H$ be a subgroup of $\text{GL}_{2}(\mathbb{Z}/N\mathbb{Z})$ with the property that the determinant homomorphism from $H$ to $\mathbb{Z}/N\mathbb{Z}^{\ast }$ is surjective. We write $X_{H}$ for the modular curve over $\mathbb{Q}$ associated to the subgroup $H$ . Let $\unicode[STIX]{x1D6E4}_{H}$ be the subgroup of $\text{SL}_{2}(\mathbb{Z})$ made up of the elements which reduce modulo $N$ to an element in $H$ . The complex points of $X_{H}$ can be identified with the orbit space ${\mathcal{H}}\cup \mathbb{Q}\cup \{\infty \}/\unicode[STIX]{x1D6E4}_{H}$ , where ${\mathcal{H}}$ is the upper half complex plane on which any element of $\text{SL}_{2}(\mathbb{R})$ acts as a Möbius transformation $\left(\!\begin{smallmatrix}a & b\\ c & d\end{smallmatrix}\!\right)\unicode[STIX]{x1D70F}=\frac{a\unicode[STIX]{x1D70F}+b}{c\unicode[STIX]{x1D70F}+d}$ , for every $\unicode[STIX]{x1D70F}\in {\mathcal{H}}$ . Let $\text{Norm}(\unicode[STIX]{x1D6E4}_{H})$ be the normalizer of $\unicode[STIX]{x1D6E4}_{H}$ in $\text{SL}_{2}(\mathbb{R})$ . An element of $\text{Norm}(\unicode[STIX]{x1D6E4}_{H})$ induces naturally an automorphism of the curve $X_{H}$ ; hence the group $B(X_{H})\stackrel{\text{def}}{=}\text{Norm}(\unicode[STIX]{x1D6E4}_{H})/\unicode[STIX]{x1D6E4}_{H}$ can be considered as a subgroup of the automorphism group $\text{Aut}(X_{H})$ of $X_{H}$ . We call an element of $B(X_{H})$ a modular automorphism of $X_{H}$ . A nonmodular automorphism is also called exceptional.

If $H$ is a Borel subgroup of $\text{GL}_{2}(\mathbb{Z}/N\mathbb{Z})$ , the curve $X_{H}$ is the classical modular curve $X_{0}(N)$ and $\unicode[STIX]{x1D6E4}_{H}$ is the classical congruence subgroup $\unicode[STIX]{x1D6E4}_{0}(N)$ . For every positive integer $N$ , the automorphism group of $X_{0}(N)$ has been determined in [Ogg77, KM88, Elk90, Har11], and if the genus of $X_{0}(N)$ is at least 2 and $N\neq 37,63,108$ , we have $\text{Aut}(X_{0}(N))=B(X_{0}(N))$ . If we take $N=p$ a prime number, the group $B(X_{0}(p))$ is generated by the Atkin–Lehner involution $w_{p}$ . The automorphism group of $X_{0}^{\ast }(p)\stackrel{\text{def}}{=}X_{0}(p)/\langle w_{p}\rangle$ has been determined in [BH03] and it is trivial when the genus exceeds $2$ . When $X_{0}^{\ast }(p)$ has genus $2$ , the curve $X_{0}^{\ast }(p)$ is hyperelliptic, so it must admit an involution.

In this paper, we concentrate on the case of $H$ being a nonsplit Cartan subgroup of $\text{GL}_{2}(\mathbb{F}_{p})$ , and we write $X_{\text{ns}}(p)$ for the modular curve associated to $H$ . This curve always has a modular involution $w$ , because of the fact that a nonsplit Cartan subgroup of $\text{GL}_{2}(\mathbb{F}_{p})$ has index 2 in its normalizer. We have $B(X_{\text{ns}}(p))=\langle w\rangle$ (see Section 3). Let $X_{\text{ns}}^{+}(p)\stackrel{\text{def}}{=}X_{\text{ns}}(p)/\langle w\rangle$ be the modular curve associated to the normalizer of a nonsplit Cartan subgroup of $\text{GL}_{2}(\mathbb{F}_{p})$ . The curve $X_{\text{ns}}(p)$ has genus smaller than $2$ when $p\leqslant 7$ . The same happens for $X_{\text{ns}}^{+}(p)$ when $p\leqslant 11$ . It is not true that $\text{Aut}(X_{\text{ns}}(p))=\langle w\rangle$ whenever the genus is at least $2$ . For example, the automorphism group of $X_{\text{ns}}(11)$ , determined in [DFGS14], is the Klein four group generated by $w$ and an exceptional involution.

Here, extending techniques of [Ogg74, KM88, BH03], we prove the following

Theorem 1.1 .

If $p\geqslant 11$ the modular curve $X_{\text{ns}}(p)$ is not hyperelliptic. If $p\geqslant 13$ the modular curve $X_{\text{ns}}^{+}(p)$ is not hyperelliptic.

We remark that this result already follows directly from [Abr96, Theorem 0.1] for $X_{\text{ns}}(p)$ when $p\geqslant 23$ and for $X_{\text{ns}}^{+}(p)$ when $p\geqslant 29$ .

Theorem 5.2 .

If $p\geqslant 29$ , all the automorphisms of $X_{\text{ns}}(p)$ preserve the cusps. If $p\equiv 1~\text{mod}~12$ and $p\neq 13$ , the automorphism group of $X_{\text{ns}}(p)$ is generated by the modular involution $w$ .

Some of the techniques used in this paper also apply to $X_{\text{ns}}^{+}(p)$ , but the main result does not seem to be reachable right away. In general, one can expect the automorphism group of $X_{\text{ns}}(p)$ to be generated by the modular involution $w$ and the automorphism group of $X_{\text{ns}}^{+}(p)$ to be trivial for $p\geqslant 13$ .

The interest in nonsplit Cartan modular curves comes mainly from Serre’s uniformity conjecture, which is an important statement regarding Galois representations attached to elliptic curves. It is equivalent to assert that for almost all $p$ and for every maximal subgroup $H$ of $\text{GL}_{2}(\mathbb{F}_{p})$ , the modular curve $X_{H}$ has no rational points except the cusps and the points associated to elliptic curves with complex multiplication, usually called CM points. The only case left to prove, concerns modular curves associated to the normalizer of a nonsplit Cartan subgroup of $\text{GL}_{2}(\mathbb{F}_{p})$ .

We prove the following statement relating the existence of exceptional automorphisms of $X_{\text{ns}}(p)$ with the existence of non-CM rational points on $X_{\text{ns}}^{+}(p)$ .

Theorem 5.3 .

Let $p\geqslant 29$ . If there exists an exceptional automorphism of $X_{\text{ns}}(p)$ defined over $\mathbb{Q}$ , then $X_{\text{ns}}^{+}(p)$ has a non-CM rational point.

Therefore, Serre’s uniformity conjecture implies the nonexistence of rational exceptional automorphisms of $X_{\text{ns}}(p)$ for almost all  $p$ .

1 Hyperelliptic modular curves of type $X_{\text{ns}}(p)$ and $X_{\text{ns}}^{+}(p)$

Let $p$ be a prime number. We recall that the modular curve $X_{\text{ns}}(p)$ has genus $0$ for $p\leqslant 5$ and is elliptic only for $p=7$ . The modular curve $X_{\text{ns}}^{+}(p)$ has genus 0 for $p\leqslant 7$ and is elliptic only for $p=11$ [Bar10].

Theorem 1.1. If $p\geqslant 11$ the modular curve $X_{\text{ns}}(p)$ is not hyperelliptic. If $p\geqslant 13$ the modular curve $X_{\text{ns}}^{+}(p)$ is not hyperelliptic.

Proof. (Following [Ogg74, p. 455, Theorem 3]). Let $q\neq p$ be a prime number, let $\mathbb{F}_{q}$ be the finite field with $q$ elements and let $\overline{\mathbb{F}_{q}}$ be an algebraic closure. The points of $X_{\text{ns}}(p)(\overline{\mathbb{F}_{q}})$ parametrize pairs $(E,\unicode[STIX]{x1D711})$ , where $E$ is an elliptic curve over $\overline{\mathbb{F}_{q}}$ and $\unicode[STIX]{x1D711}$ is an isomorphism from the $p$ -torsion points of $E$ to $\mathbb{Z}/p\mathbb{Z}\times \mathbb{Z}/p\mathbb{Z}$ . Moreover, two pairs $(E,\unicode[STIX]{x1D711})$ , $(E^{\prime },\unicode[STIX]{x1D711}^{\prime })$ are parametrized by the same point of $X_{\text{ns}}(p)(\overline{\mathbb{F}_{q}})$ if and only if there exist an isomorphism $f$ from $E$ to $E^{\prime }$ such that, on the $p$ -torsion of $E$ , we have $\unicode[STIX]{x1D711}^{\prime }\circ f=M\circ \unicode[STIX]{x1D711}$ , where $M$ is a generator of a nonsplit Cartan subgroup $C$ of $\text{GL}_{2}(\mathbb{F}_{p})$ . The Frobenius automorphism $\unicode[STIX]{x1D70E}$ generating $\text{Gal}(\overline{\mathbb{F}_{q}}/\mathbb{F}_{q^{2}})$ acts on $X_{\text{ns}}(p)(\overline{\mathbb{F}_{q}})$ as $\unicode[STIX]{x1D70E}(E,\unicode[STIX]{x1D711})=(E^{\unicode[STIX]{x1D70E}},\unicode[STIX]{x1D711}\circ \unicode[STIX]{x1D70E}^{-1}),$ so that every elliptic curve $E$ over $\mathbb{F}_{q^{2}}$ gives a point $(E,\unicode[STIX]{x1D711})$ in $X_{\text{ns}}(p)(\mathbb{F}_{q^{2}})$ if and only if $\unicode[STIX]{x1D711}\circ \unicode[STIX]{x1D70E}^{-1}\circ \unicode[STIX]{x1D711}^{-1}$ is contained in $C$ .

Take a supersingular elliptic curve $E$ over $\overline{\mathbb{F}_{q}}$ . This means that the $q$ -torsion of $E$ is trivial, that the endomorphism algebra $\text{End}(E)$ is an order in a quaternion algebra, and implies that $E$ can be defined over $\mathbb{F}_{q^{2}}$ [Sil09, p. 144, Theorem 3.1, (a)]. The fact that multiplication by $q$ is purely inseparable implies that the $q^{2}$ th power Frobenius endomorphism $\unicode[STIX]{x1D711}_{q^{2}}$ and multiplication by $q$ differ by an automorphism of $E$ , which means that $\unicode[STIX]{x1D711}_{q^{2}}=\unicode[STIX]{x1D701}q$ in $\text{End}(E)$ and $\unicode[STIX]{x1D701}$ is a root of unity. If $j\neq 0,1728$ , we have $\unicode[STIX]{x1D711}_{q^{2}}=\pm q$ . If $j=0,1728$ , the elliptic curve $E$ is isomorphic over $\overline{\mathbb{F}_{q}}$ to a curve $E^{\prime }$ defined over $\mathbb{F}_{q}$ with equation $y^{2}+y=x^{3}$ or $y^{2}=x^{3}-x$ . We have $\#E^{\prime }(\mathbb{F}_{q})=q+1$ ([Sil09, p. 154, Exercise 5.10, (b)] when $q\geqslant 5$ , a direct count when $q=2,3$ ). Hence the $q$ th power Frobenius endomorphism of $E^{\prime }$ has characteristic polynomial $x^{2}+q=0$ which implies that $\unicode[STIX]{x1D711}_{q^{2}}=-q$ in $\text{End}(E)$ .

This shows that every supersingular elliptic curve over $\overline{\mathbb{F}_{q}}$ is isomorphic to a supersingular elliptic curve $E$ defined over $\mathbb{F}_{q^{2}}$ , with the property that $\unicode[STIX]{x1D711}_{q^{2}}$ acts on the $p$ -torsion points as multiplication by $q$ or $-q$ . Which says that $\unicode[STIX]{x1D711}_{q^{2}}$ acts on the $p$ -torsion of $E$ as a scalar matrix of $\text{GL}_{2}(\mathbb{F}_{p})$ and therefore as an element contained in every conjugate of the nonsplit Cartan subgroup $C$ in $\text{GL}_{2}(\mathbb{F}_{p})$ . Hence, $E$ gives a point in $X_{\text{ns}}(p)(\mathbb{F}_{q^{2}})$ for each conjugate of $C$ in $\text{GL}_{2}(\mathbb{F}_{p})$ , up to automorphisms of $E$ , in the sense that a pair $(E,\unicode[STIX]{x1D711})$ gives the same point on $X_{\text{ns}}(p)$ as $(E,\unicode[STIX]{x1D711}\circ f)$ for any automorphism $f$ of $E$ . Among the automorphisms of any supersingular elliptic curve, we have multiplication by $-1$ , which acts on torsion points as an element contained in $C$ . Thus, since the index of a nonsplit Cartan subgroup in $\text{GL}_{2}(\mathbb{F}_{p})$ is equal to $p(p-1)$ , we have the following inequality

$$\begin{eqnarray}\#X_{\text{ns}}(p)(\mathbb{F}_{q^{2}})\geqslant p(p-1)\cdot 2\cdot \mathop{\sum }_{\substack{ E\;\text{over}\;\overline{\mathbb{F}_{q}} \\ \text{supersingular}}}\frac{1}{\#\text{Aut}(E)}=\frac{p(p-1)(q-1)}{12},\end{eqnarray}$$

where in the last equality we used the Deuring–Eichler formula [Sil09, p. 154, Exercise 5.9]

$$\begin{eqnarray}\mathop{\sum }_{\substack{ E\;\text{over}\;\overline{\mathbb{F}_{q}} \\ \text{supersingular}}}\frac{1}{\#\text{Aut}(E)}=\frac{q-1}{24}.\end{eqnarray}$$

Note that this lower bound on the number of supersingular points can also easily be deduced from [BGJGP05, Lemmas 3.20 and 3.21].

Suppose now that $X_{\text{ns}}(p)$ is hyperelliptic. Then we have

$$\begin{eqnarray}\frac{p(p-1)(q-1)}{12}\leqslant \#X_{\text{ns}}(p)(\mathbb{F}_{q^{2}})\leqslant 2(q^{2}+1),\end{eqnarray}$$

where the second inequality holds because $X_{\text{ns}}(p)$ is hyperelliptic and because the projective line over $\mathbb{F}_{q^{2}}$ has exactly $q^{2}+1$ points. Now, putting $q=2$ we have

$$\begin{eqnarray}\frac{p(p-1)}{12}\leqslant \#X_{\text{ns}}(p)(\mathbb{F}_{4})\leqslant 10\end{eqnarray}$$

and we obtain $p\leqslant 11$ which ends the proof for $X_{\text{ns}}(p)$ in the case $p\geqslant 13$ . With an analogous reasoning, taking into account that the index of the normalizer of a nonsplit Cartan subgroup in $\text{GL}_{2}(\mathbb{F}_{p})$ is equal to $\frac{p(p-1)}{2}$ , we obtain the statement for $X_{\text{ns}}^{+}(p)$ in the case $p\geqslant 17$ .

The case $p=11$ for $X_{\text{ns}}(p)$ and the case $p=13$ for $X_{\text{ns}}^{+}(p)$ can be worked out by explicitly counting the points over $\mathbb{F}_{4}$ of the respective curves. Recall that the Jacobian of $X_{\text{ns}}(p)$ is isogenous to the new part of the Jacobian of $X_{0}(p^{2})$ and the Jacobian of $X_{\text{ns}}^{+}(p)$ is isogenous to the new part of the Jacobian of $X_{0}^{\ast }(p^{2})\stackrel{\text{def}}{=}X_{0}(p^{2})/\langle w_{p^{2}}\rangle$ , where $w_{p^{2}}$ is the Atkin–Lehner involution [Che98, dSE00]. Then we can count the points over $\mathbb{F}_{4}$ by looking at the roots of the characteristic polynomial of a Frobenius at $2$ on the Jacobian of $X_{0}(p^{2})$ , and using the well- known formula

$$\begin{eqnarray}\#C(\mathbb{F}_{q^{r}})=q^{r}+1-\mathop{\sum }_{i=1}^{2g}\unicode[STIX]{x1D6FC}_{i}^{r}\end{eqnarray}$$

where $C$ is a nonsingular projective curve of genus $g$ over the finite field $\mathbb{F}_{q}$ with $q$ prime, and the $\unicode[STIX]{x1D6FC}_{i}$ ’s are the roots of the characteristic polynomial of the Frobenius automorphism at $q$ , acting on the Jacobian variety of $C$ .

Looking at the tables in [Ste12] of weight-2 newforms for $\unicode[STIX]{x1D6E4}_{0}(121)$ , we see that the Jacobian of $X_{\text{ns}}(11)$ is isogenous to the product of four elliptic curves for which the traces of a Frobenius automorphism at 2 are respectively $-1$ , $0$ , $1$ , $2$ , so that the roots of the characteristic polynomial of a Frobenius at 2 acting on the Jacobian of $X_{\text{ns}}(11)$ are $\frac{-1\pm \sqrt{-7}}{2}$ , $\pm \sqrt{-2}$ , $\frac{1\pm \sqrt{-7}}{2}$ , $1\pm i$ . Hence $\#X_{\text{ns}}(11)(\mathbb{F}_{4})=15>10$ .

Now, in the same tables, we look at the weight-2 newforms for $\unicode[STIX]{x1D6E4}_{0}(169)$ . We see that the Jacobian of $X_{\text{ns}}^{+}(13)$ is isogenous to a simple abelian variety $A$ over $\mathbb{Q}$ of dimension $3$ . The eigenvalues of the Hecke operator $T_{2}$ acting on $A$ are the roots $a_{i}$ , with $i=1,2,3$ , of the polynomial $x^{3}+2x^{2}-x-1$ . Then the characteristic polynomial of a Frobenius at 2 is the product

$$\begin{eqnarray}\mathop{\prod }_{i=1}^{3}(x^{2}-a_{i}x+2).\end{eqnarray}$$

It allows us to compute $\#X_{\text{ns}}^{+}(13)(\mathbb{F}_{4})=11>10$ .◻

2 Endomorphisms of the Jacobian variety of $X_{\text{ns}}(p)$

In this section, for any abelian variety $A$ over a field $K$ , an endomorphism of $A$ is an endomorphism defined over an algebraic closure $\overline{K}$ of $K$ , and we write $\text{End}(A)$ for the ring of endomorphisms of $A$ defined over $\overline{K}$ . The endomorphism algebra $\text{End}(A)\otimes \mathbb{Q}$ is an invariant of the isogeny class of $A$ over $\overline{K}$ . The minimal field where every endomorphism of $A$ is defined is an invariant of the isogeny class over $K$ of $A$ . We say that $A$ is of CM-type if $\text{End}(A)\otimes \mathbb{Q}$ contains a commutative semisimple $\mathbb{Q}$ -algebra of dimension equal to $2\cdot \dim A$ .

Proposition 2.1. Let $A$ be a semistable abelian variety over a field $K$ , complete with respect to a discrete valuation. Then every endomorphism of $A$ is defined over an unramified extension of $K$ .

Proof. [Rib75, p. 556, Theorem 1.1]. ◻

Proposition 2.2. Let $A$ be an abelian variety over a field $K$ . Let $E$ be a subalgebra of $\text{End}_{K}(A)\otimes \mathbb{Q}$ such that $[E:\mathbb{Q}]=\dim A$ and suppose that $E$ is a product $E_{1}\times \cdots \times E_{t}$ of totally real number fields. This gives rise to a decomposition $A\sim A_{1}\times \cdots \times A_{t}$ of abelian varieties up to isogeny over $K$ . Suppose that no factor $A_{i}$ is of CM-type. Then $E$ is its own commutant in $\text{End}(A)\otimes \mathbb{Q}$ and every endomorphism of $A$ is defined over a compositum of quadratic extensions of $K$ .

Proof. [Rib75, p. 557, Theorem 2.3] ◻

Let $J_{0}(N)$ be the Jacobian variety of the modular curve $X_{0}(N)$ . It is an abelian variety over $\mathbb{Q}$ .

Proposition 2.3. The abelian variety $J_{0}(N)$ is semistable at each prime $p$ such that $p^{2}$ does not divide $N$ .

Proof. [DR73, p. 286, Theorem 6.9]. ◻

For every integer $N>0$ , there exists a set of representatives $f_{1},\ldots ,f_{t}$ of $\text{Gal}(\overline{\mathbb{Q}}/\mathbb{Q})$ -conjugacy classes of normalized weight-2 cuspforms for $\unicode[STIX]{x1D6E4}_{0}(N)$ , such that every $f_{i}$ is a newform at some level $M_{i}$ dividing $N$ , and $\{f_{i}(n\unicode[STIX]{x1D70F}),n|N/M_{i},i=1\ldots t\}$ with their $\text{Gal}(\overline{\mathbb{Q}}/\mathbb{Q})$ -conjugates form a basis of the complex vector space of weight-2 cuspforms for $\unicode[STIX]{x1D6E4}_{0}(N)$ . There is a decomposition of abelian varieties up to isogeny over $\mathbb{Q}$

(2.1) $$\begin{eqnarray}J_{0}(N)\sim A_{1}^{m_{i}}\times \cdots \times A_{t}^{m_{t}}\end{eqnarray}$$

where $A_{i}\stackrel{\text{def}}{=}A_{f_{i}}$ is the abelian variety up to isogeny over $\mathbb{Q}$ associated to the $\text{Gal}(\overline{\mathbb{Q}}/\mathbb{Q})$ -conjugacy class of $f_{i}$ , and $m_{i}$ is the number of divisors of $N/M_{i}$ (see [DS05, §6.6]). Every $A_{i}$ is a simple abelian variety over $\mathbb{Q}$ .

Let $J_{0}^{C}(N)$ be the abelian variety up to isogeny over $\mathbb{Q}$ which is a product of the factors $A_{i}^{m_{i}}$ such that $A_{i}$ is of CM-type and let $J_{0}^{H}(N)$ be the product of the remaining factors of the decomposition above. We recall here that if $A_{i}$ is of CM-type, then it is isogenous (over $\overline{\mathbb{Q}}$ ) to a product of copies of an elliptic curve with complex multiplication by an imaginary quadratic field $K$ of discriminant $-D$ and $f_{i}$ is a weight-2 cuspform for $\unicode[STIX]{x1D6E4}_{0}(D\cdot \text{N}(\mathfrak{c}))$ , where $\mathfrak{c}$ is the conductor of a primitive Hecke character of $K$ (see [Shi72, p. 138, Proposition 1.6], [Shi71]). Any factor $A_{i}$ of $J_{0}^{H}(N)$ cannot be isogenous to any factor $A_{j}$ of $J_{0}^{C}(N)$ because $A_{i}$ and $A_{j}$ have different endomorphism algebras. Hence, we have $J_{0}(N)\sim J_{0}^{H}(N)\times J_{0}^{C}(N)$ and $\text{End}(J_{0}(N))\otimes \mathbb{Q}\cong (\text{End}(J_{0}^{H}(N))\otimes \mathbb{Q})\times (\text{End}(J_{0}^{C}(N))\otimes \mathbb{Q})$ .

Let $p$ be an odd prime number. We call $K(p)=\mathbb{Q}(\sqrt{p})$ if $p\equiv 1~\text{mod}~4$ and $K(p)=\mathbb{Q}(\sqrt{-p})$ if $p\equiv 3~\text{mod}~4$ .

Proposition 2.4. Every endomorphism of $J_{0}^{H}(p^{2})$ is defined over $K(p)$ .

Proof. Proposition 2.2 tells us that endomorphisms of $J_{0}^{H}(p^{2})$ can be defined over a field $K$ which is a compositum of quadratic extensions of $\mathbb{Q}$ . Furthermore, Propositions 2.1 and 2.3 imply that $K$ can be taken unramified outside $p$ . Thus, we can take $K=\mathbb{Q}(\sqrt{p})$ if $p\equiv 1~\text{mod}~4$ and $K=\mathbb{Q}(\sqrt{-p})$ if $p\equiv 3~\text{mod}~4$ .◻

Let $g_{0}(N)$ , $g_{0}^{H}(N)$ , $g_{0}^{C}(N)$ be respectively the dimensions of $J_{0}(N)$ , $J_{0}^{H}(N)$ , $J_{0}^{C}(N)$ .

Let $J_{\text{ns}}(p)$ be the Jacobian of the modular curve $X_{\text{ns}}(p)$ . The abelian variety $J_{\text{ns}}(p)$ is isogenous over $\mathbb{Q}$ to the new part of $J_{0}(p^{2})$ (see [Che98, dSE00]). Let $J_{0}^{\text{new}}(p^{2})$ and $J_{0}^{\text{old}}(p^{2})$ be respectively the new and the old part of $J_{0}(p^{2})$ and let us fix an isomorphism $\unicode[STIX]{x1D711}$ from $\text{End}(J_{\text{ns}}(p))\otimes \mathbb{Q}$ to $\text{End}(J_{0}^{\text{new}}(p^{2}))\otimes \mathbb{Q}$ . We say, with abuse of notation, that an automorphism $v$ of $X_{\text{ns}}(p)$ , induces the invertible element $v\stackrel{\text{def}}{=}\unicode[STIX]{x1D711}(v)$ of

$$\begin{eqnarray}\displaystyle \text{End}(J_{0}^{\text{new}}(p^{2}))\otimes \mathbb{Q} & \subset & \displaystyle (\text{End}(J_{0}^{\text{new}}(p^{2}))\otimes \mathbb{Q})\times (\text{End}(J_{0}^{\text{old}}(p^{2}))\otimes \mathbb{Q})\nonumber\\ \displaystyle & \subset & \displaystyle \text{End}(J_{0}(p^{2}))\otimes \mathbb{Q}\nonumber\end{eqnarray}$$

where the first inclusion is given by sending $v$ to the invertible element that acts like $v$ on the new part and as the identity on the old part.

We can then define in an analogous way $J_{\text{ns}}^{H}(p)$ , $J_{\text{ns}}^{C}(p)$ , as factors of the new part of the decomposition (2.1). We also define the dimensions $g_{\text{ns}}(p)$ , $g_{\text{ns}}^{H}(p)$ , $g_{\text{ns}}^{C}(p)$ . We have $J_{\text{ns}}(p)\sim J_{\text{ns}}^{H}(p)\times J_{\text{ns}}^{C}(p)$ and $\text{End}(J_{\text{ns}}(p))\otimes \mathbb{Q}\cong (\text{End}(J_{\text{ns}}^{H}(p))\otimes \mathbb{Q})\times (\text{End}(J_{\text{ns}}^{C}(p))\otimes \mathbb{Q})$ .

Theorem 2.5. If $g_{\text{ns}}(p)>p$ every automorphism of $X_{\text{ns}}(p)$ is defined over $K(p)$ .

Proof. (Following [KM88, p. 55, Lemma 1.4]) Let $u$ be an automorphism of $X_{\text{ns}}(p)$ and $\unicode[STIX]{x1D70E}$ a element of $\text{Gal}(\overline{\mathbb{Q}}/K(p))$ . We define $v=u^{\unicode[STIX]{x1D70E}}u^{-1}$ and let $d$ be the order of $v$ in the automorphism group of $X_{\text{ns}}(p)$ . We call $Y$ the quotient of $X_{\text{ns}}(p)$ by the automorphism $v$ . Let $g_{Y}$ be the genus of $Y$ . We have that $v$ induces an automorphism of the Jacobian $J_{\text{ns}}(p)\sim J_{\text{ns}}^{H}(p)\times J_{\text{ns}}^{C}(p)$ which acts trivially on $J_{\text{ns}}^{H}(p)$ by Proposition 2.4, because $J_{\text{ns}}^{H}(p)$ is a factor of $J_{0}^{H}(p^{2})$ . Thus we have the following morphisms of abelian varieties with finite kernel

$$\begin{eqnarray}\displaystyle J_{\text{ns}}^{H}(p) & \rightarrow & \displaystyle J_{\text{ns}}^{H}(p)\times J_{\text{ns}}^{C}(p)\rightarrow J_{\text{ns}}(p)\nonumber\\ \displaystyle x & \mapsto & \displaystyle (x,0)\nonumber\end{eqnarray}$$

where the image of the composition is contained in $J_{\text{ns}}(p)^{v}\stackrel{\text{def}}{=}\{D\in J_{\text{ns}}(p):vD=D\}$ . Hence there are morphisms of group varieties

$$\begin{eqnarray}J_{\text{ns}}^{H}(p)\longrightarrow J_{\text{ns}}(p)^{v}\longrightarrow \text{Jac}(Y)\end{eqnarray}$$

where the second map is the push-forward of divisors, and the composition is a morphism of abelian varieties. Furthermore, both maps have finite kernel, hence $g_{Y}\geqslant g_{\text{ns}}^{H}(p)$ and we have by the Riemann–Hurwitz formula

$$\begin{eqnarray}g_{\text{ns}}(p)-1\geqslant d(g_{Y}-1)\geqslant 2g_{\text{ns}}^{H}(p)-2,\end{eqnarray}$$

where in the second inequality we supposed $d>1$ . Now using $g_{\text{ns}}(p)=g_{\text{ns}}^{H}(p)+g_{\text{ns}}^{C}(p)$ we obtain $g_{\text{ns}}(p)\leqslant 2g_{\text{ns}}^{C}(p)+1$ . Since $J_{\text{ns}}(p)$ is isogenous over $\mathbb{Q}$ to the new part of $J_{0}(p^{2})$ we have $g_{\text{ns}}^{C}(p)=g_{0}^{C}(p^{2})$ . Indeed, the old part of $J_{0}(p^{2})$ is isogenous to the product of two copies of the Jacobian of $J_{0}(p)$ and these cannot have abelian subvarieties of CM-type (see [Shi72, p. 140, Remark 1.7]). Furthermore, $g_{0}^{C}(p^{2})=0$ if $p\equiv 1~\text{mod}~4$ and $g_{0}^{C}(p^{2})=h(-p)$ if $p\equiv 3~\text{mod}~4$ , where $h(-p)$ is the class number of $\mathbb{Q}(\sqrt{-p})$ . Indeed, in the decomposition (2.1) with $N=p^{2}$ , if a factor is of CM-type, then it is the product of copies of an elliptic curve with complex multiplication by an imaginary quadratic field $K$ of discriminant $-p$ . This leaves no possibilities if $p\equiv 1~\text{mod}~4$ and it implies $K=\mathbb{Q}(\sqrt{-p})$ when $p\equiv 3~\text{mod}~4$ . In the latter case the dimension is $h(-p)$ because by the theory of complex multiplication, the elliptic curves with complex multiplication by $\mathbb{Q}(\sqrt{-p})$ are all isogenous, $\text{Gal}(\overline{\mathbb{Q}}/\mathbb{Q})$ -conjugate and their cardinality is $h(-p)$ .

Then we can estimate $h(-p)$ using Dirichlet’s class number formula:

$$\begin{eqnarray}h(-p)=-\frac{1}{p}\mathop{\sum }_{m=1}^{p-1}m\left(\frac{m}{p}\right)\leqslant \frac{1}{p}\mathop{\sum }_{m=1}^{p-1}m=\frac{p-1}{2}\end{eqnarray}$$

where the first equality is [Dav80, p. 51, (19)] (we are assuming $p\equiv 3~\text{mod}~4$ and $p\neq 3$ ), and in the inequality we used that the Legendre symbol $\left(\frac{m}{p}\right)$ takes values in $\{\pm 1\}$ . This implies $g_{\text{ns}}(p)\leqslant p$ . Contradiction.◻

Corollary 2.6. If $p\geqslant 11$ every automorphism of $X_{\text{ns}}(p)$ is defined over $K(p)$ .

Proof. By [Bar10, Theorem 7.2], we have that $g_{\text{ns}}(p)>p$ for $p\geqslant 19$ . Regarding $p=13$ , $17$ , they are both congruent to $1$ modulo $4$ , so in these cases $g_{\text{ns}}^{C}(p)=g_{0}^{C}(p^{2})=0$ and we can use Proposition 2.4 together with the isogeny over $\mathbb{Q}$ between $J_{\text{ns}}(p)$ and the new part of $J_{0}(p^{2})$ . The automorphisms of $X_{\text{ns}}(11)$ are explicitly computed in [DFGS14] and they are all defined over $\mathbb{Q}$ .◻

The modular curve $X_{\text{ns}}(7)$ has genus $1$ , and it has the automorphisms defined as translation by points. Thus, in this case there are automorphisms defined over larger number fields.

Proposition 2.7. If an automorphism of $X_{\text{ns}}(p)$ is defined over $\mathbb{Q}$ , then it induces an element of $\text{End}(J_{0}^{H}(p^{2}))\otimes \mathbb{Q}$ contained in the Hecke algebra.

Proof. Since $J_{\text{ns}}(p)$ is isogenous over $\mathbb{Q}$ to the new part of $J_{0}(p^{2})$ , an automorphism $u$ of $X_{\text{ns}}(p)$ defined over $\mathbb{Q}$ induces an element of $\text{End}_{\mathbb{Q}}(J_{0}(p^{2}))\otimes \mathbb{Q}\cong (\text{End}_{\mathbb{Q}}(J_{\text{ns}}^{H}(p))\otimes \mathbb{Q})\times (\text{End}_{\mathbb{Q}}(J_{\text{ns}}^{C}(p))\otimes \mathbb{Q})$ , as explained above. Since $J_{0}(p^{2})$ has good reduction outside $p$ , the reduction modulo $l$ map is injective on the endomorphism ring of $J_{0}(p^{2})$ for every prime number $l\neq p$ . Hence, by the Eichler–Shimura relations, $u$ commutes with every Hecke operator $T_{l}$ with $l\neq p$ , because $u$ is defined over $\mathbb{Q}$ . Moreover, the Hecke algebra acting on the new part is generated by the operators $T_{l}$ with $l\neq p$ , thus $u$ commutes with the whole Hecke algebra. Then, to prove the Proposition, we note that every factor of $J_{0}^{H}(p^{2})$ is without complex multiplication and so the Hecke algebra is its own commutant in $\text{End}(J_{0}^{H}(p^{2}))\otimes \mathbb{Q}$ (Proposition 2.2).◻

Lemma 2.8. If $g_{\text{ns}}(p)>p$ every automorphism of $X_{\text{ns}}(p)$ defined over $\mathbb{Q}$ commutes with the modular involution $w$ .

Proof. Let $u$ be an automorphism of $X_{\text{ns}}(p)$ defined over $\mathbb{Q}$ . Then, by the previous Proposition, $u$ and $w$ induce elements in the Hecke algebra of $J_{0}^{H}(p^{2})$ , which is a commutative algebra. Hence, if we define $v=uwu^{-1}w^{-1}$ we have that $v$ induces an invertible element of $\text{End}(J_{0}(p^{2}))\otimes \mathbb{Q}$ which acts trivially on $J_{0}^{H}(p^{2})$ and $J_{\text{ns}}^{H}(p)$ . Then we can use the same proof as for Theorem 2.5 to see that $v$ is trivial if $g_{\text{ns}}(p)>p$ .◻

Corollary 2.9. If $p\geqslant 11$ every automorphism of $X_{\text{ns}}(p)$ defined over $\mathbb{Q}$ commutes with the modular involution $w$ .

Proof. As for Corollary 2.6, we have that $g_{\text{ns}}(p)>p$ for $p\geqslant 19$ . Regarding $p=13$ , $17$ , in these cases $g_{\text{ns}}^{C}(p)=g_{0}^{C}(p^{2})=0$ . Moreover, $g_{\text{ns}}(p)\geqslant 2$ , so the natural map from the automorphism group of $X_{\text{ns}}(p)$ to the automorphism group of $J_{\text{ns}}(p)$ is injective [BH03, Lemma 2.1], and we can use Proposition 2.7 together with the fact that the Hecke algebra is commutative. The automorphism group of $X_{\text{ns}}(11)$ is the Klein four group (see [DFGS14]).◻

Corollary 2.10. If $p\geqslant 11$ every automorphism of $X_{\text{ns}}(p)$ defined over $\mathbb{Q}$ induces an automorphism of $X_{\text{ns}}^{+}(p)$ defined over $\mathbb{Q}$ .

3 Automorphisms of $X_{\text{ns}}(p)$ preserving the cusps

Let $X_{H}$ be a modular curve. Then the orbit space ${\mathcal{H}}\cup \mathbb{Q}\cup \{\infty \}/\unicode[STIX]{x1D6E4}_{H}$ can be identified with $X_{H}(\mathbb{C})$ . Let $E\subset {\mathcal{H}}$ be the set of ramification points of the natural quotient map $\unicode[STIX]{x1D70B}:{\mathcal{H}}\rightarrow {\mathcal{H}}/\unicode[STIX]{x1D6E4}_{H}$ . This is a discrete set in ${\mathcal{H}}$ and it is the counter image of the set of elliptic points in $X_{H}(\mathbb{C})$ . For every $z\in {\mathcal{H}}\smallsetminus E$ there exists a neighborhood $U\ni z$ such that $\unicode[STIX]{x1D6FE}_{i}(U)\cap \unicode[STIX]{x1D6FE}_{j}(U)=\emptyset$ for every $\unicode[STIX]{x1D6FE}_{i}\neq \pm \unicode[STIX]{x1D6FE}_{j}$ in $\unicode[STIX]{x1D6E4}_{H}$ . This means that ${\mathcal{H}}\smallsetminus E$ is a proper covering space of $X_{H}(\mathbb{C})$ without the cusps and the elliptic points.

Proposition 3.1. Let $u$ be an automorphism of $X_{H}$ . If $u$ preserves the set of cusps and preserves the set of elliptic points of $X_{H}$ , then $u$ is induced by an automorphism of the Riemann surface ${\mathcal{H}}$ preserving $E$ .

Proof. Let $C$ be the set of cusps on $X_{H}$ . Note that an automorphism $u$ of $X_{H}$ preserving the cusps and the elliptic points, defines naturally an automorphism of the open Riemann surface $X_{H}(\mathbb{C})\smallsetminus (C\cup \unicode[STIX]{x1D70B}(E))$ . If $X_{H}$ has no elliptic points ( $E=\emptyset$ ), then the quotient ${\mathcal{H}}\rightarrow {\mathcal{H}}/\unicode[STIX]{x1D6E4}_{H}\cong X_{H}(\mathbb{C})\smallsetminus C$ is the universal cover of $X_{H}(\mathbb{C})\smallsetminus C$ . Then $u$ automatically lifts to an automorphism of ${\mathcal{H}}$ . If instead $E\neq \emptyset$ , to lift $u$ to ${\mathcal{H}}\smallsetminus E$ , we need the push-forward by $u\circ \unicode[STIX]{x1D70B}$ of the fundamental group of ${\mathcal{H}}\smallsetminus E$ to be contained in its push-forward by $\unicode[STIX]{x1D70B}$ . But this is true because $u$ actually extends to an automorphism of $X_{H}(\mathbb{C})$ preserving $C$ and $\unicode[STIX]{x1D70B}(E)$ . Hence $u$ lifts to an automorphism of ${\mathcal{H}}\smallsetminus E$ , which extends uniquely to ${\mathcal{H}}$ .◻

Let $p$ be a prime number.

Proposition 3.2. Suppose that the genus of $X_{\text{ns}}(p)$ is at least 2. Let $\unicode[STIX]{x1D6E4}_{\text{ns}}(p)$ be the subgroup of $\text{SL}_{2}(\mathbb{Z})$ made up of the elements which reduce modulo $p$ to an element in a nonsplit Cartan subgroup of $\text{GL}_{2}(\mathbb{F}_{p})$ . Then the subgroup

$$\begin{eqnarray}B(X_{\text{ns}}(p))=\text{Norm}(\unicode[STIX]{x1D6E4}_{\text{ns}}(p))/\unicode[STIX]{x1D6E4}_{\text{ns}}(p)\subset \text{Aut}(X_{\text{ns}}(p))\end{eqnarray}$$

is generated by the modular involution $w$ .

Proof. Since the genus of $X_{\text{ns}}(p)$ is at least 2, the group $B(X_{\text{ns}}(p))$ must be finite. This means that $\text{Norm}(\unicode[STIX]{x1D6E4}_{\text{ns}}(p))$ is commensurable with $\text{SL}_{2}(\mathbb{Z})$ in the sense that $\text{Norm}(\unicode[STIX]{x1D6E4}_{\text{ns}}(p))\cap \text{SL}_{2}(\mathbb{Z})$ has finite index in both $\text{Norm}(\unicode[STIX]{x1D6E4}_{\text{ns}}(p))$ and $\text{SL}_{2}(\mathbb{Z})$ . Hence $\text{Norm}(\unicode[STIX]{x1D6E4}_{\text{ns}}(p))$ is a group acting on lattices of $\mathbb{R}^{2}$ that are commensurable with $\mathbb{Z}\times \mathbb{Z}$ , with the action given by right multiplication of row vectors in the lattice. Lattices commensurable with $\mathbb{Z}\times \mathbb{Z}$ are the ones generated by two vectors $(a,b)$ , $(c,d)$ with $a,b,c,d\in \mathbb{Q}$ , and if we consider them up to multiplication by a scalar (i.e., homothety), they all have a basis of the form $\{(M,\frac{g}{h}),(0,1)\}$ with $M\in \mathbb{Q}^{+}$ , $g,h\in \mathbb{Z}$ , $\text{gcd}(g,h)=1$ and $0\leqslant g<h$ (see [Con96] for more details).

Since $\unicode[STIX]{x1D6E4}_{\text{ns}}(p)\subset \unicode[STIX]{x1D6E4}(p)=\left\{\unicode[STIX]{x1D6FE}\in \text{SL}_{2}(\mathbb{Z})\text{ s. t. }\unicode[STIX]{x1D6FE}\equiv \text{Id mod }p\right\}$ , the lattices fixed by $\unicode[STIX]{x1D6E4}_{\text{ns}}(p)$ are a subset of the lattices fixed by $\unicode[STIX]{x1D6E4}(p)$ . The lattices fixed by $\unicode[STIX]{x1D6E4}(p)$ are those who belong to one of the following type (see [Lan02, Lemma 4.1])

• ∙ $M=\frac{1}{p}$ , $h=p$ , $g=0,\ldots ,p-1$ , which gives a lattice homothetic to $\langle (1,g),(0,p)\rangle$ ;

• ∙ $M=\frac{1}{p}$ , $h=1$ , $g=0$ , which gives a lattice homothetic to $\langle (1,0),(0,p)\rangle$ ;

• ∙ $M=p$ , $h=1$ , $g=0$ , which gives the lattice $\langle (p,0),(0,1)\rangle$ ;

• ∙ $M=1$ , $h=1$ , $g=0$ , which gives the lattice $\langle (1,0),(0,1)\rangle =\mathbb{Z}\times \mathbb{Z}$ .

Recall that $\unicode[STIX]{x1D6E4}_{\text{ns}}(p)$ is the subgroup of $\text{SL}_{2}(\mathbb{Z})$ made up of the elements $\unicode[STIX]{x1D6FE}$ such that

$$\begin{eqnarray}\unicode[STIX]{x1D6FE}\equiv \left(\begin{array}{@{}cc@{}}x & \unicode[STIX]{x1D6FC}y\\ y & x\end{array}\right)~\text{mod}~p\end{eqnarray}$$

for some $(x,y)\in \mathbb{F}_{p}^{2}$ and a nonsquare element $\unicode[STIX]{x1D6FC}\in \mathbb{F}_{p}$ . Then, a straightforward computation modulo $p$ shows that $\unicode[STIX]{x1D6FE}$ fixes lattices of the first three types, only if $\unicode[STIX]{x1D6FE}\in \unicode[STIX]{x1D6E4}(p)$ , so that the only lattice up to homothety fixed by $\unicode[STIX]{x1D6E4}_{\text{ns}}(p)$ is $\mathbb{Z}\times \mathbb{Z}$ .

The normalizer in $\text{SL}_{2}(\mathbb{R})$ of $\unicode[STIX]{x1D6E4}_{\text{ns}}(p)$ must preserve, by definition, the set of lattices that are fixed by $\unicode[STIX]{x1D6E4}_{\text{ns}}(p)$ . Thus, also $\text{Norm}(\unicode[STIX]{x1D6E4}_{\text{ns}}(p))$ fixes $\mathbb{Z}\times \mathbb{Z}$ . The stabilizer of $\mathbb{Z}\times \mathbb{Z}$ is $\text{SL}_{2}(\mathbb{Z})$ , implying $\text{Norm}(\unicode[STIX]{x1D6E4}_{\text{ns}}(p))\subset \text{SL}_{2}(\mathbb{Z})$ . Therefore, $\text{Norm}(\unicode[STIX]{x1D6E4}_{\text{ns}}(p))$ is made up of the elements of $\text{SL}_{2}(\mathbb{Z})$ which reduce modulo $p$ to an element in the normalizer of a nonsplit Cartan subgroup of $\text{GL}_{2}(\mathbb{F}_{p})$ , which proves the Proposition.◻

Corollary 3.3. Suppose that the genus of $X_{\text{ns}}(p)$ is at least 2. If $p\equiv 1~\text{mod}~12$ , the only nontrivial automorphism of $X_{\text{ns}}(p)$ preserving the cusps is the modular involution $w$ . If $p\not \equiv 1~\text{mod}~12$ , the only nontrivial automorphism of $X_{\text{ns}}(p)$ preserving the cusps and the elliptic points, is the modular involution $w$ . The only automorphism of $X_{\text{ns}}^{+}(p)$ preserving the cusps and the elliptic points is the identity.

Proof. Let $\unicode[STIX]{x1D6E4}_{\text{ns}}(p)$ be the subgroup of $\text{SL}_{2}(\mathbb{Z})$ made up of the elements which reduce modulo $p$ to an element in a nonsplit Cartan subgroup of $\text{GL}_{2}(\mathbb{F}_{p})$ , so that $X_{\text{ns}}(p)(\mathbb{C})\cong {\mathcal{H}}\cup \mathbb{Q}\cup \{\infty \}/\unicode[STIX]{x1D6E4}_{\text{ns}}(p)$ . Thus, $X_{\text{ns}}(p)$ has elliptic points if and only if $\unicode[STIX]{x1D6E4}_{\text{ns}}(p)$ contains an element with characteristic polynomial equal to $x^{2}+1$ or $x^{2}+x+1$ . Recall that the elements of a nonsplit Cartan subgroup of $\text{GL}_{2}(\mathbb{F}_{p})$ (not belonging to the center) have irreducible characteristic polynomials. If $p\equiv 1~\text{mod}~4$ and $p\equiv 1~\text{mod}~3$ , the two polynomials above have both roots over $\mathbb{F}_{p}$ . Hence, when $p\equiv 1~\text{mod}~12$ there are no elliptic points on $X_{\text{ns}}(p)$ . The Corollary then follows from the two previous Propositions.◻

4 Automorphisms of $X_{\text{ns}}(p)$ not preserving the cusps

Let $p$ be a prime number and $l$ be another prime number different from $p$ . Let $T_{l}$ be the $l$ th Hecke operator for the modular curve $X_{\text{ns}}(p)$ , defined as a modular correspondence (see [Shi58, §2]). Let $E$ be an elliptic curve over $\mathbb{C}$ and let $\unicode[STIX]{x1D711}$ be an isomorphism from the $p$ -torsion $E[p]$ of $E$ to $\mathbb{Z}/p\mathbb{Z}\times \mathbb{Z}/p\mathbb{Z}$ . Then the pair $(E,\unicode[STIX]{x1D711})$ gives a point on $X_{\text{ns}}(p)$ . Let $A$ be a subgroup of $E$ of order $l$ . Since $l\neq p$ , taking the quotient by $A$ on $E$ induces an isomorphism from the $p$ -torsion of $E$ to the $p$ -torsion of $E/A$ so that $\unicode[STIX]{x1D711}$ induces an isomorphism $\unicode[STIX]{x1D711}_{A}$ from the $p$ -torsion of $E/A$ to $\mathbb{Z}/p\mathbb{Z}\times \mathbb{Z}/p\mathbb{Z}$ . The operator $T_{l}$ acts on a noncuspidal point of $X_{\text{ns}}(p)$ in the following way

$$\begin{eqnarray}T_{l}(E,\unicode[STIX]{x1D711})=\mathop{\sum }_{\substack{ A\subset E(\mathbb{C}) \\ \#A=l}}(E/A,\unicode[STIX]{x1D711}_{A})\end{eqnarray}$$

and it induces an element in the endomorphism algebra of $J_{\text{ns}}(p)$ .

For every pair $(E,\unicode[STIX]{x1D711})$ with $E$ defined over $\overline{\mathbb{Q}}$ let $\widetilde{E}$ be the reduction of $E$ modulo a prime ideal containing $l$ , and $\widetilde{\unicode[STIX]{x1D711}}:\widetilde{E}[p]\rightarrow \mathbb{Z}/p\mathbb{Z}\times \mathbb{Z}/p\mathbb{Z}$ be the map naturally induced by $\unicode[STIX]{x1D711}$ , since $l\neq p$ . Then the pair $(\widetilde{E},\widetilde{\unicode[STIX]{x1D711}})$ gives a point on $X_{\text{ns}}(p)$ over $\overline{\mathbb{F}_{l}}$ . Let $\text{Frob}_{l}$ be the generator of the Galois group $\text{Gal}(\overline{\mathbb{F}_{l}}/\mathbb{F}_{l})$ and let $[l^{-1}]$ be the inverse of the multiplication by $l$ map on $E[p]$ . We have

$$\begin{eqnarray}\mathop{\sum }_{\substack{ A\subset \widetilde{E}(\mathbb{\overline{}}\mathbb{F}_{l}) \\ \#A=l}}(\widetilde{E}/A,\widetilde{\unicode[STIX]{x1D711}}_{A})=(\widetilde{E}^{\text{Frob}_{l}},\widetilde{\unicode[STIX]{x1D711}}\circ {\text{Frob}_{l}}^{-1})+l\cdot (\widetilde{E}^{{\text{Frob}_{l}}^{-1}},\widetilde{\unicode[STIX]{x1D711}}\circ [l^{-1}]\circ \text{Frob}_{l})\end{eqnarray}$$

as divisors of $X_{\text{ns}}(p)$ reduced modulo $l$ (see, for example, the proofs in [DS05, §8.7] regarding the modular curve $X_{1}(N)$ ). This implies the Eichler–Shimura relation:

$$\begin{eqnarray}T_{l}=\text{Frob}_{l}+l\cdot \text{Frob}_{l}^{-1}\end{eqnarray}$$

where both terms of the equation are homomorphisms on the divisor group of $X_{\text{ns}}(p)$ reduced modulo $l$ . Note that the Eichler–Shimura relation acquires this form in this case because a nonsplit Cartan subgroup of $\text{GL}_{2}(\mathbb{F}_{p})$ always contains the scalar matrix associated to $[l^{-1}]$ ; hence, the pairs $(\widetilde{E}^{{\text{Frob}_{l}}^{-1}},\widetilde{\unicode[STIX]{x1D711}}\circ [l^{-1}]\circ \text{Frob}_{l})$ and $(\widetilde{E}^{{\text{Frob}_{l}}^{-1}},\widetilde{\unicode[STIX]{x1D711}}\circ \text{Frob}_{l})$ give the same point on $X_{\text{ns}}(p)$ .

Since $T_{l}$ induces an endomorphism of $J_{\text{ns}}(p)$ , we will also write, with abuse of notation, the relation $T_{l}=\text{Frob}_{l}+l\cdot \text{Frob}_{l}^{-1}$ as endomorphisms of $J_{\text{ns}}(p)$ reduced modulo $l$ .

Lemma 4.1. Let $u$ be an automorphism of $X_{\text{ns}}(p)$ defined over the quadratic field $K(p)$ contained in the $p$ th cyclotomic field. Let $\unicode[STIX]{x1D70E}_{l}\in \text{Gal}(\overline{\mathbb{Q}}/\mathbb{Q})$ be a Frobenius element at $l$ . Then if $l\neq p$ we have

$$\begin{eqnarray}u^{\unicode[STIX]{x1D70E}_{l}}T_{l}=T_{l}u\end{eqnarray}$$

as endomorphisms of $J_{\text{ns}}(p)$ , that is,

$$\begin{eqnarray}T_{l}u=\left\{\begin{array}{@{}ll@{}}uT_{l}\quad & \text{if }l\text{ is a square modulo }p\\ \overline{u}T_{l}\quad & \text{if }l\text{ is not a square modulo }p\end{array}\right.\end{eqnarray}$$

where $\overline{u}$ is the $\text{Gal}(K(p)/\mathbb{Q})$ -conjugate of $u$ .

Proof. (Following [KM88, p. 64, Lemma 2.6]) Since $J_{\text{ns}}(p)$ has good reduction outside $p$ , the reduction modulo $l$ map is injective on the endomorphism ring of $J_{\text{ns}}(p)$ . Hence, the Lemma follows from the Eichler–Shimura relations and the fact that $K(p)$ is a quadratic field.◻

Let $\unicode[STIX]{x1D701}_{p}$ be a primitive $p$ th root of unity. The modular curve $X_{\text{ns}}(p)$ has exactly $p-1$ cusps, all defined over $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ and all conjugate under the action of $\text{Gal}(\mathbb{Q}(\unicode[STIX]{x1D701}_{p})/\mathbb{Q})$ (see [Ser89, p. 194–195]).

Proposition 4.2. Let $\unicode[STIX]{x1D70E}_{l}\in \text{Gal}(\overline{\mathbb{Q}}/\mathbb{Q})$ be a Frobenius element at $l$ . Then we have

$$\begin{eqnarray}T_{l}C=C^{\unicode[STIX]{x1D70E}_{l}}+l\cdot C^{{\unicode[STIX]{x1D70E}_{l}}^{-1}}\end{eqnarray}$$

as divisors on $X_{\text{ns}}(p)$ , for every cusp $C$ of $X_{\text{ns}}(p)$ .

Proof. The equality must hold modulo any prime ideal $\mathfrak{l}$ of $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ over $l$ because of the Eichler–Shimura relations. Moreover, since $l\neq p$ , the modular curve $X_{\text{ns}}(p)$ and its reduction modulo $l$ have the same number of cusps. Therefore, reduction modulo $\mathfrak{l}$ is injective on the set of cusps of $X_{\text{ns}}(p)$ , and the equality holds also over $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ .◻

Lemma 4.3. Let $p\geqslant 11$ and let $u$ be an automorphism of $X_{\text{ns}}(p)$ defined over $K(p)$ . Let $\unicode[STIX]{x1D70E}_{l}\in \text{Gal}(\overline{\mathbb{Q}}/\mathbb{Q})$ be a Frobenius element at $l$ and let $C$ be a cusp of $X_{\text{ns}}(p)$ . If for every cusp $C^{\prime }$ of $X_{\text{ns}}(p)$ the divisor

$$\begin{eqnarray}\displaystyle D_{l} & \stackrel{\text{def}}{=} & \displaystyle (u^{\unicode[STIX]{x1D70E}_{l}}T_{l}-T_{l}u)(C-C^{\prime })\nonumber\\ \displaystyle & = & \displaystyle u^{\unicode[STIX]{x1D70E}_{l}}T_{l}C+T_{l}uC^{\prime }-T_{l}uC-u^{\unicode[STIX]{x1D70E}_{l}}T_{l}C^{\prime }\nonumber\end{eqnarray}$$

is the zero divisor, then $uC$ is a cusp, that is:

If $u^{\unicode[STIX]{x1D70E}_{l}}T_{l}-T_{l}u$ is the zero operator on the group of divisors of $X_{\text{ns}}(p)$ of degree zero and supported in the cusps, then $u$ preserves the cusps.

Proof. By the previous Proposition, for every cusp $C$ , we have $T_{l}C=C^{\unicode[STIX]{x1D70E}_{l}}+l\cdot C^{{\unicode[STIX]{x1D70E}_{l}}^{-1}}$ . Thus, we can choose a cusp $C^{\prime }$ such that the supports of $T_{l}C$ and $T_{l}C^{\prime }$ are completely disjoint. For $D_{l}$ to be the zero divisor, we must have

$$\begin{eqnarray}u^{\unicode[STIX]{x1D70E}_{l}}C^{\unicode[STIX]{x1D70E}_{l}}+l\cdot u^{\unicode[STIX]{x1D70E}_{l}}C^{{\unicode[STIX]{x1D70E}_{l}}^{-1}}+T_{l}uC^{\prime }=T_{l}uC+u^{\unicode[STIX]{x1D70E}_{l}}{C^{\prime }}^{\unicode[STIX]{x1D70E}_{l}}+l\cdot u^{\unicode[STIX]{x1D70E}_{l}}{C^{\prime }}^{{\unicode[STIX]{x1D70E}_{l}}^{-1}}.\end{eqnarray}$$

Since we chose $C^{\prime }$ such that the supports of $T_{l}C$ and $T_{l}C^{\prime }$ are completely disjoint, we have

(4.2) $$\begin{eqnarray}T_{l}uC=u^{\unicode[STIX]{x1D70E}_{l}}C^{\unicode[STIX]{x1D70E}_{l}}+l\cdot u^{\unicode[STIX]{x1D70E}_{l}}C^{{\unicode[STIX]{x1D70E}_{l}}^{-1}}.\end{eqnarray}$$

If $uC$ is not a cusp, it corresponds to an elliptic curve $E$ defined over $\mathbb{Q}(uC)\subset \mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ .

Let $A$ and $B$ be two different cyclic groups of order $l$ in $E$ . Then the natural map $E/A\rightarrow E\rightarrow E/B$ has a cyclic kernel of order $l^{2}$ . Therefore, recalling the interpretation of the action of $T_{l}$ on noncuspidal points, we have that equation (4.2) can hold only if $E$ admits an endomorphism of degree $l^{2}$ with cyclic kernel. An elliptic curve without complex multiplication cannot have such an endomorphism, hence we obtain the Lemma as a consequence of the following Proposition.◻

Proposition 4.4. Let $p\geqslant 11$ and let $E$ be an elliptic curve over $\mathbb{C}$ with complex multiplication by an imaginary quadratic field $K$ . No point of $X_{\text{ns}}(p)$ associated to $E$ is defined over $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ .

Proof. Let $P$ be a point of $X_{\text{ns}}(p)$ associated to $E$ and defined over $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ . There are three cases: $p$ is inert, $p$ splits or $p$ ramifies in $K$ .

If $p$ is inert in $K$ , we consider the point $Q$ of $X_{\text{ns}}^{+}(p)$ given by $\{P,wP\}$ . This point $Q$ is defined over $L\subset \mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ and lifts to points of $X_{\text{ns}}(p)$ defined over $LK$ (and no smaller fields) as explained in [Ser89, p. 194–195]. But this is impossible since $P,wP$ are defined over $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ where $p$ ramifies completely.

If $p$ splits or ramifies in $K$ , this means that the image of the $\text{Gal}(\overline{\mathbb{Q}}/K(\unicode[STIX]{x1D701}_{p}))$ -representation modulo $p$ attached to $E$ is contained in respectively a split Cartan or a Borel subgroup of $\text{GL}_{2}(\mathbb{F}_{p})$ (see loc. cit.). But since $P$ is defined over $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ , the image of such representation is also contained in a nonsplit Cartan subgroup. Therefore, in both cases, it is contained in the subgroup of scalar matrixes. Then, the properties of the Weil pairing imply that $\text{Gal}(\overline{\mathbb{Q}}/K(\unicode[STIX]{x1D701}_{p}))$ acts on the $p$ -torsion of $E$ as a subgroup of $\{\pm 1\}$ . After choosing a suitable Weierstrass equation for $E$ , this means that the $x$ -coordinates of the $p$ -torsion points of $E$ are in $K(\unicode[STIX]{x1D701}_{p})$ . The theory of complex multiplication tells us that the ray class field $K_{(p)}$ modulo $(p)$ of $K$ is generated over $K$ by the $j$ -invariant of $E$ and by some rational function of the $x$ -coordinates of the $p$ -torsion points of $E$ [Sil94, p. 135, Theorem 5.6]. Thus we obtain $K_{(p)}\subset K(\unicode[STIX]{x1D701}_{p})$ . It is always true that $K(\unicode[STIX]{x1D701}_{p})\subset K_{(p)}$ , so the opposite inclusion holds if and only if $[K_{(p)}:K]\leqslant [K(\unicode[STIX]{x1D701}_{p}):K]$ . We have

$$\begin{eqnarray}[K_{(p)}:K]=h_{K}\frac{\#(O_{K}/pO_{K})^{\ast }}{w_{K}}\end{eqnarray}$$

where $h_{K}$ is the class number of $K$ , $O_{K}$ is the ring of integers of $K$ and $w_{K}$ is the number of roots of unity in $K$ taken modulo $(p)$ . The group $(O_{K}/pO_{K})^{\ast }$ is isomorphic to $\mathbb{F}_{p^{2}}^{\ast }$ , $\mathbb{F}_{p}^{\ast }\times \mathbb{F}_{p}^{\ast }$ or $\mathbb{F}_{p}^{\ast }\times \langle a\rangle$ with $a$ an element of order $p$ . Thus to have the inequality $[K_{(p)}:K]\leqslant [K(\unicode[STIX]{x1D701}_{p}):K]$ we must have

$$\begin{eqnarray}\frac{(p-1)^{2}}{w_{K}}\leqslant (p-1)\end{eqnarray}$$

and hence $p\leqslant w_{K}+1$ . Since the roots of unity in an imaginary quadratic field are at most $6$ , we get $p\leqslant 7$ .◻

Corollary 4.5. Let $p\geqslant 11$ and let $u$ be an automorphism of $X_{\text{ns}}(p)$ defined over $K(p)$ . If $u$ does not preserve the cusps, then there exists a nonconstant morphism $f$ from $X_{\text{ns}}(p)$ to $\mathbb{P}^{1}$ , defined over the field $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ and with degree less or equal to $6$ , such that $f\circ w\neq f$ .

Proof. Since $u$ is defined over $K(p)$ and does not preserve the cusps, we can apply Lemma 4.3 to obtain, for every prime $l\neq p$ , a nonzero divisor $D_{l}$ on $X_{\text{ns}}(p)$ , of degree zero, defined over $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ , which is the difference of two effective divisors of degree at most $2(l+1)$ . Moreover, we have $D_{l}=(u^{\unicode[STIX]{x1D70E}_{l}}T_{l}-T_{l}u)(C-C^{\prime })$ for two cusps $C$ , $C^{\prime }$ of $X_{\text{ns}}(p)$ . Hence, by Lemma 4.1 we deduce that $D_{l}$ must be zero in $J_{\text{ns}}(p)$ when $p\geqslant 11$ , which is to say that $D_{l}$ is a principal divisor. This gives us the existence of a nonconstant morphism $f$ from $X_{\text{ns}}(p)$ to $\mathbb{P}^{1}$ , defined over $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ and with degree at most $2(l+1)$ .

Now we apply $w$ to $D_{l}$ . If $D_{l}$ were invariant under the action of $w$ , we would have

(4.3) $$\begin{eqnarray}wT_{l}uC+wu^{\unicode[STIX]{x1D70E}_{l}}{C^{\prime }}^{\unicode[STIX]{x1D70E}_{l}}+l\cdot wu^{\unicode[STIX]{x1D70E}_{l}}{C^{\prime }}^{{\unicode[STIX]{x1D70E}_{l}}^{-1}}=T_{l}uC+u^{\unicode[STIX]{x1D70E}_{l}}{C^{\prime }}^{\unicode[STIX]{x1D70E}_{l}}+l\cdot u^{\unicode[STIX]{x1D70E}_{l}}{C^{\prime }}^{{\unicode[STIX]{x1D70E}_{l}}^{-1}},\end{eqnarray}$$

where we used the computation in the proof of Lemma 4.3. In that proof, we also showed that the divisor $T_{l}uC$ is the sum of $l+1$ different points. Hence since $l\geqslant 2$ , to satisfy equation (4.3), the supports of $T_{l}uC$ and $wT_{l}uC$ must have a nonempty intersection. This means that the support of $T_{l}uC$ has either two different points exchanged by $w$ or a point fixed by $w$ . The first case is impossible because the modular automorphism $w$ preserves the isomorphism class over $\mathbb{C}$ of the elliptic curve associated to noncuspidal points, and we can apply the same argument at the end of the proof of Lemma 4.3. The second case is also impossible by the previous Proposition, because the points fixed by $w$ are all associated with the elliptic curve with $j$ -invariant equal to $1728$ (see [Bar10, Proposition 7.10]), which is an elliptic curve with complex multiplication.

Thus, for $l\geqslant 2$ , we have $wD_{l}\neq D_{l}$ which implies $f\circ w\neq f$ . The degree of $f$ is $2(l+1)$ , so by choosing $l=2$ we get the Corollary.◻

5 Proof of main results

We first introduce a general Lemma.

Lemma 5.1. Let $X$ be a smooth projective curve over an algebraically closed field, and let $w$ be a nontrivial automorphism of $X$ having $r$ fixed points. If there exists a morphism $f$ from $X$ to $\mathbb{P}^{1}$ of degree $k$ such that $f\circ w\neq f$ , then $r\leqslant 2k$ .

Proof. [BH03, Lemma 3.5] Consider the rational function $g=f\circ w-f$ on $X$ . If $g$ is constant, then the fixed points of $w$ must be poles of $f$ , implying $r\leqslant k$ . If $g$ is not constant then the degree of $g$ is at most $2k-h$ , where $h$ is the number of poles that are also points fixed by $w$ . On the other hand, the other $r-h$ points fixed by $w$ are zeros of $g$ , hence there are at most $\text{deg }g$ of them. So we have $r-h\leqslant 2k-h$ which implies $r\leqslant 2k$ .◻

Let $p$ be an odd prime number.

Theorem 5.2. If $p\geqslant 29$ , all the automorphisms of $X_{\text{ns}}(p)$ preserve the cusps. If $p\equiv 1~\text{mod}~12$ and $p\neq 13$ , the automorphism group of $X_{\text{ns}}(p)$ is generated by the modular involution $w$ .

Proof. Let $u$ be an automorphism of $X_{\text{ns}}(p)$ not preserving the cusps. By Corollary 2.6 we have that $u$ is defined over $K(p)$ . Then Corollary 4.5 tells us that there exists a nonconstant morphism from $X_{\text{ns}}(p)$ to $\mathbb{P}^{1}$ , defined over $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ , with degree less or equal to $6$ and such that $f\circ w\neq f$ . Then by the previous Lemma we have

$$\begin{eqnarray}\#\{\text{fixed points of }w\}\leqslant 12.\end{eqnarray}$$

Recall that the number of fixed points of $w$ is $\frac{p-1}{2}$ if $p\equiv 1~\text{mod}~4$ and it is $\frac{p+1}{2}$ if $p\equiv 3~\text{mod}~4$ (see [Bar10, Proposition 7.10]). This implies $p\leqslant 23$ and proves the first part of the Theorem. The second part is now a consequence of Corollary 3.3.◻

We remark that the first part of the Theorem above can also be achieved avoiding the discussion about the invariance of $f$ under the action of $w$ , by using [Abr96, Theorem 0.1].

Now we prove that Serre’s uniformity conjecture, implies the absence of rational exceptional automorphisms of $X_{\text{ns}}(p)$ for almost all $p$ .

Theorem 5.3. Let $p\geqslant 29$ . If there exists an exceptional automorphism of $X_{\text{ns}}(p)$ defined over $\mathbb{Q}$ , then $X_{\text{ns}}^{+}(p)$ has a non-CM rational point.

Proof. Let $u$ be an automorphism of $X_{\text{ns}}(p)$ defined over $\mathbb{Q}$ . By Corollary 2.9 we have that $u$ commutes with $w$ and it induces an automorphism of $X_{\text{ns}}^{+}(p)$ defined over $\mathbb{Q}$ . Then it is enough to prove that $u$ preserves the rational CM points of $X_{\text{ns}}^{+}(p)$ only if $u$ is modular. Let $\unicode[STIX]{x1D70B}:X_{\text{ns}}(p)\rightarrow X_{\text{ns}}^{+}(p)$ be the degree-2 modular morphism given by the inclusion of a nonsplit Cartan subgroup in its normalizer, and let $Q$ be a rational CM point of $X_{\text{ns}}^{+}(p)$ . Note that the two points in $\unicode[STIX]{x1D70B}^{-1}(Q)$ are defined over the CM field of the elliptic curve associated to $Q$ (see [Ser89, p. 194–195]). Hence $uQ$ cannot be a rational CM point different from $Q$ because the two points in $\unicode[STIX]{x1D70B}^{-1}(uQ)$ are defined over the same field as the points in $\unicode[STIX]{x1D70B}^{-1}(Q)$ . Now we observe that the elliptic points of $X_{\text{ns}}(p)$ are the inverse images by $\unicode[STIX]{x1D70B}$ of the rational CM points associated to the elliptic curves with $j$ -invariant equal to 0 or 1728 (see [Bar10, Proposition 7.10]). Thus, if $u$ preserves the rational CM points of $X_{\text{ns}}^{+}(p)$ , then it preserves the elliptic points of $X_{\text{ns}}(p)$ . Furthermore, $u$ also preserves the cusps by Theorem 5.2, hence it is modular by Corollary 3.3.◻