The spin $L$-function on $\text{GSp}_{6}$ for Siegel modular forms

Abstract

We give a Rankin–Selberg integral representation for the Spin (degree eight) $L$-function on $\operatorname{PGSp}_{6}$ that applies to the cuspidal automorphic representations associated to Siegel modular forms. If $\unicode[STIX]{x1D70B}$ corresponds to a level-one Siegel modular form $f$ of even weight, and if $f$ has a nonvanishing maximal Fourier coefficient (defined below), then we deduce the functional equation and finiteness of poles of the completed Spin $L$-function $\unicode[STIX]{x1D6EC}(\unicode[STIX]{x1D70B},\text{Spin},s)$ of $\unicode[STIX]{x1D70B}$.

1 Introduction

1.1 Main theorems

The purpose of this paper is to give a family of Rankin–Selberg integrals $I_{B}(\unicode[STIX]{x1D719},s)$ for the Spin $L$ -function of the cuspidal automorphic representations of $\operatorname{GSp}_{6}(\mathbf{A})$ that correspond to Siegel modular forms. The Rankin–Selberg integrals are indexed by quaternion algebras $B$ over $\mathbf{Q}$ that are ramified at the infinite place.

Denote by ${\mathcal{H}}_{3}$ the Siegel upper half-space of degree three; ${\mathcal{H}}_{3}$ is the space of $3\times 3$ complex symmetric matrices $Z$ whose imaginary part is positive definite:

$$\begin{eqnarray}{\mathcal{H}}_{3}=\{Z=X+iY\in M_{3}(\mathbf{C}):\text{}^{t}Z=Z,Y>0\}.\end{eqnarray}$$

Denote by $\operatorname{GSp}_{6}^{+}(\mathbf{R})$ the subgroup of $\operatorname{GSp}_{6}(\mathbf{R})$ whose elements have positive similitude. Then $\operatorname{GSp}_{6}^{+}(\mathbf{R})$ acts on ${\mathcal{H}}_{3}$ by the formula $gZ=(AZ+B)(CZ+D)^{-1}$ if $g=(\!\begin{smallmatrix}A & B\\ C & D\end{smallmatrix}\!)$ . Define $K_{\infty ,\operatorname{Sp}_{6}}\simeq U(3)$ to be the subgroup of $\operatorname{Sp}_{6}(\mathbf{R})$ that stabilizes $i\in {\mathcal{H}}_{3}$ for this action. For $Z\in {\mathcal{H}}_{3}$ and $g\in \operatorname{GSp}_{6}^{+}(\mathbf{R})$ , define $j(g,Z)=\unicode[STIX]{x1D708}(g)^{-2}\det (CZ+D)$ , where $\unicode[STIX]{x1D708}$ is the similitude.

Suppose $r\geqslant 0$ is an integer, and $\unicode[STIX]{x1D70B}$ is a cuspidal automorphic representation of $\operatorname{GSp}_{6}(\mathbf{A})$ .

Definition 1.1. We say that $\unicode[STIX]{x1D70B}$ is associated to a level-one Siegel modular form of weight $2r$ if the following conditions are satisfied.

1. – There is a nonzero $\unicode[STIX]{x1D719}$ in the space of $\unicode[STIX]{x1D70B}$ satisfying $\unicode[STIX]{x1D719}(gk_{f}k_{\infty })=j(k_{\infty },i)^{-2r}\unicode[STIX]{x1D719}(g)$ for all $g\in \operatorname{GSp}_{6}(\mathbf{A})$ , $k_{f}\in \prod _{p}\operatorname{GSp}_{6}(\mathbf{Z}_{p})$ , and $k_{\infty }\in K_{\infty ,\operatorname{Sp}_{6}}$ .

2. – $\unicode[STIX]{x1D70B}$ has trivial central character.

3. – The function $f_{\unicode[STIX]{x1D719}}:{\mathcal{H}}_{3}\rightarrow \mathbf{C}$ well defined by the equality $f_{\unicode[STIX]{x1D719}}(g_{\infty }(i))=\unicode[STIX]{x1D708}(g_{\infty })^{r}j(g_{\infty },i)^{2r}\unicode[STIX]{x1D719}(g_{\infty })$ for all $g_{\infty }\in \operatorname{GSp}_{6}(\mathbf{R})^{+}$ is a classical Siegel modular form of weight  $2r$ .

See, for example, [AS01, § 4] for more on the relationship between Siegel modular forms and their associated automorphic representations. The content of the final condition is that $f_{\unicode[STIX]{x1D719}}$ is a holomorphic function on ${\mathcal{H}}_{3}$ , and thus has a Fourier expansion of the form

$$\begin{eqnarray}f_{\unicode[STIX]{x1D719}}(Z)=\mathop{\sum }_{T>0}a_{f}(T)e^{2\unicode[STIX]{x1D70B}i\operatorname{tr}(TZ)}.\end{eqnarray}$$

Here the sum ranges over all half-integral, symmetric, $3\times 3$ matrices $T$ that are positive definite.

Via the even Clifford construction, half-integral, symmetric, $3\times 3$ matrices $T$ correspond to pairs $(\unicode[STIX]{x1D6EC},\{1,v_{1},v_{2},v_{3}\})$ consisting of orders $\unicode[STIX]{x1D6EC}$ in quaternion algebras over $\mathbf{Q}$ equipped with a particular type of nice ordered basis $\{1,v_{1},v_{2},v_{3}\}$ of $\unicode[STIX]{x1D6EC}$ . This correspondence goes back at least to Brzezinski [Brz83, Proposition 5.2], and more recently it has been studied and generalized by Gross and Lucianovic [GL09]. The condition that $T$ be positive (or negative) definite is exactly that the associated quaternion algebra $\unicode[STIX]{x1D6EC}\,\otimes \,\mathbf{Q}$ be ramified at infinity. As an example, the identity matrix $1_{3}$ corresponds to the order $\mathbf{Z}+\mathbf{Z}i+\mathbf{Z}j+\mathbf{Z}k$ in Hamilton’s quaternions, while the matrix

$$\begin{eqnarray}\left(\begin{array}{@{}ccc@{}}1 & {\textstyle \frac{1}{2}} & {\textstyle \frac{1}{2}}\\ {\textstyle \frac{1}{2}} & 1 & {\textstyle \frac{1}{2}}\\ {\textstyle \frac{1}{2}} & {\textstyle \frac{1}{2}} & 1\end{array}\right)\end{eqnarray}$$

corresponds to Hurwitz’s maximal order

$$\begin{eqnarray}\{a+xi+yj+zk:a,x,y,z\in \mathbf{Z}~\text{or}~a,x,y,z\in \mathbf{Z}+{\textstyle \frac{1}{2}}\}.\end{eqnarray}$$

A cuspidal representation $\unicode[STIX]{x1D70B}$ that is associated to a level-one Siegel modular form is spherical at all finite places, and thus has a Spin $L$ -function $L(\unicode[STIX]{x1D70B},\text{Spin},s)$ , defined for $\text{Re}(s)\gg 0$ as an absolutely convergent Euler product over all the primes of $\mathbf{Q}$ . If $\unicode[STIX]{x1D70B}$ is associated to Siegel modular form of weight $2r$ , we define the completed Spin $L$ -function $\unicode[STIX]{x1D6EC}(\unicode[STIX]{x1D70B},\text{Spin},s)$ to be

$$\begin{eqnarray}\unicode[STIX]{x1D6EC}(\unicode[STIX]{x1D70B},\text{Spin},s)=\unicode[STIX]{x1D6E4}_{\mathbf{C}}(s+r-2)\unicode[STIX]{x1D6E4}_{\mathbf{C}}(s+r-1)\unicode[STIX]{x1D6E4}_{\mathbf{C}}(s+r)\unicode[STIX]{x1D6E4}_{\mathbf{C}}(s+3r-3)L(\unicode[STIX]{x1D70B},\text{Spin},s)\end{eqnarray}$$

where, as usual, $\unicode[STIX]{x1D6E4}_{\mathbf{C}}(s)=2(2\unicode[STIX]{x1D70B})^{-s}\unicode[STIX]{x1D6E4}(s)$ . Our main theorem is the following.

Theorem 1.2. Suppose that $\unicode[STIX]{x1D70B}$ is a cuspidal automorphic representation of $\operatorname{GSp}_{6}(\mathbf{A})$ associated to a level-one Siegel modular form $f$ of weight $2r$ . Suppose furthermore that $a_{f}(T)\neq 0$ for some Fourier coefficient $a_{f}(T)$ of $f$ , such that $T$ corresponds to a maximal order in some quaternion algebra over $\mathbf{Q}$ ramified at infinity. Then $\unicode[STIX]{x1D6EC}(\unicode[STIX]{x1D70B},\text{Spin},s)$ has meromorphic continuation in $s$ to the entire complex plane with at worst finitely many poles, and satisfies the functional equation $\unicode[STIX]{x1D6EC}(\unicode[STIX]{x1D70B},\text{Spin},s)=\unicode[STIX]{x1D6EC}(\unicode[STIX]{x1D70B},\text{Spin},1-s)$ .

The meromorphic continuation of the $L$ -function, but not the functional equation or finiteness of the poles, was known previously. Namely, the Spin $L$ -function on $\operatorname{GSp}_{6}$ appears in the constant term of a cuspidal Eisenstein series on an exceptional group of type $F_{4}$ . Thus its meromorphic continuation follows from the method of Langlands [Lan71], see Shahidi [Sha88, Theorem 6.1]. Since the underlying cuspidal representation of a Siegel modular form is not generic, the Langlands–Shahidi method cannot be used to obtain the functional equation or the finiteness of the poles of this $L$ -function.

The bound we get on the poles is that they are contained in the set of integer and half-integer points in the interval $[-2,3]$ ; this bound is not expected to be optimal. One should be able to get much better control of the poles by applying the theorems of Yamana [Yam13]. Let us also remark that it is known [Yam09] that every level-one Siegel modular form has a nonvanishing Fourier coefficient $a(T)$ for a $T$ that is primitive, meaning that $T/n$ is not half-integral for all integers $n>1$ . The primitive $T$ correspond to Gorenstein quaternion orders [GL09], the set of which strictly contain the set of maximal orders.

It is an interesting question to determine if every level-one Siegel modular form $f$ on $\operatorname{GSp}_{6}$ as in Theorem 1.2 has a nonvanishing Fourier coefficient $a_{f}(T)$ with $T$ corresponding to a maximal order. The analogous statement for Siegel modular forms on $\operatorname{GSp}_{4}$ was proved by Saha in [Sah13]. On $\operatorname{GSp}_{4}$ , Fourier coefficients $a(T)$ of level-one Siegel modular forms are parametrized by $2\times 2$ half-integral symmetric matrices. Applying Gauss composition, such $T$ correspond to pairs $(S,I)$ where $S$ is a quadratic ring and $I$ is an ideal class for $S$ . It is a consequence of the main results of [Sah13] that every level-one Siegel modular form on $\operatorname{GSp}_{4}$ has a nonvanishing Fourier coefficient $a(T)$ for a $T$ that corresponds to a pair $(S,I)$ with $S$ the maximal order in a quadratic imaginary extension of $\mathbf{Q}$ . Analogously to Theorem 1.2, the need for a nonvanishing ‘maximal’ Fourier coefficient for Siegel modular forms on $\operatorname{GSp}_{4}$ arises in the integral representation of Furusawa [Fur93].

Since we obtain the functional equation of the completed Spin $L$ -function for level-one Siegel modular forms, we thus get the exact order of vanishing of $L(\unicode[STIX]{x1D70B},\text{Spin},s)$ for these $\unicode[STIX]{x1D70B}$ at sufficiently negative integers.

Corollary 1.3. Suppose that $\unicode[STIX]{x1D70B}$ is a cuspidal automorphic representation of $\operatorname{GSp}_{6}(\mathbf{A})$ associated to a level-one Siegel modular form $f$ of even weight, and that $a_{f}(T)\neq 0$ for a Fourier coefficient of $f$ corresponding to a maximal order in a quaternion algebra over $\mathbf{Q}$ . Then $\text{ord}_{s=-j}L(\unicode[STIX]{x1D70B},\text{Spin},s)=4$ for sufficiently large integers  $j$ .

Theorem 1.2 follows from the analysis of the Rankin–Selberg integral $I_{B}(\unicode[STIX]{x1D719},s)$ for the Spin $L$ -function, together with the analytic properties of the normalized Eisenstein series used in the integral. Suppose that $B$ is a quaternion algebra over $\mathbf{Q}$ , ramified at infinity. Denote by $J$ the set of $3\times 3$ Hermitian matrices over  $B$ : $J=\{h\in M_{3}(B):h^{\ast }=h\}$ , where for $x\in M_{3}(B)$ , $x^{\ast }$ denotes the transpose conjugate of $x$ , the conjugation being from the quaternion structure on  $B$ . Associated to $J$ , the Freudenthal construction produces a reductive $\mathbf{Q}$ -group $\text{G}$ of type $D_{6}$ , and there is an embedding $\operatorname{GSp}_{6}\rightarrow \text{G}$ . The group $\text{G}$ has a maximal parabolic $P$ , whose unipotent radical is abelian, and naturally isomorphic to $J$ under addition. Suppose $r\geqslant 0$ is an integer, and $B_{0}$ is a maximal order in  $B$ . Associated to $r,B_{0}$ , and a character $\unicode[STIX]{x1D712}_{s}:P(\mathbf{A})\rightarrow \mathbf{C}^{\times }$ , we define a normalized Eisenstein series $E_{2r}^{\ast }(g,s)$ for  $\text{G}$ .

For a cusp form $\unicode[STIX]{x1D719}$ in the space of a cuspidal representation $\unicode[STIX]{x1D70B}$ associated to a Siegel modular form of weight $2r$ , the Rankin–Selberg integral is

$$\begin{eqnarray}I_{2r}(\unicode[STIX]{x1D719},s)=\int _{\operatorname{GSp}_{6}(\mathbf{Q})Z(\mathbf{A})\backslash \!\operatorname{GSp}_{6}(\mathbf{A})}\unicode[STIX]{x1D719}(g)E_{2r}^{\ast }(g,s)\,dg.\end{eqnarray}$$

Suppose that the maximal order $B_{0}$ corresponds to the half-integral symmetric matrix  $T$ . When $\unicode[STIX]{x1D719}$ is as in Definition 1.1, we show that $I_{2r}(\unicode[STIX]{x1D719},s)$ is equal to $a_{f_{\unicode[STIX]{x1D719}}}(T)\unicode[STIX]{x1D6EC}(\unicode[STIX]{x1D70B},\text{Spin},s-2)$ , up to a nonzero constant. To show this equality, we unfold the integral $I_{2r}(\unicode[STIX]{x1D719},s)$ , and then apply the result [Evd84] of Evdokimov, who computed a Dirichlet series for the Spin $L$ -function on $\operatorname{GSp}_{6}$ . More precisely, [Evd84, Theorem 3] is not quite a Dirichlet series for the Spin $L$ -function. However, when one combines [Evd84, Theorem 3] with Lemma 3.8 below, which involves the arithmetic invariant theory of quaternion orders, and the fact that $B_{0}$ is a maximal order, one does obtain a Dirichlet series for the Spin $L$ -function. The integral $I_{2r}(\unicode[STIX]{x1D719},s)$ then unfolds to exactly this Dirichlet series.

In § 7, we use Langlands’ theorem of the constant term and functional equation, together with Maass–Shimura differential operators, to analyze the analytic properties of the normalized Eisenstein series $E_{2r}^{\ast }(g,s)$ . We prove that it satisfies the functional equation $E_{2r}^{\ast }(g,s)=E_{2r}^{\ast }(g,5-s)$ and has at most finitely many poles, all contained in the set of integral and half-integral points in the interval $[0,5]$ . Theorem 1.2 then follows.

1.2 The Spin $L$ -function

When a reductive $\mathbf{Q}$ -group has an associated Shimura variety, the minuscule cocharacter $\unicode[STIX]{x1D707}$ that is part of the Shimura datum singles out a representation of the Langlands dual group, and thus an automorphic $L$ -function, that is of particular importance. These $L$ -functions are expected to control the arithmetic of the Shimura variety. On Siegel modular varieties this is the Spin $L$ -function. For Siegel modular forms on $\operatorname{GSp}_{4}$ , the desired analytic properties of this $L$ -function were first worked out by Andrianov [And71, And74]. Other Rankin–Selberg integrals for spinor $L$ -functions that apply to holomorphic modular forms are the triple product of Garrett [Gar87] on $\operatorname{GL}_{2}\times \operatorname{GL}_{2}\times \operatorname{GL}_{2}$ , which was extended to $\operatorname{GL}_{2}$ over a general étale cubic algebra by Piatetski-Shapiro and Rallis [PSR87], and the integral of Furusawa [Fur93] for the Spin $L$ -function on $\operatorname{GSp}_{4}\times \operatorname{GU}(1,1)$ . The Spin $L$ -function of Siegel modular forms on $\operatorname{GSp}_{2n}$ for $n>3$ remains poorly understood.

For the Spin $L$ -function on $\operatorname{GSp}_{6}$ , there is also the integral of Bump and Ginzburg [BG92] that applies to generic cusp forms, and more recently the result [PS15] of Pollack and Shah that applies to cuspidal representations that support a rank-two Fourier coefficient. The integrals in [BG92] and [PS15] vanish identically for holomorphic Siegel modular forms.

It turns out that the integral in this paper, and the older integrals [Gar87, PSR87, Fur93, BG92], and [PS15] all appear to be mysteriously connected to the magic triangle of Deligne and Gross [DG02], as we explain in Appendix B. This connection seems to be closely related to the so-called towers of Rankin–Selberg integrals introduced by Ginzburg and Rallis [GR94]. Furthermore, that Freudenthal’s magic square might be connected to the Rankin–Selberg method is discussed, from a perspective different from our own, in [Bum05]. Nevertheless, we hope the examples we discuss in the appendix will stimulate future inquiry.

1.3 Layout of paper

The contents of the paper is as follows. In § 2 we review the Freudenthal construction. In § 3, we review and slightly extend the results of [GL09] that relate ternary quadratic forms and quaternion orders. In § 4, we recall [Evd84, Theorem 3] of Evdokimov, and combine this result with the work of § 3 to obtain a Dirichlet series for the Spin $L$ -function. We define the global Rankin–Selberg integral and unfold it in § 5. In §6 we evaluate the unfolded integral in terms of $\unicode[STIX]{x1D6EC}(\unicode[STIX]{x1D70B},\text{Spin},s)$ . We give some results on the group $\text{G}(\mathbf{A})$ and analyze the normalized Eisenstein series $E_{2r}^{\ast }(g,s)$ in §7. The group $\text{G}$ is closely related to a classical group $\operatorname{GU}_{6}(B)$ . In Appendix A we make this relationship explicit. In Appendix B we briefly discuss the magic triangle of Deligne and Gross and its connection to some Rankin–Selberg integrals.

2 The Freudenthal construction

In this section we review the Freudenthal construction [Fre54], specialized to the case of interest for us. Nothing in this section is new. We follow the exposition of Bhargava and Ho [BH16, §§ 5.3 and 6.4] fairly closely, with the exception of our definition of rank-one elements. The reader is also instructed to see Springer [Spr06] and the references contained therein for more on the Freudenthal construction.

2.1 Three-by-three hermitian matrices

Suppose that $B$ is a quaternion algebra over the ground field $F$ of characteristic zero. For $a\in B$ , write $a^{\ast }$ for the conjugate of $a$ , $\operatorname{tr}(a)=a+a^{\ast }$ , $n(a)=aa^{\ast }$ , the trace and norm of  $a$ . These are considered as elements of  $F$ . The symmetric pairing on $B$ is defined by

(2.1) $$\begin{eqnarray}(u,v)=\operatorname{tr}(uv^{\ast })=(v,u).\end{eqnarray}$$

We consider $3\times 3$ hermitian matrices over $B$ . If $h$ is such a matrix, then

(2.2) $$\begin{eqnarray}h=\left(\begin{array}{@{}ccc@{}}c_{1} & a_{3} & a_{2}^{\ast }\\ a_{3}^{\ast } & c_{2} & a_{1}\\ a_{2} & a_{1}^{\ast } & c_{3}\end{array}\right)\end{eqnarray}$$

where $c_{1},c_{2},c_{3}$ are elements of $F$ and $a_{1},a_{2},a_{3}$ are elements of  $B$ . The set of such matrices is denoted $H_{3}(B)$ .

For $h$ as in (2.2), define

$$\begin{eqnarray}N(h):=c_{1}c_{2}c_{3}-c_{1}n(a_{1})-c_{2}n(a_{2})-c_{3}n(a_{3})+\operatorname{tr}(a_{1}a_{2}a_{3})\end{eqnarray}$$

the norm of $h$ , and

$$\begin{eqnarray}\operatorname{tr}(h)=c_{1}+c_{2}+c_{3}\end{eqnarray}$$

the trace of $h$ . Also define

(2.3) $$\begin{eqnarray}h^{\#}:=\left(\begin{array}{@{}ccc@{}}c_{2}c_{3}-n(a_{1}) & a_{2}^{\ast }a_{1}^{\ast }-c_{3}a_{3} & a_{3}a_{1}-c_{2}a_{2}^{\ast }\\ a_{1}a_{2}-c_{3}a_{3}^{\ast } & c_{1}c_{3}-n(a_{2}) & a_{3}^{\ast }a_{2}^{\ast }-c_{1}a_{1}\\ a_{1}^{\ast }a_{3}^{\ast }-c_{2}a_{2} & a_{2}a_{3}-c_{1}a_{1}^{\ast } & c_{1}c_{2}-n(a_{3})\end{array}\right).\end{eqnarray}$$

One has that $hh^{\#}=N(h)1_{3}=h^{\#}h$ , and $(h^{\#})^{\#}=N(h)h$ .

Definition 2.1. An element $h\in H_{3}(B)$ is said to be of rank three if $N(h)\neq 0$ , of rank two if $N(h)=0$ but $h^{\#}\neq 0$ , and of rank one if $h^{\#}=0$ but $h\neq 0$ .

For $x,y$ $3\times 3$ hermitian matrices over $B$ , define

$$\begin{eqnarray}x\times y:=(x+y)^{\#}-x^{\#}-y^{\#}\end{eqnarray}$$

and

$$\begin{eqnarray}\operatorname{tr}(x,y):={\textstyle \frac{1}{2}}\operatorname{tr}(xy+yx).\end{eqnarray}$$

Finally, denote by

$$\begin{eqnarray}(\;,\;,\;):H_{3}(B)\times H_{3}(B)\times H_{3}(B)\rightarrow F\end{eqnarray}$$

the unique symmetric trilinear form satisfying $(x,x,x)=6N(x)$ . Equivalently,

$$\begin{eqnarray}(x,y,z)=N(x+y+z)-N(x+y)-N(x+z)-N(y+z)+N(x)+N(y)+N(z).\end{eqnarray}$$

For $x,y,z\in H_{3}(B)$ , one has

$$\begin{eqnarray}\operatorname{tr}(x\times y,z)=(x,y,z)=\operatorname{tr}(x,y\times z).\end{eqnarray}$$

The trilinear form is nondegenerate in the sense that if $(x,y,z)=0$ for all $x,y$ , then $z=0$ . The formula

$$\begin{eqnarray}\operatorname{tr}(x,y)=\mathop{\sum }_{1\leqslant i\leqslant 3}x_{ii}y_{ii}+\mathop{\sum }_{1\leqslant i<j\leqslant 3}(x_{ij},y_{ij})\end{eqnarray}$$

shows that the bilinear form $\operatorname{tr}(\;,\;)$ is nondegenerate. Here $x_{ij}$ denotes the $i,j$ entry of $x$ , and $(\;,\;)$ is the symmetric pairing on $B$ defined in (2.1). For $x,y\in H_{3}(B)$ , one has the formula

(2.4) $$\begin{eqnarray}N(x+y)=N(x)+\operatorname{tr}(x^{\#},y)+\operatorname{tr}(x,y^{\#})+N(y).\end{eqnarray}$$

2.2 The defining representation

Fix the quaternion algebra $B$ over $F$ . We denote by $W$ the vector space $F\oplus H_{3}(B)\oplus H_{3}(B)\oplus F$ . Typical elements of $W$ are denoted $(a,b,c,d)$ with $a,d\in F$ , $b,c\in H_{3}(B)$ . The space $W$ is equipped with a symplectic and quartic form. The symplectic form $\langle \;,\;\rangle :W\times W\rightarrow F$ is defined by

$$\begin{eqnarray}\langle (a,b,c,d),(a^{\prime },b^{\prime },c^{\prime },d^{\prime })\rangle =ad^{\prime }-\operatorname{tr}(b,c^{\prime })+\operatorname{tr}(c,b^{\prime })-da^{\prime }.\end{eqnarray}$$

The quartic form $Q:W\rightarrow F$ is defined by

$$\begin{eqnarray}Q((a,b,c,d))=(ad-\operatorname{tr}(b,c))^{2}+4aN(c)+4dN(b)-4\operatorname{tr}(b^{\#},c^{\#}).\end{eqnarray}$$

We now define the (similitude) algebraic group $\text{G}$ .

Definition 2.2. Consider $\operatorname{GL}(W)$ to act on the right of $W$ . The group $\text{G}$ is defined to be the set of $(g,\unicode[STIX]{x1D708}(g))\in \operatorname{GL}(W)\times \operatorname{GL}_{1}$ that satisfy $\langle ug,vg\rangle =\unicode[STIX]{x1D708}(g)\langle u,v\rangle$ and $Q(wg)=\unicode[STIX]{x1D708}(g)^{2}Q(w)$ for all $u,v,w\in W$ . The map $\unicode[STIX]{x1D708}:\text{G}\rightarrow \operatorname{GL}_{1}$ sending $g\mapsto \unicode[STIX]{x1D708}(g)$ is called the similitude.

Typically, the Freudenthal construction defines the subgroup of $\text{G}$ with similitude 1, but we will need this similitude version. It is clear that $\text{G}$ is a reductive group; knowledge of the generators of $\text{G}$ proves that it is connected as well [Bro69].

We will give some explicit elements of $\text{G}$ . Most simply, we have the following maps:

1. – for $\unicode[STIX]{x1D706}\in \operatorname{GL}_{1}$ , $(a,b,c,d)\mapsto (\unicode[STIX]{x1D706}a,\unicode[STIX]{x1D706}b,\unicode[STIX]{x1D706}c,\unicode[STIX]{x1D706}d)$ , with similitude $\unicode[STIX]{x1D706}^{2}$ ;

2. – and $(a,b,c,d)\mapsto (\unicode[STIX]{x1D706}^{2}a,\unicode[STIX]{x1D706}b,c,\unicode[STIX]{x1D706}^{-1}d)$ , with similitude $\unicode[STIX]{x1D706}$ ;

3. – $(a,b,c,d)\mapsto (-d,c,-b,a)$ with similitude 1.

This last map we denote by $J_{6}$ , or $J$ , if no confusion seems likely. Under the map $\operatorname{GSp}_{6}\rightarrow \text{G}$ defined in § 5.1, $J_{6}$ is the image of the element $(\!\begin{smallmatrix} & 1_{3}\\ -1_{3} & \end{smallmatrix}\!)$ of $\operatorname{GSp}_{6}$ .

Here are some more interesting elements of $\text{G}$ . First, for $X\in H_{3}(B)$ , define $n(X):W\rightarrow W$ via the formula

(2.5) $$\begin{eqnarray}(a,b,c,d)n(X)=(a,b+aX,c+b\times X+aX^{\#},d+\operatorname{tr}(c,X)+\operatorname{tr}(b,X^{\#})+aN(X)).\end{eqnarray}$$

Similarly, for $Y\in H_{3}(B)$ , define $\bar{n}(Y):W\rightarrow W$ via the formula

$$\begin{eqnarray}(a,b,c,d)\bar{n}(Y)=(a+\operatorname{tr}(b,Y)+\operatorname{tr}(c,Y^{\#})+dN(Y),b+c\times Y+dY^{\#},c+dY,d).\end{eqnarray}$$

The following lemma says that the $n(X),\bar{n}(Y)$ are elements of $\text{G}$ with similitude 1, that $n(X_{1}+X_{2})=n(X_{1})n(X_{2})$ , and similarly for $\bar{n}$ . For a proof of the lemma, see [Spr06, Lemma 2.3].

Lemma 2.3. The maps $X\mapsto (n(X),1)$ and $Y\mapsto (\bar{n}(Y),1)$ define group homomorphisms $H_{3}(B)\rightarrow \text{G}$ .

We define one more type of (fairly) explicit element of $\text{G}$ . Consider triples

$$\begin{eqnarray}(t,\tilde{t},\unicode[STIX]{x1D706})\in \operatorname{GL}(H_{3}(B))\times \operatorname{GL}(H_{3}(B))\times \operatorname{GL}_{1}\end{eqnarray}$$

that satisfy, for all $b,c\in H_{3}(B)$ :

1. – $\operatorname{tr}(t(b),\tilde{t}(c))=\operatorname{tr}(b,c)$ ;

2. – $N(t(b))=\unicode[STIX]{x1D706}N(b)$ ;

3. – $N(\tilde{t}(c))=\unicode[STIX]{x1D706}^{-1}N(c)$ ;

4. – $t(b)^{\#}=\unicode[STIX]{x1D706}\tilde{t}(b^{\#})$ and $\tilde{t}(c)^{\#}=\unicode[STIX]{x1D706}^{-1}t(c^{\#})$ .

Since the pairing $\operatorname{tr}(\;,\;)$ is nondegenerate, the first property uniquely determines $\tilde{t}$ in terms of $t$ , and then the first two properties imply the final two. If $m\in \operatorname{GL}_{3}(B)$ , and $r\in \operatorname{GL}_{1}(F)$ , then the linear map $t(b)=r^{-1}m^{\ast }bm$ satisfies $N(t(b))=r^{-3}N(m^{\ast }m)N(b)$ for all $b$ in $H_{3}(B)$ , and thus defines such an automorphism. For such a triple $(t,\tilde{t},\unicode[STIX]{x1D706})$ , one defines a map $m(t):W\rightarrow W$ via

$$\begin{eqnarray}(a,b,c,d)m(t)=(\unicode[STIX]{x1D706}a,t(b),\tilde{t}(c),\unicode[STIX]{x1D706}^{-1}d).\end{eqnarray}$$

It is immediate that $m(t)$ defines an element of $\text{G}$ with similitude 1.

We now define the rank of an element of $W$ . First, polarizing the quartic form $Q$ , there is a unique symmetric 4-linear form on $W$ that satisfies $(v,v,v,v)=Q(v)$ for all $v$ in  $W$ .

Definition 2.4. All elements of $W$ have rank less than or equal to four. An element $v$ of $W$ is of rank less than or equal to three if $Q(v)=(v,v,v,v)=0$ . An element $v$ is of rank less than or equal to two if $(v,v,v,w)=0$ for all $w$ in  $W$ . An element $v$ is of rank less than or equal to one if $(v,v,w,w^{\prime })=0$ for all $w$ in $W$ and $w^{\prime }$ in $(Fv)^{\bot }$ . Here $(Fv)^{\bot }$ is the set of $x\in W$ with $\langle v,x\rangle =0$ . Finally, $0$ is the unique element of rank zero. Since $\text{G}$ preserves the symplectic and the 4-linear form, $\text{G}$ preserves the rank of elements of  $W$ .

Elements of $W$ of rank one will play an important role in this paper. For example, $f=(0,0,0,1)$ is of rank one, and then applying $\overline{n}(X)$ for $X$ in $H_{3}(B)$ , one gets $(N(X),X^{\#},X,1)$ is rank one. To check that $f$ is rank one, the reader must simply observe (immediately) that the coefficient of $\unicode[STIX]{x1D707}\unicode[STIX]{x1D707}^{\prime }$ in $Q(f+\unicode[STIX]{x1D707}w+\unicode[STIX]{x1D707}^{\prime }w^{\prime })$ is zero for general $w=(a,b,c,d)$ and $w^{\prime }=(0,b^{\prime },c^{\prime },d^{\prime })$ in  $W$ .

It is easy to check that all the elements of rank one are in a single $\text{G}$ orbit. For example, one argument goes as follows. First, by applying operators $n(X)$ , $J_{6}$ , and scalar multiplication, it is easy to see that every element $v$ of $W$ can be moved to one with $d=1$ . Then applying $\overline{n}(X)$ , every element has a $\text{G}$ translate of the form $v_{0}=(z,y,0,1)$ . Now, if $v$ and thus $v_{0}$ is rank one, then $(v_{0},v_{0},w,w^{\prime })=0$ for arbitrary $w=(0,b,c,0)$ , $w^{\prime }=(0,b^{\prime },0,0)$ . The coefficient of $2\unicode[STIX]{x1D707}\unicode[STIX]{x1D707}^{\prime }$ in $Q(v_{0}+\unicode[STIX]{x1D707}w+\unicode[STIX]{x1D707}w^{\prime })$ is now easy to compute, and it is found to be $2\operatorname{tr}(y,b\times b^{\prime })-z\operatorname{tr}(b^{\prime },c)$ . Since this must be zero for all $b,b^{\prime },c$ , it follows that $y$ and $z$ are zero, by the nondegeneracy of the two pairings. Hence $v_{0}=(0,0,0,1)=f$ .

If $v$ is a rank-one element, the stabilizer in $\text{G}$ of the line $Fv$ is a parabolic subgroup of $\text{G}$ . Indeed, consider the set of flags of the form $0\subseteq \ell \subseteq W^{\prime }\subseteq W$ where $\ell$ is dimension one and $W^{\prime }$ is codimension one. The conditions $\langle v,w^{\prime }\rangle =0$ and $(v,v,w,w^{\prime })=0$ for all $v\in \ell$ , $w^{\prime }\in W^{\prime }$ and $w\in W$ clearly form closed conditions on the Grassmanian of flags of these dimensions. Since these conditions single out the rank-one lines, and since $\text{G}$ acts transitively on these lines, the stabilizer of a rank-one line is a parabolic subgroup.

The argument above also shows that if $(z,y,x,1)$ is rank one, then $y=x^{\#}$ and $z=N(x)$ . (Just apply $\overline{n}(-x)$ to $(z,y,x,1)$ .) In fact, the following is true.

Proposition 2.5. The element $(a,b,c,d)$ has rank at most one if and only if $b^{\#}=ac,c^{\#}=db$ , $bc=cb$ , and $ad=bc$ . Here, the products $bc$ and $cb$ are taken in $M_{3}(B)$ .

A proof of the proposition may be found in [GS05, Proposition 11.2].

3 Arithmetic invariant theory

In this section we review the arithmetic invariant theory of Gross and Lucianovic [GL09], and extend it slightly. The results in [GL09] relate ternary quadratic forms to orders in quaternion algebras, together with a nice basis of this order, up to isomorphism. The reader should consult Gross and Lucianovic [GL09] and Voight [Voi11] for more details and related results.

The relation between ternary quadratic forms and quaternion algebras has a long history, going back at least to Brandt [Bra28], Eichler [Eic52], and Brzezinksi [Brz83, Brz82]. It has been studied by many others as well, and from varying perspectives. The formulation of [GL09] turns out to be very well suited for our analysis of the Rankin–Selberg integral.

First, following [GL09], a quaternion ring is defined as follows. Suppose $R$ is a commutative ring, and $\unicode[STIX]{x1D6EC}$ is an associative $R$ -algebra with unit. Then $\unicode[STIX]{x1D6EC}$ is a quaternion ring over $R$ if the following properties are satisfied.

1. – $\unicode[STIX]{x1D6EC}$ is free of rank four as an $R$ -module, and $\unicode[STIX]{x1D6EC}/(R\cdot 1)$ is free of rank three.

2. – There is an $R$ -linear anti-involution $x\mapsto x^{\ast }$ on $\unicode[STIX]{x1D6EC}$ , such that $x^{\ast }=x$ for $x$ in $R$ , and $\operatorname{tr}(x):=x+x^{\ast }$ , $n(x):=xx^{\ast }$ are in  $R$ .

3. – If $x\in \unicode[STIX]{x1D6EC}$ , then left multiplication by $x$ on $\unicode[STIX]{x1D6EC}$ has trace $2\operatorname{tr}(x)$ .

For instance then, if $x\in \unicode[STIX]{x1D6EC}$ , then $x^{2}=\operatorname{tr}(x)x-n(x)$ . Note that we always assume our quaternion rings are free; a vast generalization is considered in [Voi11]. The following definition and lemma from [GL09] are crucial.

Definition 3.1 [GL09].

Suppose $1,v_{1},v_{2},v_{3}$ form a basis of the quaternion ring $\unicode[STIX]{x1D6EC}$ . Then $1,v_{1},v_{2},v_{3}$ are said to form a good basis if there exist $a,b,c$ in $R$ such that the matrix

(3.1) $$\begin{eqnarray}A=\left(\begin{array}{@{}ccc@{}}a & v_{3} & v_{2}^{\ast }\\ v_{3}^{\ast } & b & v_{1}\\ v_{2} & v_{1}^{\ast } & c\end{array}\right)\end{eqnarray}$$

is rank one in the sense of Definition 2.1. Equivalently, the basis $1,v_{1},v_{2},v_{3}$ is a good basis if there exists $a,b,c$ in $R$ such that

(3.2) $$\begin{eqnarray}v_{2}v_{3}=av_{1}^{\ast };\quad v_{3}v_{1}=bv_{2}^{\ast };\quad v_{1}v_{2}=cv_{3}^{\ast };\end{eqnarray}$$

and

(3.3) $$\begin{eqnarray}n(v_{1})=bc;\quad n(v_{2})=ca;\quad n(v_{3})=ab.\end{eqnarray}$$

The formulas (3.2) and (3.3) are how good bases are defined in [GL09]. That one can equivalently define good bases in terms of the $A$ of (3.1) being rank one is suggested by [Luc03, Proposition 2.5.2]. Gross and Lucianovic prove the following lemma.

Lemma 3.2 [GL09].

If $1,w_{1},w_{2},w_{3}$ are a basis of $\unicode[STIX]{x1D6EC}$ , then there exist unique $x_{1},x_{2},x_{3}$ in $R$ so that $v_{i}:=w_{i}-x_{i}$ form a good basis.

Let us very briefly sketch a proof. Consider the structure coefficients of $\unicode[STIX]{x1D6EC}$ , namely, those elements $s_{ij}^{k}$ in $R$ so that

$$\begin{eqnarray}w_{i}w_{j}=s_{ij}^{0}+s_{ij}^{1}w_{1}+s_{ij}^{2}w_{2}+s_{ij}^{3}w_{3}.\end{eqnarray}$$

The key point is that the definition of the quaternion ring forces the matrix

$$\begin{eqnarray}S:=\left(\begin{array}{@{}ccc@{}}s_{23}^{1} & s_{23}^{2} & s_{23}^{3}\\ s_{31}^{1} & s_{31}^{2} & s_{31}^{3}\\ s_{12}^{1} & s_{12}^{2} & s_{12}^{3}\end{array}\right)\end{eqnarray}$$

to be symmetric. For example, to see that $s_{23}^{2}=s_{31}^{1}$ , one considers the trace of left multiplication by $w_{3}$ on  $\unicode[STIX]{x1D6EC}$ . Using the expression for $w_{2}w_{3}$ in terms of the structure constants, and conjugating it, one can obtain that the trace is $2\operatorname{tr}(w_{3})+s_{31}^{1}-s_{23}^{2}$ . Consequently, $s_{31}^{1}=s_{23}^{2}$ . One sets $x_{1}=s_{31}^{3}=s_{12}^{2}$ , $x_{2}=s_{12}^{1}=s_{23}^{3}$ , and $x_{3}=s_{23}^{2}=s_{31}^{1}$ . Then $v_{i}=w_{i}-x_{i}$ is the good basis. In fact, if

$$\begin{eqnarray}W=\left(\begin{array}{@{}ccc@{}} & w_{3} & w_{2}^{\ast }\\ w_{3}^{\ast } & & w_{1}\\ w_{2} & w_{1}^{\ast } & \end{array}\right),\end{eqnarray}$$

then using the associativity of the multiplication in $\unicode[STIX]{x1D6EC}$ and that $1,v_{1},v_{2},v_{3}$ is a basis, one can check that $W-S$ is rank one. The uniqueness of the $x_{i}$ is immediate. For a slightly different proof, but more details, consult [GL09].

Suppose now that $v_{1},v_{2},v_{3}$ is a good basis, and $A$ is the rank-one matrix in (3.1). Associated to $A$ , one can form the ternary quadratic form

$$\begin{eqnarray}q_{T}(x,y,z)=ax^{2}+by^{2}+cz^{2}+dyz+ezx+fxy,\end{eqnarray}$$

where $d=\operatorname{tr}(v_{1})$ , $e=\operatorname{tr}(v_{2})$ , $f=\operatorname{tr}(v_{3})$ . Equivalently, if $2$ is not a zero-divisor in $R$ , one can form the half-integral symmetric matrix

(3.4) $$\begin{eqnarray}T=\left(\begin{array}{@{}ccc@{}}a & {\displaystyle \frac{f}{2}} & {\displaystyle \frac{e}{2}}\\ {\displaystyle \frac{f}{2}} & b & {\displaystyle \frac{d}{2}}\\ {\displaystyle \frac{e}{2}} & {\displaystyle \frac{d}{2}} & c\end{array}\right).\end{eqnarray}$$

The strength of the definition of good basis is that $q_{T}$ determines all the multiplication rules in the quaternion ring. For instance, $v_{1}^{2}=dv_{1}-bc$ , $v_{1}v_{2}=cv_{3}^{\ast }=cf-cv_{3}$ , and

$$\begin{eqnarray}v_{2}v_{1}=(e-v_{2})(d-v_{1})-de+ev_{1}+dv_{2}=-de+ev_{1}+dv_{2}+cv_{3}.\end{eqnarray}$$

Conversely, starting with a ternary quadratic form $q_{T}(x,y,z)=ax^{2}+by^{2}+cz^{2}+dyz+ezx+fxy$ on $R^{3}$ , there is the data of a quaternion ring $\unicode[STIX]{x1D6EC}_{T}$ over $R$ , together with a good basis of $\unicode[STIX]{x1D6EC}_{T}$ , as follows. Let us write $N=R^{3}$ , with basis $e_{1}=(1,0,0)$ , $e_{2}=(0,1,0),e_{3}=(0,0,1)$ . One puts on $N$ the quadratic form

$$\begin{eqnarray}q_{T}(xe_{1}+ye_{2}+ze_{3})=ax^{2}+by^{2}+cz^{2}+dyz+ezx+fxy.\end{eqnarray}$$

Then, one can construct the even Clifford algebra $C^{+}((N,q_{T}))$ , defined as usual to be the even-degree part of the tensor algebra of $N$ , modulo the ideal generated by the relations $v^{2}=q_{T}(v)$ for $v\in N$ . $C^{+}((N,q_{T}))$ is a free $R$ -module of rank four; it is the quaternion ring $\unicode[STIX]{x1D6EC}_{T}$ . The map $v_{1}v_{2}\cdots v_{r}\mapsto v_{r}\cdots v_{2}v_{1}$ on the tensor algebra descends to an involution on $\unicode[STIX]{x1D6EC}_{T}$ . This involution is the conjugation $v\mapsto v^{\ast }$ on the quaternion algebra. The preferred choice of basis of $\unicode[STIX]{x1D6EC}_{T}$ is the element 1, together with $v_{1}=e_{2}e_{3},v_{2}=e_{3}e_{1},v_{3}=e_{1}e_{2}$ . It is immediate that this is a good basis, because it is essentially the definition of the Clifford algebra

$$\begin{eqnarray}\left(\begin{array}{@{}c@{}}e_{1}\\ e_{2}\\ e_{3}\end{array}\right)\left(\begin{array}{@{}ccc@{}}e_{1} & e_{2} & e_{3}\end{array}\right)=\left(\begin{array}{@{}ccc@{}}a & v_{3} & v_{2}^{\ast }\\ v_{3}^{\ast } & b & v_{1}\\ v_{2} & v_{1}^{\ast } & c\end{array}\right)\end{eqnarray}$$

that the matrix on the right factorizes as on the left, and thus (heuristically, at least) is rank one.

If $\unicode[STIX]{x1D6EC},(v_{1},v_{2},v_{3})$ , $\unicode[STIX]{x1D6EC}^{\prime },(v_{1}^{\prime },v_{2}^{\prime },v_{3}^{\prime })$ are two quaternion rings over $R$ with good bases, we say a map $\unicode[STIX]{x1D719}:\unicode[STIX]{x1D6EC}\rightarrow \unicode[STIX]{x1D6EC}^{\prime }$ is an isomorphism of good-based quaternion rings if $\unicode[STIX]{x1D719}$ is an isomorphism of $R$ -algebras, preserving the involution, and $\unicode[STIX]{x1D719}(v_{i})=v_{i}^{\prime }$ , $i=1,2,3$ . The constructions associating a ternary quadratic form $q_{T}$ to a good-based quaternion ring and the even Clifford algebra to a ternary quadratic form are inverse to one another. Furthermore, there is a notion of ‘reduced discriminant’ of quaternion rings, see [GL09, p. 8], and similarly for ternary quadratic forms.

Proposition 3.3 (Gross and Lucianovic [GL09]).

Every quaternion ring has a good basis. The even Clifford construction induces a discriminant preserving bijection between ternary quadratic forms $q_{T}$ and isomorphism classes of good-based quaternion rings.

They furthermore give a $\operatorname{GL}_{3}(R)$ action on ternary quadratic forms, and on good bases, and then descend the bijection in the proposition to a bijection between $\operatorname{GL}_{3}(R)$ orbits of ternary quadratic forms and isomorphism classes of quaternion rings over  $R$ .

Because the bijection of Proposition 3.3 preserves discriminant, one obtains the following corollary.

Corollary 3.4. Suppose the half-integral symmetric matrix $T$ corresponds to a maximal order in a quaternion algebra $B$ over $\mathbf{Q}$ . Denote by $D_{B}=\prod _{p\;\text{ram}}p$ the product of the primes $p$ for which $B\otimes \mathbf{Q}_{p}$ is nonsplit. Then $4\det (T)=\pm D_{B}$ .

We now work over $\mathbf{Z}$ . If $\unicode[STIX]{x1D6EC}$ is a quaternion ring, and $x\in \unicode[STIX]{x1D6EC}$ , we denote by $\operatorname{Im}(x)$ the element of $\unicode[STIX]{x1D6EC}\otimes \mathbf{Q}$ with $x=\operatorname{tr}(x)/2+\operatorname{Im}(x)$ . The following lemma is the same as the uniqueness of the $x_{i}$ so that $w_{i}-x_{i}$ form a good basis.

Lemma 3.5. Suppose $B$ a quaternion algebra over $\mathbf{Q}$ , $A_{1},A_{2}\in H_{3}(B)$ . Suppose furthermore that the off-diagonal elements of $A_{1}$ span the three-dimensional $\mathbf{Q}$ vector space $B/(\mathbf{Q}\cdot 1)$ , and similarly for $A_{2}$ . If $A_{1},A_{2}$ are rank-one matrices with $\operatorname{Im}(A_{1})=\operatorname{Im}(A_{2})$ , then $A_{1}=A_{2}$ .

Below we will require the following definition.

Definition 3.6. Suppose $T$ is a half-integral symmetric matrix, with entries as in (3.4). Let $\unicode[STIX]{x1D6EC}_{T},(v_{1},v_{2},v_{3})$ be the associated quaternion ring with good basis. We define $A(T)$ to be the matrix

(3.5) $$\begin{eqnarray}A(T)=\left(\begin{array}{@{}ccc@{}}a & v_{3} & v_{2}^{\ast }\\ v_{3}^{\ast } & b & v_{1}\\ v_{2} & v_{1}^{\ast } & c\end{array}\right).\end{eqnarray}$$

One has that $A(T)$ is rank one in the sense of Definition 2.1.

For further applications, one wants to understand suborders of quaternion orders, together with their good bases. Let $T$ be a half-integral symmetric matrix, and $\unicode[STIX]{x1D6EC}_{T},(v_{1},v_{2},v_{3})$ the associated quaternion ring with its good basis. Suppose $m\in M_{3}(\mathbf{Z})\cap \operatorname{GL}_{3}(\mathbf{Q})$ . Consider the three elements $w_{1},w_{2},w_{3}\in \unicode[STIX]{x1D6EC}_{T}$ defined by the matrix equation

(3.6) $$\begin{eqnarray}\left(\begin{array}{@{}ccc@{}}w_{1} & w_{2} & w_{3}\end{array}\right):=\left(\begin{array}{@{}ccc@{}}v_{1} & v_{2} & v_{3}\end{array}\right)m.\end{eqnarray}$$

That is, $w_{i}=\sum _{j}m_{ji}v_{j}$ . Now consider the lattice $\unicode[STIX]{x1D6EC}_{T}(m)\subset \unicode[STIX]{x1D6EC}_{T}$ spanned by $1,w_{1},w_{2},w_{3}$ . It is a natural question to ask when the lattice $\unicode[STIX]{x1D6EC}_{T}(m)$ is closed under multiplication. Lemma 3.8 below answers this question.

Definition 3.7. For $m\in \operatorname{GL}_{3}$ , denote $c(m)=\det (m)\text{}^{t}m^{-1}$ the cofactor matrix of  $m$ .

Lemma 3.8. Denote by $J_{T}$ the set of $3\times 3$ Hermitian matrices

$$\begin{eqnarray}\left(\begin{array}{@{}ccc@{}}c_{1} & x_{3} & x_{2}^{\ast }\\ x_{3}^{\ast } & c_{2} & x_{1}\\ x_{2} & x_{1}^{\ast } & c_{3}\end{array}\right)\end{eqnarray}$$

with $c_{i}\in \mathbf{Z}$ and $x_{i}\in \unicode[STIX]{x1D6EC}_{T}$ . For $m\in \operatorname{GL}_{3}(\mathbf{Q})$ , denote by $v_{i}^{\prime }$ the off-diagonal entries of $m^{-1}A(T)c(m)$ , and $\unicode[STIX]{x1D6EC}^{\prime }$ the lattice in $\unicode[STIX]{x1D6EC}\otimes \mathbf{Q}$ spanned by  $1$ and $v_{1}^{\prime },v_{2}^{\prime },v_{3}^{\prime }$ . Consider the following four statements:

1. (i) $\unicode[STIX]{x1D6EC}_{T}(m)$ is closed under multiplication;

2. (ii) $m^{-1}Tc(m)$ is half-integral;

3. (iii) $m^{-1}A(T)c(m)=:A^{\prime }$ is in $J_{T}$ ;

4. (iv) $\unicode[STIX]{x1D6EC}^{\prime }$ is closed under multiplication.

Then, if $m\in M_{3}(\mathbf{Z})$ , all four statements are equivalent. If statement (iii) holds, then $m\in M_{3}(\mathbf{Z})$ . In the case these equivalent conditions are satisfied, $1,v_{1}^{\prime },v_{2}^{\prime },v_{3}^{\prime }$ is a good basis of $\unicode[STIX]{x1D6EC}_{T}(m)$ .

Actually, we prove the following stronger statement: statement (iii) implies statement (ii); statements (ii) and (iv) are equivalent. Suppose statement (i) holds, and denote by $A_{0}$ the rank-one matrix associated to the good basis of $\unicode[STIX]{x1D6EC}_{T}(m)$ obtained by translating the $w_{i}$ by integers. Then $A_{0}=A^{\prime }$ , and thus $\unicode[STIX]{x1D6EC}_{T}(m)=\unicode[STIX]{x1D6EC}^{\prime }$ . Hence statement (i) implies statement (ii), equivalently statement (iv), and if $m\in M_{3}(\mathbf{Z})$ , then statement (i) implies statement (iii). If $m\in M_{3}(\mathbf{Z})$ , then statement (iv) implies $\unicode[STIX]{x1D6EC}_{T}(m)=\unicode[STIX]{x1D6EC}^{\prime }$ , and thus statement (iv) implies statement (i).

Proof of Lemma 3.8.

Denote by $V_{3}$ the defining three-dimensional representation of $\operatorname{GL}_{3}$ . One has the equivalence $\wedge ^{2}V_{3}^{\ast }\otimes \det \simeq V_{3}$ . Suppose $x_{1},x_{2},x_{3}$ are indeterminates. In coordinates, this equivalence of representations of $\operatorname{GL}_{3}$ amounts to the identity

$$\begin{eqnarray}\det (m)m^{-1}\left(\begin{array}{@{}ccc@{}} & x_{3} & -x_{2}\\ -x_{3} & & x_{1}\\ x_{2} & -x_{1} & \end{array}\right)\text{}^{t}m^{-1}=\left(\begin{array}{@{}ccc@{}} & x_{3}^{\prime } & -x_{2}^{\prime }\\ -x_{3}^{\prime } & & x_{1}^{\prime }\\ x_{2}^{\prime } & -x_{1}^{\prime } & \end{array}\right),\end{eqnarray}$$

where $x_{i}^{\prime }=\sum _{j}m_{ji}x_{j}$ . Thus since $\operatorname{Im}(A(T))$ is antisymmetric,

(3.7) $$\begin{eqnarray}\operatorname{Im}(A^{\prime })=m^{-1}\operatorname{Im}(A(T))c(m)=\left(\begin{array}{@{}ccc@{}}0 & \operatorname{Im}(w_{3}) & -\operatorname{Im}(w_{2})\\ -\operatorname{Im}(w_{3}) & 0 & \operatorname{Im}(w_{1})\\ \operatorname{Im}(w_{2}) & -\operatorname{Im}(w_{1}) & 0\end{array}\right).\end{eqnarray}$$

Now, immediately statement (iii) implies statement (ii), and statements (ii) and (iv) are equivalent because $m^{-1}Tc(m)$ exactly describes the multiplication rules for $\unicode[STIX]{x1D6EC}^{\prime }$ in $\unicode[STIX]{x1D6EC}\otimes \mathbf{Q}$ . Suppose now that statement (i) holds. Then one may change $w_{1},w_{2},w_{3}$ by integers to make them into a good basis. Hence we have $A_{0},A^{\prime }$ both rank one, and with the same imaginary parts by (3.7). Thus by Lemma 3.5, $A_{0}=A^{\prime }$ . Since by definition the off-diagonal entries of $A_{0}$ generate $\unicode[STIX]{x1D6EC}_{T}(m)$ , and the off-diagonal entries of $A^{\prime }$ generate $\unicode[STIX]{x1D6EC}^{\prime }$ , we get $\unicode[STIX]{x1D6EC}_{T}(m)=\unicode[STIX]{x1D6EC}^{\prime }$ . Hence statement (i) implies statement (iv) or equivalently statement (ii). If $m\in M_{3}(\mathbf{Z})$ , then this $A_{0}$ is in $J_{T}$ , and hence when $m$ is integral, statement (i) implies statement (iii).

Now suppose statement (iv) holds, and $m$ is integral. We have that $\operatorname{Im}(w_{i})=\operatorname{Im}(v_{i}^{\prime })$ . Since we assume statement (iv), the $v_{i}^{\prime }$ have integral trace, and since $m$ is integral, the $w_{i}$ have integral trace. Hence $w_{i}-v_{i}^{\prime }\in \frac{1}{2}\mathbf{Z}$ . We claim the difference is in  $\mathbf{Z}$ . Indeed, write $v_{i}^{\prime }=w_{i}+\ell /2$ , with $\ell$ an integer. Then

$$\begin{eqnarray}2N(v_{i}^{\prime })=2(w_{i}+\ell /2)(w_{i}^{\ast }+\ell /2)=2N(w_{i})+\ell \operatorname{tr}(w_{i})+\ell ^{2}/2.\end{eqnarray}$$

Again since $\unicode[STIX]{x1D6EC}^{\prime }$ is closed under multiplication and $m$ is integral, $v_{i}^{\prime }$ and $w_{i}$ have integral norm. Hence $\ell ^{2}/2$ is an integer, and thus $\ell$ is even. Consequently, if $m$ is integral, then statement (iv) implies $\unicode[STIX]{x1D6EC}^{\prime }=\unicode[STIX]{x1D6EC}_{T}(m)$ , and hence statement (iv) implies statement (i).

Finally, suppose statement (iii) holds. Then by definition, $v_{i}^{\prime }$ is in $\unicode[STIX]{x1D6EC}_{T}$ . On the other hand, we have $\operatorname{Im}(v_{i}^{\prime })=\sum _{j}m_{ji}\operatorname{Im}(v_{j})$ , and thus $v_{i}^{\prime }=a_{i}+\sum _{j}m_{ji}v_{j}$ for some $a_{i}$ in  $\mathbf{Q}$ . Since $v_{i}^{\prime }$ is in $\unicode[STIX]{x1D6EC}_{T}$ and $1,v_{1},v_{2},v_{3}$ are a basis on $\unicode[STIX]{x1D6EC}_{T}$ , it follows that the $a_{i}$ and $m_{ji}$ are in  $\mathbf{Z}$ , and thus $m\in M_{3}(\mathbf{Z})$ , as desired.◻

Remark 3.9. The equivalence of statements (i), (ii) and (iv) in Lemma 3.8 can also be deduced easily from [Voi, Proposition 22.4.12]. We thank John Voight for pointing this out to the author.

Remark 3.10. Lemma 3.8 suggests the following partial order on nondegenerate ternary quadratic forms. Suppose $q_{1}(x,y,z)$ and $q_{2}(x,y,z)$ are integral ternary quadratic forms, with associated half-integral symmetric matrices $T_{1}$ and $T_{2}$ , respectively. We say that $q_{i}$ is nondegenerate to mean $\det (T_{i})\neq 0$ . Define a partial order on the set of nondegenerate integral ternary quadratic forms by $q_{1}\leqslant q_{2}$ if there exists $m\in M_{3}(\mathbf{Z})\cap \operatorname{GL}_{3}(\mathbf{Q})$ such that $T_{1}=m^{-1}T_{2}c(m)$ . Suppose $\unicode[STIX]{x1D6EC}_{i}$ is the even Clifford algebra associated to $q_{i}(x,y,z)$ . Then Lemma 3.8 implies that there is an injective ring map $\unicode[STIX]{x1D6EC}_{1}\rightarrow \unicode[STIX]{x1D6EC}_{2}$ if and only if $q_{1}\leqslant q_{2}$ .

4 A Dirichlet series for the spin $L$ -function

In this section we combine Lemma 3.8 with a result of Evdokimov [Evd84, Theorem 3] to give a Dirichlet series for the spin $L$ -function of level-one Siegel modular forms on $\operatorname{GSp}_{6}$ . Most of the work has already been done in [Evd84]. Namely, [Evd84, Theorem 3] very nearly gives a Dirichlet series for this $L$ -function, except this result of Evdokimov involves an ‘error term’ that must be removed on the way to proving Theorem 1.2. ¹ However, it turns out that this error term can be trivialized by applying the arithmetic invariant theory of ternary quadratic forms, in the form of Lemma 3.8, together with the fact that $B_{0}$ is a maximal order. We first recall precisely [Evd84, Theorem 3], and then apply Lemma 3.8 to obtain Corollary 4.2, which gives the desired Dirichlet series.

To state [Evd84, Theorem 3], we need to introduce some notation, largely following [Evd84]. Denote by $P_{6}$ the Siegel parabolic of $\operatorname{GSp}_{6}$ , i.e., the set of elements $(\!\begin{smallmatrix}A & B\\ C & D\end{smallmatrix}\!)$ in $\operatorname{GSp}_{6}$ with $C=0$ . Set $S_{p}=\{M\in \operatorname{GSp}_{6}(\mathbf{Z}[p^{-1}]):\unicode[STIX]{x1D708}(M)\in p^{\mathbf{Z}}\}$ , $\unicode[STIX]{x1D6E4}_{0}=\operatorname{Sp}_{6}(\mathbf{Z})\cap P_{6}(\mathbf{Q})$ , and $S_{0,p}=S_{p}\cap P_{6}(\mathbf{Q})$ . Denote by $L_{0,p}$ the double coset ring of $\unicode[STIX]{x1D6E4}_{0}$ in $S_{0,p}$ .

We define some elements of $L_{0,p}$ . Set

$$\begin{eqnarray}\unicode[STIX]{x1D6F1}_{0}(p)=\unicode[STIX]{x1D6E4}_{0}\operatorname{diag}(p,p,p,1,1,1)\unicode[STIX]{x1D6E4}_{0}\quad \text{and}\quad \unicode[STIX]{x1D6F1}_{1}(p)=\unicode[STIX]{x1D6E4}_{0}\operatorname{diag}(p,p,1,1,1,p)\unicode[STIX]{x1D6E4}_{0}.\end{eqnarray}$$

Define

$$\begin{eqnarray}\unicode[STIX]{x1D6F1}_{2,0}^{0}(p)=\unicode[STIX]{x1D6E4}_{0}\operatorname{diag}(p^{2},p,p,1,p,p)\unicode[STIX]{x1D6E4}_{0}\quad \text{and}\quad \unicode[STIX]{x1D6F1}_{3,0}^{0}(p)=\unicode[STIX]{x1D6E4}_{0}\operatorname{diag}(p,p,p,p,p,p)\unicode[STIX]{x1D6E4}_{0}.\end{eqnarray}$$

For integers $0\leqslant r\leqslant i$ , set $S_{i}^{r}$ to be the set of $A\in M_{i}(\mathbf{Z})$ with $^{t}A=A$ , and $\text{rank}_{\mathbf{F}_{p}}A=r$ . Define an equivalence relation on $S_{i}^{r}$ by $A\sim A^{\prime }$ if there exists $U\in \operatorname{GL}_{i}(\mathbf{Z})$ such that $A^{\prime }\equiv ^{t}UAU$ modulo  $p$ . Now, define

where $A=(\!\begin{smallmatrix}a_{22} & a_{23}\\ a_{23} & a_{33}\end{smallmatrix}\!)$ . Similarly, define

where $A=(a_{ij})$ . Set $\unicode[STIX]{x1D6F1}_{1}^{2}(p)=p(\unicode[STIX]{x1D6F1}_{2,0}^{1}(p)+\unicode[STIX]{x1D6F1}_{2,0}^{0}(p))$ and $\unicode[STIX]{x1D6F1}_{1}^{3}(p)=p^{3}(\unicode[STIX]{x1D6F1}_{3,0}^{1}(p)+\unicode[STIX]{x1D6F1}_{3,0}^{0}(p))\unicode[STIX]{x1D6F1}_{0}(p)$ .

Denote by ${\mathcal{F}}_{k}$ the space of holomorphic functions $f:{\mathcal{H}}_{3}\rightarrow \mathbf{C}$ that satisfy $f((AZ+B)D^{-1})=\det (D)^{k}f(Z)$ for all $(\!\begin{smallmatrix}A & B\\ & D\end{smallmatrix}\!)\in P_{6}(\mathbf{Q})\cap \operatorname{Sp}_{6}(\mathbf{Z})$ . The ring $L_{0,p}$ acts on ${\mathcal{F}}_{k}$ as follows. Suppose $\unicode[STIX]{x1D6E4}_{0}h\unicode[STIX]{x1D6E4}_{0}\subseteq S_{0,p}$ is a double coset. Then $\unicode[STIX]{x1D6E4}_{0}h\unicode[STIX]{x1D6E4}_{0}$ has a single coset decomposition

$$\begin{eqnarray}\unicode[STIX]{x1D6E4}_{0}h\unicode[STIX]{x1D6E4}_{0}=\bigsqcup _{i}\unicode[STIX]{x1D6E4}_{0}\left(\begin{array}{@{}cc@{}}A_{i} & B_{i}\\ & D_{i}\end{array}\right),\end{eqnarray}$$

$\unicode[STIX]{x1D708}((\!\begin{smallmatrix}A_{i} & B_{i}\\ & D_{i}\end{smallmatrix}\!))=p^{\unicode[STIX]{x1D6FF}_{i}}$ . One defines

$$\begin{eqnarray}(f|\unicode[STIX]{x1D6E4}_{0}h\unicode[STIX]{x1D6E4}_{0})(Z)=\mathop{\sum }_{i}p^{(3k-6)\unicode[STIX]{x1D6FF}_{i}}(\det (D_{i}))^{-k}f((A_{i}Z+B_{i})D_{i}^{-1}).\end{eqnarray}$$

Suppose $\unicode[STIX]{x1D719}$ is as in Definition 1.1, and $f_{\unicode[STIX]{x1D719}}(Z)=\sum _{T^{\prime }>0}a(T^{\prime })e^{2\unicode[STIX]{x1D70B}i\operatorname{tr}(T^{\prime }Z)}$ is the associated level-one Siegel modular form. Let us write $L_{\text{classical}}(\unicode[STIX]{x1D70B},\text{Spin},s)=L(\unicode[STIX]{x1D70B},\text{Spin},s-3r+3)$ for the classically normalized Spin $L$ -function of $\unicode[STIX]{x1D70B}$ . Define $Q_{p}^{1}(z)=1-\unicode[STIX]{x1D6F1}_{0}(p)z$ and $Q_{p}^{2}(z)=1-\unicode[STIX]{x1D6F1}_{1}(p)z+\unicode[STIX]{x1D6F1}_{1}^{2}(p)z^{2}-\unicode[STIX]{x1D6F1}_{1}^{3}(p)z^{3}$ , elements of $L_{0,p}[z]$ , the polynomial ring over $L_{0,p}$ in the indeterminate  $z$ . For $m\in \operatorname{GL}_{3}(\mathbf{Q})$ , define

$$\begin{eqnarray}\unicode[STIX]{x1D6EF}_{T}(m)=\left\{\begin{array}{@{}ll@{}}1\quad & \text{if}~m^{-1}Tc(m)\in M_{3}(\mathbf{Z}),\\ 0\quad & \text{otherwise}.\end{array}\right.\end{eqnarray}$$

The following theorem is essentially [Evd84, Theorem 3].

Theorem 4.1 (Evdokimov).

Let the notations be as above. Set $M_{3}^{+}(\mathbf{Z})$ to be the elements of $M_{3}(\mathbf{Z})$ with positive determinant. Then

(4.1) $$\begin{eqnarray}\mathop{\sum }_{\unicode[STIX]{x1D706}\geqslant 1,m\in M_{3}^{+}(\mathbf{Z})/\!\operatorname{SL}_{3}(\mathbf{Z})}\frac{\unicode[STIX]{x1D6EF}_{T}(m)}{\unicode[STIX]{x1D706}^{s}\det (m)^{s-2r+3}}a(\unicode[STIX]{x1D706}m^{-1}Tc(m))=C_{f_{\unicode[STIX]{x1D719}}}(s,T)\frac{L_{\text{classical}}(\unicode[STIX]{x1D70B},\text{Spin},s)}{\unicode[STIX]{x1D701}_{D_{B}}(2s-6r+8)\unicode[STIX]{x1D701}(2s-6r+6)},\end{eqnarray}$$

where

$$\begin{eqnarray}\unicode[STIX]{x1D701}_{D_{B}}(s)=\mathop{\prod }_{p<\infty ,p\nmid D_{B}}(1-p^{-s})^{-1}\end{eqnarray}$$

and $C_{f_{\unicode[STIX]{x1D719}}}(s,T)$ is the $T$ th Fourier coefficient of

$$\begin{eqnarray}f_{\unicode[STIX]{x1D719}}\biggl|\mathop{\prod }_{p|4\det (T)}Q_{p}^{1}(p^{-s})Q_{p}^{2}(p^{-s}).\end{eqnarray}$$

The function $C_{f_{\unicode[STIX]{x1D719}}}(s,T)$ in Theorem 4.1 is the ‘error term’ that we eluded to above. By applying Lemma 3.8, we will see that the hypothesis that $T$ corresponds to a maximal order implies that $C_{f_{\unicode[STIX]{x1D719}}}(s,T)=a(T)$ is independent of  $s$ . Hence under this assumption, the left-hand side of (4.1) yields a Dirichlet series for the Spin $L$ -function.

If ${\mathcal{O}}$ is an order in some quaternion algebra over $\mathbf{Q}$ ramified at infinity, then ${\mathcal{O}}=\unicode[STIX]{x1D6EC}_{T^{\prime }}$ is the even Clifford algebra associated to some half-integral, symmetric, positive-definite matrix $T^{\prime }$ . We write $a({\mathcal{O}}):=a(T^{\prime })$ . Since $a(T^{\prime })=a(k^{-1}T^{\prime }c(k))$ for $k\in \operatorname{SL}_{3}(\mathbf{Z})$ , the notation $a({\mathcal{O}})$ is independent of the choice of $T^{\prime }$ , and thus well defined. We have the following corollary of Lemma 3.8 and Theorem 4.1.

Corollary 4.2. Suppose $\unicode[STIX]{x1D70B}$ is as in Definition 1.1, $f_{\unicode[STIX]{x1D719}}(Z)=\sum _{T^{\prime }>0}a(T^{\prime })e^{2\unicode[STIX]{x1D70B}i\operatorname{tr}(T^{\prime }Z)}$ is the Fourier expansion of the associated Siegel modular form, and the even Clifford algebra $B_{0}$ associated to $T$ is a maximal order in $B_{0}\otimes \mathbf{Q}$ . Then

(4.2) $$\begin{eqnarray}\mathop{\sum }_{\unicode[STIX]{x1D706}\geqslant 1,{\mathcal{O}}\subseteq B_{0}}\frac{a(\mathbf{Z}+\unicode[STIX]{x1D706}{\mathcal{O}})}{\unicode[STIX]{x1D706}^{s}[B_{0}:{\mathcal{O}}]^{s-2r+3}}=a(B_{0})\frac{L_{\text{classical}}(\unicode[STIX]{x1D70B},\text{Spin},s)}{\unicode[STIX]{x1D701}_{D_{B}}(2s-6r+8)\unicode[STIX]{x1D701}(2s-6r+6)},\end{eqnarray}$$

where $[B_{0}:{\mathcal{O}}]$ denotes the index of ${\mathcal{O}}$ in $B_{0}$ .

Proof. Applying Lemma 3.8, the left-hand side of (4.2) is seen to be equal to the left-hand side of (4.1).

The key point of the corollary is that, as mentioned above, $B_{0}$ being a maximal order implies $C_{f_{\unicode[STIX]{x1D719}}}(s,T)=a(T)=a(B_{0})$ . To see this equality, consider the action of the operators $\unicode[STIX]{x1D6F1}_{0}(p)$ , $\unicode[STIX]{x1D6F1}_{1}(p)$ , $\unicode[STIX]{x1D6F1}_{1}^{2}(p)$ and $\unicode[STIX]{x1D6F1}_{1}^{3}(p)$ on the Fourier coefficients of $f_{\unicode[STIX]{x1D719}}$ . Suppose $\unicode[STIX]{x1D6F1}(p)$ is one of these operators. For a quaternion order ${\mathcal{O}}$ and a Siegel modular form $f$ , set $a({\mathcal{O}},f|\unicode[STIX]{x1D6F1}(p))$ the ${\mathcal{O}}$ Fourier coefficient of $f|\unicode[STIX]{x1D6F1}(p)$ . Applying Lemma 3.8, one verifies that $a({\mathcal{O}},f|\unicode[STIX]{x1D6F1}(p))$ is a linear combination of $a({\mathcal{O}}^{\prime },f)$ for certain orders ${\mathcal{O}}^{\prime }\supseteq {\mathcal{O}}$ with $[{\mathcal{O}}^{\prime }:{\mathcal{O}}]=p,p^{2}$ or $p^{3}$ . Since $B_{0}$ is a maximal order, $a(B_{0},f|\unicode[STIX]{x1D6F1}(p))=0$ , and the equality $C_{f_{\unicode[STIX]{x1D719}}}(s,T)=a(B_{0})$ follows.◻

Remark 4.3. In [Evd84, Theorem 4], Evdokimov also uses his result [Evd84, Theorem 3] to obtain a Dirichlet series for the Spin $L$ -function, but only under the assumption that $f_{\unicode[STIX]{x1D719}}$ satisfies $a(T^{\prime })=0$ for all $T^{\prime }$ with $4\det (T^{\prime })$ strictly dividing $4\det (T)$ .

5 The Rankin–Selberg integral

In this section we define the map $\operatorname{GSp}_{6}\rightarrow \text{G}$ and the Eisenstein series on  $\text{G}$ . We also give the global Rankin–Selberg convolution and unfold it.

5.1 Construction of global integral

We first embed $\operatorname{GSp}_{6}$ into $\text{G}$ , preserving the similitude. Denote by $W_{6}$ the defining six-dimensional representation of $\operatorname{GSp}_{6}$ , and $\langle \;,\;\rangle$ the invariant symplectic form. We let $\operatorname{GSp}_{6}$ act on $W_{6}$ on the right. In matrices, $\operatorname{GSp}_{6}$ is the set of $g\in \operatorname{GL}_{6}$ satisfying $gJ_{6}\text{}^{t}g=\unicode[STIX]{x1D708}(g)J_{6}$ , where $J_{6}=(\!\begin{smallmatrix} & 1_{3}\\ -1_{3} & \end{smallmatrix}\!)$ and the similitude $\unicode[STIX]{x1D708}(g)\in \operatorname{GL}_{1}$ . The action of $\operatorname{GSp}_{6}$ on $W_{6}$ is then the usual right action of matrices on row vectors.

Pick a symplectic basis $e_{1},e_{2},e_{3},f_{1},f_{2},f_{3}$ of $W_{6}$ ; that is, $\langle e_{i},f_{j}\rangle =\unicode[STIX]{x1D6FF}_{ij}$ . We fix a quaternion algebra $B$ over $\mathbf{Q}$ that is ramified at infinity. For such a $B$ , denote by $W=\mathbf{Q}\,\oplus \,H_{3}(B)\oplus H_{3}(B)\oplus \mathbf{Q}$ Freudenthal’s space, as considered in § 2. Now consider the linear map $W\rightarrow (\bigwedge ^{3}W_{6}\otimes \unicode[STIX]{x1D708}^{-1})\otimes _{\mathbf{Q}}B$ defined as follows: set $e_{i}^{\ast }=e_{i+1}\wedge e_{i+2}$ and $f_{i}^{\ast }=f_{i+1}\wedge f_{i+2}$ with the indices taken modulo 3. Then we map $(a,b,c,d)$ in $W$ to

$$\begin{eqnarray}ae_{1}\wedge e_{2}\wedge e_{3}+\biggl(\mathop{\sum }_{i,j}b_{ij}e_{i}^{\ast }\wedge f_{j}\biggr)+\biggl(\mathop{\sum }_{i,j}c_{ij}f_{i}^{\ast }\wedge e_{j}\biggr)+df_{1}\wedge f_{2}\wedge f_{3}.\end{eqnarray}$$

With this definition, and the natural action of $\operatorname{GSp}_{6}$ on $\bigwedge ^{3}W_{6}\otimes \unicode[STIX]{x1D708}^{-1}$ , $\operatorname{GSp}_{6}$ preserves the image of $W$ , and preserves the symplectic and quartic form on $W$ , up to similitude. These facts are checked in the following lemma.

Lemma 5.1. Identify $W$ with its image in $(\bigwedge ^{3}W_{6}\otimes \unicode[STIX]{x1D708}^{-1})\otimes _{\mathbf{Q}}B$ . Then, the element $(\!\begin{smallmatrix}1 & X\\ & 1\end{smallmatrix}\!)$ of $\operatorname{GSp}_{6}$ acts on $W$ as $n(X)$ , and the element $(\!\begin{smallmatrix}0 & 1\\ -1 & 0\end{smallmatrix}\!)$ acts on $W$ via $(a,b,c,d)\mapsto (-d,c,-b,a)$ . If $(\!\begin{smallmatrix}m & \\ & n\end{smallmatrix}\!)\in \operatorname{GSp}_{6}$ has similitude $\unicode[STIX]{x1D708}$ , then

$$\begin{eqnarray}(a,b,c,d)(\!\begin{smallmatrix}m & \\ & n\end{smallmatrix}\!)=\unicode[STIX]{x1D708}^{-1}(\det (m)a,\det (m)m^{-1}bn,\det (n)n^{-1}cm,\det (n)\,d).\end{eqnarray}$$

It follows that the inclusion $W\rightarrow (\bigwedge ^{3}W_{6}\,\otimes \,\unicode[STIX]{x1D708}^{-1})\,\otimes _{\mathbf{Q}}B$ induces an inclusion $\operatorname{GSp}_{6}\rightarrow \text{G}$ , preserving similitudes.

Proof. The formula for the action of $(\!\begin{smallmatrix} & 1\\ -1 & \end{smallmatrix}\!)$ is immediate. We check some of the other formulas.

Consider the action of $(\!\begin{smallmatrix}m & \\ & n\end{smallmatrix}\!)$ . One first computes that under the action of $m$ , $e_{i}^{\ast }\mapsto \sum _{j}c(m)_{ij}e_{j}^{\ast }$ , where $c(m)=\det (m)\text{}^{t}m^{-1}$ is the cofactor matrix of  $m$ . Ignoring the twist by the similitude $\unicode[STIX]{x1D708}^{-1}$ in $\bigwedge ^{3}W_{6}\otimes \unicode[STIX]{x1D708}^{-1}$ , one then obtains

$$\begin{eqnarray}\displaystyle \mathop{\sum }_{i,j}b_{ij}e_{i}^{\ast }\wedge f_{j} & \mapsto & \displaystyle \mathop{\sum }_{i,j,t,s}b_{ij}c(m)_{it}n_{js}e_{t}^{\ast }\wedge f_{s}\nonumber\\ \displaystyle & = & \displaystyle \mathop{\sum }_{t,s}(\text{}^{t}c(m)bn)_{ts}e_{t}^{\ast }\wedge f_{s}.\nonumber\end{eqnarray}$$

This proves the formula for the action of $(\!\begin{smallmatrix}m & \\ & n\end{smallmatrix}\!)$ on the $b$ values of $(a,b,c,d)$ . The action on the $c$ values is similar, and the action on the $a$ values and $b$ values are trivial.

For $X$ in $H_{3}(B)$ , denote by $L(X)$ the logarithm, inside $\text{End}(W)$ , of the element $n(X)$ of $\operatorname{GL}(W)$ . That is, set $L(X)=\sum _{m\geqslant 1}(-1)^{m-1}((n(X)-1)^{m}/m)$ . From the definition (2.5) of $n(X)$ , one sees $(n(X)-1)^{4}=0$ , and thus the sum defining $L(X)$ terminates after four terms. One gets

$$\begin{eqnarray}(a,b,c,d)L(X)=(0,aX,b\times X,\operatorname{tr}(c,X))\end{eqnarray}$$

and $n(X)=1+L(X)+L(X)^{2}/2+L(X)^{3}/6$ .

Similarly, for $(\!\begin{smallmatrix}1 & X\\ & 1\end{smallmatrix}\!)$ in $\operatorname{GSp}_{6}$ , denote by $n_{6}(X)$ its action on $W_{6}$ , $n^{\prime }(X)$ its action on $(\bigwedge ^{3}W_{6}\otimes \unicode[STIX]{x1D708}^{-1})\otimes B$ , and $L_{6}(X)$ , respectively $L^{\prime }(X)$ , the logarithms of these actions. Since $n_{6}(X)=1+L_{6}(X)$ , and the action of $n(X)$ on $\bigwedge ^{3}W_{6}$ is defined by cubic polynomials in the entries of $(\!\begin{smallmatrix}1 & X\\ & 1\end{smallmatrix}\!)$ , one obtains $n^{\prime }(X)=1+L^{\prime }(X)+L^{\prime }(X)^{2}/2+L^{\prime }(X)^{3}/6.$ Hence to check that $n^{\prime }(X)$ restricts to $n(X)$ on $W$ , it suffices to check the analogous statement for $L^{\prime }(X)$ and $L(X)$ .

So, suppose $X$ is a $3\times 3$ symmetric matrix. We consider the action $(0,b,0,0)L^{\prime }(X)$ . One has

$$\begin{eqnarray}(e_{i}^{\ast }\wedge f_{j})L^{\prime }(X)=\mathop{\sum }_{k}X_{i+1,k}f_{k}\wedge e_{i+2}\wedge f_{j}+X_{i+2,k}e_{i+1}\wedge f_{k}\wedge f_{j},\end{eqnarray}$$

where $X_{a,b}$ denotes the $a,b$ matrix entry of $X$ . Hence

(5.1) $$\begin{eqnarray}\displaystyle \biggl(\mathop{\sum }_{ij}b_{ij}e_{i}^{\ast }\wedge f_{j}\biggr)L^{\prime }(X) & = & \displaystyle \mathop{\sum }_{j,k,\ell }(b_{\ell +1,j}X_{\ell -1,k}-X_{\ell +1,k}b_{\ell -1,j})f_{j}\wedge f_{k}\wedge e_{\ell }\nonumber\\ \displaystyle & = & \displaystyle \mathop{\sum }_{\unicode[STIX]{x1D6FC},\unicode[STIX]{x1D6FD}}V_{\unicode[STIX]{x1D6FC},\unicode[STIX]{x1D6FD}}f_{\unicode[STIX]{x1D6FC}}^{\ast }\wedge e_{\unicode[STIX]{x1D6FD}}\end{eqnarray}$$

for some element $V$ of $M_{3}(B)$ . For a general element $Y$ of $M_{3}(B)$ , denote by $c_{i}(Y),a_{i}(Y),a_{i}(Y)^{\prime }$ the coordinates of  $Y$ :

$$\begin{eqnarray}Y=\left(\begin{array}{@{}ccc@{}}c_{1}(Y) & a_{3}(Y) & a_{2}(Y)^{\prime }\\ a_{3}(Y)^{\prime } & c_{2}(Y) & a_{1}(Y)\\ a_{2}(Y) & a_{1}(Y)^{\prime } & c_{3}(Y)\end{array}\right).\end{eqnarray}$$

Then (5.1) yields

(5.2) $$\begin{eqnarray}\begin{array}{@{}rcl@{}}c_{t}(V)\ & =\ & c_{t-1}(X)c_{t+1}(b)+c_{t+1}(X)c_{t-1}(b)-(a_{t}(X),a_{t}(b)),\\ a_{t}(V)\ & =\ & a_{t+1}^{\prime }(b)a_{t-1}^{\prime }(X)+a_{t+1}^{\prime }(X)a_{t-1}^{\prime }(b)-(c_{t}(X)a_{t}(b)+c_{t}(b)a_{t}(X)),\\ a_{t}^{\prime }(V)\ & =\ & a_{t+1}(b)a_{t-1}(X)+a_{t+1}(X)a_{t-1}(b)-(c_{t}(X)a_{t}^{\prime }(b)+c_{t}(b)a_{t}^{\prime }(X)).\end{array}\end{eqnarray}$$

The indices in (5.2) are taken modulo 3. Linearizing (2.3) for the map $\#$ yields exactly the formulas (5.2), and thus $V=b\times X$ . Hence $(0,b,0,0)L^{\prime }(X)=(0,0,b\times X,0)$ .

The formulas $(a,0,0,0)L^{\prime }(X)=(0,aX,0,0)$ and $(0,0,c,d)L^{\prime }(X)=(0,0,0,\operatorname{tr}(c,X))$ are much easier to establish. Hence $L^{\prime }(X)$ preserves $W$ , and restricts to the action of $L(X)$ . This completes the proof of the lemma.◻

We need to define an integral structure on $W$ . To do this, first we fix a maximal order $B_{0}$ in  $B$ . Then

(5.3) $$\begin{eqnarray}W(\mathbf{Z}):=W(\mathbf{Q})\cap \{(\wedge ^{3}W_{6}(\mathbf{Z})\otimes \unicode[STIX]{x1D708}^{-1})\otimes _{\boldsymbol{ Z}}B_{0}\},\end{eqnarray}$$

and for a finite prime $p$ , $W(\mathbf{Z}_{p}):=W(\mathbf{Z})\otimes \mathbf{Z}_{p}$ . Define $J_{0}$ to be the lattice in $H_{3}(B)$ consisting of those

(5.4) $$\begin{eqnarray}\left(\begin{array}{@{}ccc@{}}c_{1} & x_{3} & x_{2}^{\ast }\\ x_{3}^{\ast } & c_{2} & x_{1}\\ x_{2} & x_{1}^{\ast } & c_{3}\end{array}\right)\end{eqnarray}$$

with $c_{i}\in \mathbf{Z}$ and $x_{i}\in B_{0}$ . Then $(a,b,c,d)\in W(\mathbf{Q})$ is in $W(\mathbf{Z})$ if and only if $a,d\in \mathbf{Z}$ and $b,c\in J_{0}$ . It is clear from (5.3) that $\operatorname{GSp}_{6}(\mathbf{Z}_{p})$ stabilizes $W(\mathbf{Z}_{p})$ for all finite primes  $p$ .

We now define the Eisenstein series on $\text{G}$ . Recall the rank-one element $f=(0,0,0,1)$ of $W$ . Denote by $P$ the parabolic subgroup of $\text{G}$ stabilizing the line $\mathbf{Q}f$ . The Eisenstein series will be associated to a character of  $P$ . To define this Eisenstein series, we select a particular Schwartz–Bruhat function $\unicode[STIX]{x1D6F7}=\prod _{v}\unicode[STIX]{x1D6F7}_{v}$ on $W(\mathbf{A})$ . At finite primes, $\unicode[STIX]{x1D6F7}_{p}$ is the characteristic function of $W(\mathbf{Z}_{p})$ . In § 6.2 will define a Schwartz function $\unicode[STIX]{x1D6F7}_{\infty ,2r}$ on $W(\mathbf{R})$ that is appropriate for cusp forms $\unicode[STIX]{x1D719}$ that correspond to a Siegel modular form of weight $2r$ . The Schwartz function $\unicode[STIX]{x1D6F7}_{\infty ,2r}$ will be a Gaussian times a polynomial of degree $2r$ . We set

$$\begin{eqnarray}f^{\unicode[STIX]{x1D6F7}}(g,s)=|\unicode[STIX]{x1D708}(g)|^{s}\int _{\operatorname{GL}_{1}(\mathbf{A})}\unicode[STIX]{x1D6F7}(tfg)|t|^{2s}\,dt,\end{eqnarray}$$

and

$$\begin{eqnarray}E^{\unicode[STIX]{x1D6F7}}(g,s)=\mathop{\sum }_{\unicode[STIX]{x1D6FE}\in P(\mathbf{Q})\backslash \!\text{G}(\mathbf{Q})}f^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D6FE}g,s).\end{eqnarray}$$

This is not yet the normalized Eisenstein series. Assume $\unicode[STIX]{x1D6F7}_{\infty ,2r}$ is chosen as in § 6.2, and denote by $D_{B}=\prod _{p<\infty ,\text{ramified}}p$ the product of the finite primes ramified in  $B$ . Then the normalized Eisenstein series is

$$\begin{eqnarray}\displaystyle E_{2r}^{\ast }(g,s) & = & \displaystyle \unicode[STIX]{x1D70B}^{-(s+r)}D_{B}^{s}\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s+2r-4)\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s+2r-8)\biggl(\mathop{\prod }_{p<\infty ~\text{split}}\unicode[STIX]{x1D701}_{p}(2s-2)\unicode[STIX]{x1D701}_{p}(2s-4)\biggr)\nonumber\\ \displaystyle & & \displaystyle \times \,\biggl(\mathop{\prod }_{p<\infty ~\text{ramified}}\unicode[STIX]{x1D701}_{p}(2s-4)\biggr)E^{\unicode[STIX]{x1D6F7}}(g,s).\nonumber\end{eqnarray}$$

Here, as usual, $\unicode[STIX]{x1D6E4}_{\mathbf{R}}(s)=\unicode[STIX]{x1D70B}^{-s/2}\unicode[STIX]{x1D6E4}(s/2)$ . The following theorem is proved in § 7.

Theorem 5.2. The Eisenstein series $E_{2r}^{\ast }(g,s)$ has finitely many poles, all contained in the set $\frac{1}{2}\mathbf{Z}\cap [0,5]$ , and satisfies the functional equation $E_{2r}^{\ast }(g,s)=E_{2r}^{\ast }(g,5-s)$ .

For a cusp form $\unicode[STIX]{x1D719}$ associated to a level-one, holomorphic Siegel modular form of weight $2r$ , the global integral is

$$\begin{eqnarray}I_{2r}^{\ast }(\unicode[STIX]{x1D719},s)=\int _{\operatorname{ GSp}_{6}(\mathbf{Q})Z(\mathbf{A})\backslash \!\operatorname{GSp}_{6}(\mathbf{A})}\unicode[STIX]{x1D719}(g)E_{2r}^{\ast }(g,s)\,dg.\end{eqnarray}$$

By Theorem 5.2, $I_{2r}^{\ast }(\unicode[STIX]{x1D719},s)=I_{2r}^{\ast }(\unicode[STIX]{x1D719},5-s)$ .

Remark 5.3. This is the first place where defining the group $\text{G}$ in terms of the Freudenthal construction gives one a substantial technical advantage. Namely, because we have defined $\text{G}$ in terms of the representation  $W$ , we were able to construct the Eisenstein section $f^{\unicode[STIX]{x1D6F7}}$ in terms of a characteristic function on  $W$ . We will see in § 6 that this definition plays a significant role in easing the computation.

5.2 Unfolding

To unfold this integral, we rely on the following lemma.

Lemma 5.4. Every rank-one element of $W$ has a $\operatorname{GSp}_{6}(\mathbf{Q})$ translate with $d=1$ .

Proof. Suppose our rank-one element $v$ is $(a,b,c,d)$ . If either $a$ or $d$ is nonzero, we are done. So, suppose $v=(0,b,c,0)$ . Then $(0,b,c,0)n(X)=(0,b,c+b\times X,\operatorname{tr}(c,X)+\operatorname{tr}(b,X^{\sharp }))$ . Since $v$ is rank one, by Proposition 2.5, $b^{\sharp }=c^{\sharp }=0$ . From the formula (2.3) for $\sharp$ , since $B$ is not split, we cannot have both $c$ purely imaginary and $c^{\sharp }=0$ , unless $c=0$ . The same for $b$ . Without loss of generality, assume $c\neq 0$ . Since $c$ is not purely imaginary, there exists $X\in H_{3}(\mathbf{Q})$ with $\operatorname{tr}(c,X)\neq 0$ . Hence $\unicode[STIX]{x1D706}\operatorname{tr}(c,X)+\unicode[STIX]{x1D706}^{2}\operatorname{tr}(b,X^{\sharp })$ is not zero for some $\unicode[STIX]{x1D706}$ in  $\mathbf{Q}$ , proving the lemma.◻

To do the unfolding, we need to understand the double coset space $P(\mathbf{Q})\backslash \text{G}(\mathbf{Q})/\!\operatorname{GSp}_{6}(\mathbf{Q})$ , or equivalently, the $\operatorname{GSp}_{6}(\mathbf{Q})$ orbits on the rank-one lines. Recall that $T$ denotes a half-integral symmetric $3\times 3$ matrix that corresponds to the order $B_{0}$ , and set $A(T)$ to be the rank-one matrix in $H_{3}(B)$ from Lemma 3.8. Define $f_{{\mathcal{O}}}=(0,0,A(T),0)$ in $W$ . By Proposition 2.5, $f_{{\mathcal{O}}}$ is rank one. If $v$ is a rank-one element of $W(\mathbf{Q})$ , an orbit $\mathbf{Q}v\operatorname{GSp}_{6}(\mathbf{Q})$ is said to be negligible if the stabilizer of $\mathbf{Q}v$ in $\operatorname{GSp}_{6}(\mathbf{Q})$ contains the unipotent radical of a parabolic subgroup of $\operatorname{GSp}_{6}$ .

Proposition 5.5. The line $\mathbf{Q}f_{{\mathcal{O}}}$ represents the unique open orbit in $P(\mathbf{Q})\backslash \text{G}(\mathbf{Q})/\!\operatorname{GSp}_{6}(\mathbf{Q})$ . There are an infinite number of other orbits, but they are all negligible. The stabilizer of the line $\mathbf{Q}f_{{\mathcal{O}}}$ is the set of elements of $\operatorname{GSp}_{6}$ of the form $g=(\!\begin{smallmatrix}\unicode[STIX]{x1D706}_{1} & \\ & \unicode[STIX]{x1D706}_{2}\end{smallmatrix}\!)(\!\begin{smallmatrix}1 & U\\ & 1\end{smallmatrix}\!)$ with $\unicode[STIX]{x1D706}_{1},\unicode[STIX]{x1D706}_{2}$ in $\operatorname{GL}_{1}(\mathbf{Q})$ and $\operatorname{tr}(TU)=0$ .

Remark 5.6. This is another spot where defining the group $\text{G}$ in terms of the Freudenthal construction gives one a serious technical advantage. Here, since $P(\mathbf{Q})\backslash \text{G}(\mathbf{Q})$ parametrizes the rank-one lines in $W(\mathbf{Q})$ , the use of $W$ allows one to see the action of $\operatorname{GSp}_{6}(\mathbf{Q})$ on the coset space $P(\mathbf{Q})\backslash \text{G}(\mathbf{Q})$ as a linear action. This linearization makes the double coset computation simple.

Before giving the proof, define $J_{6}=e_{1}\bigwedge f_{1}+e_{2}\bigwedge f_{2}+e_{3}\bigwedge f_{3}$ . Note that if $a_{1},a_{2},a_{3}$ in $B$ are purely imaginary, and $c_{1},c_{2},c_{3}$ are in $\mathbf{Q}$ , then

(5.5) $$\begin{eqnarray}(a_{1}f_{1}+a_{2}f_{2}+a_{3}f_{3})\wedge J_{6}+c_{1}e_{1}\wedge f_{2}\wedge f_{3}+c_{2}f_{1}\wedge e_{2}\wedge f_{3}+c_{3}f_{1}\wedge f_{2}\wedge e_{3}\end{eqnarray}$$

is the element

$$\begin{eqnarray}\left(0,0,\left(\begin{array}{@{}ccc@{}}c_{1} & a_{3} & -a_{2}\\ -a_{3} & c_{2} & a_{1}\\ a_{2} & -a_{1} & c_{3}\end{array}\right),0\right)\end{eqnarray}$$

of $W$ .

Proof of Proposition 5.5.

Suppose we are given a rank-one element $v=(a,b,c,d)$ of $W(\mathbf{Q})$ . We will first use the action of $\operatorname{GSp}_{6}(\mathbf{Q})$ to put it into a standard form. By Lemma 5.4, we may assume $d=1$ . Then our element is of the form $(N(c),c^{\#},c,1)$ . Now use the transformations $\overline{n}(Y)$ to make $c$ purely imaginary.

Denote by $M_{P,6}$ the Levi subgroup of the Siegel parabolic on $\operatorname{GSp}_{6}$ , i.e., the elements of $\operatorname{GSp}_{6}$ of the form $(\!\begin{smallmatrix}m & \\ & n\end{smallmatrix}\!)$ . By formula (5.5), $M_{P,6}$ acts on the space of such $c$ values as $B^{\operatorname{tr}=0}\otimes _{\mathbf{Q}}\,\text{span}\{f_{1},f_{2},f_{3}\}$ . That is, $M_{P,6}$ acts on these $c$ values via its usual action on $\text{span}\{f_{1},f_{2},f_{3}\}\subseteq W_{6}$ , linearly extended to $B^{\operatorname{tr}=0}\otimes _{\mathbf{Q}}\,\text{span}\{f_{1},f_{2},f_{3}\}$ . Pick a basis $i,j,k$ of $B^{\operatorname{tr}=0}$ , and suppose $c=i\otimes v_{1}+j\otimes v_{2}+k\otimes v_{3}$ . We have four different cases, depending on whether $\text{dim}\,\text{span}\{v_{1},v_{2},v_{3}\}$ is 0, 1, 2 or 3. (Note that this dimension is not a $\operatorname{GSp}_{6}$ invariant of the orbit, just an $M_{P,6}$ invariant.) If the dimension is 0, i.e, $c=0$ , then we are in the orbit of $f$ , which is negligible. There are an infinite number of orbits with $\text{dim}\,\text{span}\{v_{1},v_{2},v_{3}\}=1$ . Given such an orbit, we can use $M_{P,6}$ to move $c$ to $h\otimes f_{1}$ with $h\in B^{\operatorname{tr}=0}$ . Then this orbit is stabilized by $U_{Q}(f_{1})$ , the unipotent radical of the parabolic subgroup of $\operatorname{GSp}_{6}$ stabilizing the line $\mathbf{Q}f_{1}\subseteq W_{6}$ . Thus, this orbit is also negligible. We will show that there is a unique orbit with $\dim =3$ , and that this orbit is represented by $f_{{\mathcal{O}}}$ . Furthermore, we will see that the orbit corresponding to $\dim =2$ is the same as the $\dim =3$ orbit.

First, for $\dim =3$ , use $M_{P,6}$ to move $c$ to $i\otimes f_{1}+j\otimes f_{2}+k\otimes f_{3}$ , proving that there is one such orbit.

Now we consider a $\dim =2$ orbit. Here we have $c=i\otimes v_{1}+j\otimes v_{2}+k\otimes v_{3}$ with one of the $v_{i}$ in the span of the other two. Thus we may use $M_{P,6}$ to move this orbit to one with $c=h_{1}\otimes f_{1}+h_{2}\otimes f_{2}$ , with $h_{1},h_{2}$ in $B^{\operatorname{tr}=0}$ . If $\dim \text{span}\{h_{1},h_{2}\}=1$ , then we are really in the $\dim =1$ case. Thus we have that $h_{1},h_{2}$ are linearly independent.

Now

$$\begin{eqnarray}c=\left(\begin{array}{@{}ccc@{}}0 & 0 & -h_{2}\\ 0 & 0 & h_{1}\\ h_{2} & -h_{1} & 0\end{array}\right),\end{eqnarray}$$

from which we obtain

$$\begin{eqnarray}c^{\sharp }=\left(\begin{array}{@{}ccc@{}}-n(h_{1}) & h_{2}h_{1} & 0\\ h_{1}h_{2} & -n(h_{2}) & 0\\ 0 & 0 & 0\end{array}\right)\end{eqnarray}$$

and $N(c)=0$ . Hence we have that $v=(N(c),c^{\#},c,1)$ is

$$\begin{eqnarray}-n(h_{1})e_{1}^{\ast }\wedge f_{1}-n(h_{2})e_{2}^{\ast }\wedge f_{2}+h_{1}h_{2}e_{2}^{\ast }\wedge f_{1}+h_{2}h_{1}e_{1}^{\ast }\wedge f_{2}+(h_{1}f_{1}+h_{2}f_{2})\wedge J_{6}+f_{1}\wedge f_{2}\wedge f_{3}.\end{eqnarray}$$

Now apply to $v$ the element of $\operatorname{GSp}_{6}$ that takes $e_{3}\mapsto f_{3}$ , $f_{3}\mapsto -e_{3}$ and is the identity on $e_{1},e_{2},f_{1},f_{2}$ . One obtains the element $v^{\prime }=(0,0,h,0)$ , with

$$\begin{eqnarray}h=\left(\begin{array}{@{}ccc@{}}-n(h_{2}) & -h_{2}h_{1} & -h_{2}\\ -h_{1}h_{2} & -n(h_{1}) & h_{1}\\ h_{2} & -h_{1} & -1\end{array}\right).\end{eqnarray}$$

Since $B$ is not split, $h_{1}h_{2}$ is not in the span of $1,h_{1},h_{2}$ . Take $h_{3}$ in $B^{\operatorname{tr}=0}$ with $h_{1}h_{2}-h_{3}$ in $\mathbf{Q}$ . Then $\operatorname{Im}(h)=(h_{1}f_{1}+h_{2}f_{2}+h_{3}f_{3})\bigwedge J_{6}$ . Since

$$\begin{eqnarray}h_{1}\otimes f_{1}+h_{2}\otimes f_{2}+h_{3}\otimes f_{3}=i\otimes v_{1}+j\otimes v_{2}+k\otimes v_{3}\end{eqnarray}$$

for some $v_{1},v_{2},v_{3}$ spanning $\langle f_{1},f_{2},f_{3}\rangle$ , the $\dim =2$ orbit is the same as the $\dim =3$ orbit.

The element $f_{{\mathcal{O}}}$ clearly represents the $\dim =3$ orbit. To compute the stabilizer, note that by (5.5) we immediately see that if $f_{{\mathcal{O}}}g=f_{{\mathcal{O}}}$ , then $f_{1}g=f_{1},f_{2}g=f_{2},f_{3}g=f_{3}$ . Hence an element of the stabilizer must be of the form $g=(\!\begin{smallmatrix}\unicode[STIX]{x1D706} & \\ & 1\end{smallmatrix}\!)(\!\begin{smallmatrix}1 & U\\ & 1\end{smallmatrix}\!)$ . Applying this element to $f_{{\mathcal{O}}}$ , we find it stabilizes $f_{{\mathcal{O}}}$ if and only if $\operatorname{tr}(TU)=0$ , completing the proof.◻

Recall the partially normalized Eisenstein series $E^{\unicode[STIX]{x1D6F7}}(g,s)$ from above, and set

$$\begin{eqnarray}I^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D719},s)=\int _{\operatorname{GSp}_{6}(\mathbf{Q})Z(\mathbf{A})\backslash \!\operatorname{GSp}_{6}(\mathbf{A})}\unicode[STIX]{x1D719}(g)E^{\unicode[STIX]{x1D6F7}}(g,s)\,dg.\end{eqnarray}$$

We will consider the integral $I^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D719},s)$ for now, and add back in the other normalizing factors later.

Write $U_{P}$ for the unipotent radical of the Siegel parabolic of $\operatorname{GSp}_{6}$ . The group $U_{P}$ consists of the matrices $(\!\begin{smallmatrix}1 & u\\ & 1\end{smallmatrix}\!)$ with $u=^{t}u$ . Define $U_{0}\subseteq U_{P}$ to be the elements with $\operatorname{tr}(Tu)=0$ . Write $\unicode[STIX]{x1D713}:\mathbf{Q}\backslash \mathbf{A}\rightarrow \mathbf{C}^{\times }$ for the usual additive character of conductor 1 satisfying $\unicode[STIX]{x1D713}_{\infty }(x)=e^{2\unicode[STIX]{x1D70B}ix}$ for $x\in \mathbf{R}$ . Define $\unicode[STIX]{x1D712}:U_{P}(\mathbf{Q})\backslash U_{P}(\mathbf{A})\rightarrow \mathbf{C}^{\times }$ via $\unicode[STIX]{x1D712}((\!\begin{smallmatrix}1 & h\\ & 1\end{smallmatrix}\!))=\unicode[STIX]{x1D713}(\operatorname{tr}(Th))$ , and set

(5.6) $$\begin{eqnarray}\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(g)=\int _{U_{P}(\mathbf{Q})\backslash U_{P}(\mathbf{A})}\unicode[STIX]{x1D712}^{-1}(u)\unicode[STIX]{x1D719}(ug)\,du.\end{eqnarray}$$

Theorem 5.7. The global integrals $I^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D719},s),I_{2r}^{\ast }(\unicode[STIX]{x1D719},s)$ converge absolutely for all $s$ for which the Eisenstein series $E^{\unicode[STIX]{x1D6F7}}(g,s)$ , respectively, $E_{2r}^{\ast }(g,s)$ is finite, and thus define a meromorphic function of  $s$ . For $\text{Re}(s)$ sufficiently large, the integral $I^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D719},s)$ unfolds as

$$\begin{eqnarray}\displaystyle I^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D719},s) & = & \displaystyle \int _{U_{0}(\mathbf{A})\backslash \!\operatorname{GSp}_{6}(\mathbf{A})}\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(g)|\unicode[STIX]{x1D708}(g)|^{s}\unicode[STIX]{x1D6F7}(f_{{\mathcal{O}}}g)\,dg\nonumber\\ \displaystyle & = & \displaystyle \int _{U_{P}(\mathbf{A})\backslash \!\operatorname{GSp}_{6}(\mathbf{A})}\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(g)|\unicode[STIX]{x1D708}(g)|^{s}\biggl(\int _{U_{0}(\mathbf{A})\backslash U_{P}(\mathbf{A})}\unicode[STIX]{x1D712}(u)\unicode[STIX]{x1D6F7}(f_{{\mathcal{O}}}ug)\,du\biggr)\,dg.\nonumber\end{eqnarray}$$

Proof. The Eisenstein series $E^{\unicode[STIX]{x1D6F7}}(g,s)$ is a function of moderate growth wherever it is finite. Since the cusp form $\unicode[STIX]{x1D719}$ decreases rapidly on Siegel sets, the integral $I^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D719},s)$ converges absolutely for all $s$ for which the Eisenstein series is finite.

Suppose now $\text{Re}(s)\gg 0$ , and denote by ${\mathcal{F}}$ a fundamental domain for $\operatorname{GSp}_{6}(\mathbf{Q})Z(\mathbf{A})\backslash \operatorname{GSp}_{6}(\mathbf{A})$ . Similar to the above, but easier, $\sum _{\unicode[STIX]{x1D6FE}\in P(\mathbf{Q})\backslash \text{G}(\mathbf{Q})}|f^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D6FE}g,s)|$ converges to a function of moderate growth since $\text{Re}(s)\gg 0$ . Then again since $\unicode[STIX]{x1D719}$ decreases rapidly on Siegel sets, the sum

$$\begin{eqnarray}\mathop{\sum }_{\unicode[STIX]{x1D6FE}\in P(\mathbf{Q})\backslash \text{G}(\mathbf{Q})}\int _{{\mathcal{F}}}|f^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D6FE}g,s)|\,|\unicode[STIX]{x1D719}(g)|\,dg\end{eqnarray}$$

is finite. Hence even though there are an infinite number of $\operatorname{GSp}_{6}(\mathbf{Q})$ orbits on $P(\mathbf{Q})\backslash \text{G}(\mathbf{Q})$ , we may analyze them, and check the vanishing of the associated integrals, one by one.

By Proposition 5.5, we have

(5.7) $$\begin{eqnarray}I^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D719},s)=\int _{\operatorname{GL}_{1}(\mathbf{Q})U_{0}(\mathbf{Q})Z(\mathbf{A})\backslash \!\operatorname{GSp}_{6}(\mathbf{A})}\unicode[STIX]{x1D719}(g)f^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D6FE}_{0}g,s)\,dg\end{eqnarray}$$

where the $\operatorname{GL}_{1}(\mathbf{Q})$ is the set of elements of $\operatorname{GSp}_{6}(\mathbf{Q})$ of the form $(\!\begin{smallmatrix}\unicode[STIX]{x1D706} & \\ & 1\end{smallmatrix}\!)$ , with $\unicode[STIX]{x1D706}\in \mathbf{Q}^{\times }$ , and $\unicode[STIX]{x1D6FE}_{0}\in \text{G}(\mathbf{Q})$ satisfies $(0,0,0,1)\unicode[STIX]{x1D6FE}_{0}=f_{{\mathcal{O}}}$ . Indeed, this is the term from the open orbit, and the negligible orbits contribute zero by the cuspidality of  $\unicode[STIX]{x1D719}$ .

Now we integrate $\unicode[STIX]{x1D719}$ over $U_{0}(\mathbf{Q})\backslash U_{0}(\mathbf{A})$ and Fourier expand:

$$\begin{eqnarray}\int _{U_{0}(\mathbf{Q})\backslash U_{0}(\mathbf{A})}\unicode[STIX]{x1D719}(ug)\,du=\mathop{\sum }_{\unicode[STIX]{x1D6FE}\in \operatorname{GL}_{1}(\mathbf{Q})}\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(\unicode[STIX]{x1D6FE}g).\end{eqnarray}$$

Placing this Fourier expansion into (5.7) we get

(5.8) $$\begin{eqnarray}I^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D719},s)=\int _{U_{0}(\mathbf{A})Z(\mathbf{A})\backslash \!\operatorname{GSp}_{6}(\mathbf{A})}\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(g)f^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D6FE}_{0}g,s)\,dg.\end{eqnarray}$$

Integrating

$$\begin{eqnarray}\int _{U_{0}(\mathbf{A})\backslash \!\operatorname{GSp}_{6}(\mathbf{A})}|\unicode[STIX]{x1D708}(g)|^{s}\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(g)\unicode[STIX]{x1D6F7}(f_{{\mathcal{O}}}g)\,dg\end{eqnarray}$$

over $Z(\mathbf{A})$ one obtains (5.8), giving the theorem.◻

Remark 5.8. In Theorem 5.7, we could have chosen to unfold the global integral $I^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D719},s)$ to any Fourier coefficient $\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}^{\prime }}(g)$ , where $\unicode[STIX]{x1D712}^{\prime }((\!\begin{smallmatrix}1 & h\\ & 1\end{smallmatrix}\!))=\unicode[STIX]{x1D713}(\operatorname{tr}(T^{\prime }h))$ and the quaternion algebra associated to $T^{\prime }$ is  $B$ . Using $T^{\prime }$ instead of $T$ would have no effect on the results or proofs in § 5.2. However, using $T$ that corresponds to the maximal order $B_{0}$ did make a difference in Corollary 4.2, and will also make a difference in Proposition 6.1 below.

6 Evaluation of the integral

In this section we relate the unfolded Rankin–Selberg integral to the Spin $L$ -function.

6.1 The finite places

Our first task is to compute the unipotent integral $\int _{U_{0}\backslash U_{P}}\unicode[STIX]{x1D712}(u)\unicode[STIX]{x1D6F7}(f_{{\mathcal{O}}}ug)\,du$ . For an element $m\in \operatorname{GL}_{3}$ , and an element $\unicode[STIX]{x1D706}\in \operatorname{GL}_{1}$ , we set

an element of $\operatorname{GSp}_{6}$ . For $(\unicode[STIX]{x1D706},m)\in \operatorname{GL}_{1}(\mathbf{A}_{f})\times \operatorname{GL}_{3}(\mathbf{A}_{f})$ , we define

$$\begin{eqnarray}\unicode[STIX]{x1D6EF}(\unicode[STIX]{x1D706},m):=\left\{\begin{array}{@{}ll@{}}1\quad & \text{if}~\unicode[STIX]{x1D706}\in \widehat{\mathbf{Z}},m\in M_{3}(\widehat{\mathbf{Z}}),\text{and}~m^{-1}Tc(m)\in M_{3}(\widehat{\mathbf{Z}}),\\ 0\quad & \text{otherwise}.\end{array}\right.\end{eqnarray}$$

Proposition 6.1. The integral

(6.1) $$\begin{eqnarray}\int _{U_{0}(\mathbf{A}_{f})\backslash U_{P}(\mathbf{A}_{f})}\unicode[STIX]{x1D712}(u)\unicode[STIX]{x1D6F7}_{f}(f_{{\mathcal{O}}}u(\unicode[STIX]{x1D706},m))\,du=|\unicode[STIX]{x1D706}|_{f}\unicode[STIX]{x1D6EF}(\unicode[STIX]{x1D706},m).\end{eqnarray}$$

Proof. Recall the lattice $J_{0}$ in $H_{3}(B)$ defined in (5.4), and set $J_{0}(\widehat{\mathbf{Z}})=J_{0}\otimes _{\mathbf{Z}}\widehat{\mathbf{Z}}$ . Suppose $h=(\unicode[STIX]{x1D706},m)$ . One computes

(6.2) $$\begin{eqnarray}f_{{\mathcal{O}}}n(u)h=\biggl(0,0,m^{-1}A(T)c(m),\frac{\operatorname{tr}(Tu)}{\unicode[STIX]{x1D706}}\biggr).\end{eqnarray}$$

Hence for the integral (6.1) to be nonzero, one needs $m^{-1}A(T)c(m)$ to be in $J_{0}(\widehat{\mathbf{Z}})$ and the integral

(6.3) $$\begin{eqnarray}\int _{U_{0}(\mathbf{A}_{f})\backslash U_{P}(\mathbf{A}_{f})}\unicode[STIX]{x1D713}(\operatorname{tr}(Tu))\operatorname{char}\biggl(\frac{\operatorname{tr}(Tu)}{\unicode[STIX]{x1D706}}\in \widehat{\mathbf{Z}}\biggr)\,du\end{eqnarray}$$

to be nonzero. By Lemma 3.8, $m^{-1}A(T)c(m)$ is in $J_{0}(\widehat{\mathbf{Z}})$ if and only if $m\in M_{3}(\widehat{\mathbf{Z}})$ and $m^{-1}Tc(m)\in M_{3}(\widehat{\mathbf{Z}})$ . The integral (6.3) is zero when $\unicode[STIX]{x1D706}\notin \widehat{\mathbf{Z}}$ , and is $|\unicode[STIX]{x1D706}|_{f}$ when $\unicode[STIX]{x1D706}\in \widehat{\mathbf{Z}}$ . The proposition follows.◻

The global integral we must compute is

$$\begin{eqnarray}\displaystyle I^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D719},s) & = & \displaystyle \int _{U_{0}(\mathbf{R})\backslash \!\operatorname{GSp}_{6}(\mathbf{R})}|\unicode[STIX]{x1D708}(g_{\infty })|_{\infty }^{s}\unicode[STIX]{x1D6F7}_{\infty }(f_{{\mathcal{O}}}g_{\infty })\nonumber\\ \displaystyle & & \displaystyle \times \,\biggl(\int _{U_{0}(\mathbf{A}_{f})\backslash \!\operatorname{GSp}_{6}(\mathbf{A}_{f})}\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(g_{f}g_{\infty })|\unicode[STIX]{x1D708}(g_{f})|_{f}^{s}\unicode[STIX]{x1D6F7}_{f}(f_{{\mathcal{O}}}g_{f})\,dg_{f}\biggr)\,dg_{\infty }\nonumber\\ \displaystyle & = & \displaystyle \int _{U_{0}(\mathbf{R})\backslash \!\operatorname{GSp}_{6}(\mathbf{R})}|\unicode[STIX]{x1D708}(g_{\infty })|_{\infty }^{s}\unicode[STIX]{x1D6F7}_{\infty }(f_{{\mathcal{O}}}g_{\infty })I_{f}(\unicode[STIX]{x1D719},s,g_{\infty })\,dg_{\infty }\nonumber\end{eqnarray}$$

where

(6.4) $$\begin{eqnarray}\displaystyle I_{f}(\unicode[STIX]{x1D719},s,g_{\infty }) & = & \displaystyle \int _{U_{0}(\mathbf{A}_{f})\backslash \!\operatorname{GSp}_{6}(\mathbf{A}_{f})}\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(g_{f}g_{\infty })|\unicode[STIX]{x1D708}(g_{f})|_{f}^{s}\unicode[STIX]{x1D6F7}_{f}(f_{{\mathcal{O}}}g_{f})\,dg_{f}\nonumber\\ \displaystyle & = & \displaystyle \int _{(\operatorname{GL}_{1}\times \operatorname{GL}_{3})(\mathbf{A}_{f})}\unicode[STIX]{x1D6FF}_{P}^{-1}(h)|\text{det}(m)|_{f}^{-1}\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(\unicode[STIX]{x1D70B}(h)g_{\infty })|\unicode[STIX]{x1D708}(h)|_{f}^{s+1}\unicode[STIX]{x1D6EF}(\unicode[STIX]{x1D706},m)\,dh.\end{eqnarray}$$

Here $h=(\unicode[STIX]{x1D706},m)$ , $\unicode[STIX]{x1D6FF}_{P}$ denotes the modulus character of the Siegel parabolic in $\operatorname{GSp}_{6}$ , and we have applied Proposition 6.1, the Iwasawa decomposition, and the fact that $\unicode[STIX]{x1D706}=\unicode[STIX]{x1D708}(h)\det (m)^{-1}$ .

Translating between the classical and adelic language, the Dirichlet series of Corollary 4.2 exactly evaluates the integral (6.4), as we check in the following proposition.

Proposition 6.2. Assume $\unicode[STIX]{x1D719}$ is as in Definition 1.1. Then

$$\begin{eqnarray}I_{f}(\unicode[STIX]{x1D719},s,g_{\infty })=\frac{L(\unicode[STIX]{x1D70B},\text{Spin},s-2)}{\unicode[STIX]{x1D701}(2s-4)\unicode[STIX]{x1D701}_{D_{B}}(2s-2)}\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(g_{\infty }).\end{eqnarray}$$

Proof. If $\unicode[STIX]{x1D708}(g_{\infty })>0$ , one has

(6.5) $$\begin{eqnarray}\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(g_{\infty })=a_{f}(T)\unicode[STIX]{x1D708}(g_{\infty })^{-r}j(g_{\infty },i)^{-2r}e^{2\unicode[STIX]{x1D70B}i\operatorname{tr}(Tg_{\infty }\cdot i)},\end{eqnarray}$$

while $\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(g_{\infty })=0$ if $\unicode[STIX]{x1D708}(g_{\infty })<0$ . Suppose $h=(\unicode[STIX]{x1D706},m)\in M_{P,6}(\mathbf{A}_{f})$ . Then $h=h_{\mathbf{Q}}h_{\infty }^{-1}k$ for some $h\in M_{P,6}(\mathbf{Q})$ and $k\in M_{P,6}(\mathbf{Z}_{p})$ , where $h_{\infty }=(h_{\mathbf{Q}})_{\infty }$ is the component of $h_{\mathbf{Q}}$ at the archimedean place. Hence $\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(hg_{\infty })=\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(h_{\mathbf{Q}}h_{\infty }^{-1}g_{\infty })=\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}\cdot h_{\mathbf{Q}}}(h_{\infty }^{-1}g_{\infty })$ , where $\unicode[STIX]{x1D712}\cdot h_{\mathbf{Q}}$ is the Fourier coefficient (5.6) except with $T$ replaced by $\unicode[STIX]{x1D706}m^{-1}Tc(m)$ . Thus,

$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(hg_{\infty }) & = & \displaystyle a_{f}(\unicode[STIX]{x1D706}m^{-1}Tc(m))\unicode[STIX]{x1D708}(h_{\infty }^{-1}g_{\infty })^{-r}j(h_{\infty }^{-1}g_{\infty },i)^{-2r}e^{2\unicode[STIX]{x1D70B}i\operatorname{tr}((\unicode[STIX]{x1D706}m^{-1}Tc(m))(h_{\infty }^{-1}g_{\infty })\cdot i)}\nonumber\\ \displaystyle & = & \displaystyle a_{f}(\unicode[STIX]{x1D706}m^{-1}Tc(m))\unicode[STIX]{x1D706}^{-3r}\det (m)^{-r}(\unicode[STIX]{x1D708}(g_{\infty })^{-r}j(g_{\infty },i)^{-2r}e^{2\unicode[STIX]{x1D70B}i\operatorname{tr}(Tg_{\infty }\cdot i)}).\nonumber\end{eqnarray}$$

Hence if $h\in (h_{\mathbf{Q}})_{f}M_{P,6}(\mathbf{Z}_{p})$ , with $h_{\mathbf{Q}}=(\unicode[STIX]{x1D706},m)$ , then the integrand in (6.4) is equal to

$$\begin{eqnarray}\displaystyle & & \displaystyle (\unicode[STIX]{x1D706}^{6}\det (m)^{2})\det (m)(a_{f}(\unicode[STIX]{x1D706}m^{-1}Tc(m))\unicode[STIX]{x1D706}^{-3r}\det (m)^{-r})(\unicode[STIX]{x1D706}\det (m))^{-s-1}\unicode[STIX]{x1D6EF}(\unicode[STIX]{x1D706},m)\nonumber\\ \displaystyle & & \displaystyle \qquad \times (\unicode[STIX]{x1D708}(g_{\infty })^{-r}j(g_{\infty },i)^{-2r}e^{2\unicode[STIX]{x1D70B}i\operatorname{tr}(Tg_{\infty }\cdot i)})\nonumber\\ \displaystyle & & \displaystyle \quad =\unicode[STIX]{x1D706}^{-(s+3r-5)}\det (m)^{-(s+r-2)}\unicode[STIX]{x1D6EF}(\unicode[STIX]{x1D706},m)a_{f}(\unicode[STIX]{x1D706}m^{-1}Tc(m))\times (\unicode[STIX]{x1D708}(g_{\infty })^{-r}j(g_{\infty },i)^{-2r}e^{2\unicode[STIX]{x1D70B}i\operatorname{tr}(Tg_{\infty }\cdot i)}).\nonumber\end{eqnarray}$$

Applying Corollary 4.2 and (6.5) again gives the proposition. ◻

6.2 The infinite place

Suppose $\unicode[STIX]{x1D719}$ is a cusp form corresponding to a level-one, holomorphic Siegel modular form of weight $2r$ , as in Definition 1.1. We now compute the normalized archimedean integral

$$\begin{eqnarray}I_{\infty ,2r}^{\ast }(\unicode[STIX]{x1D719},s)=\unicode[STIX]{x1D70B}^{-(s+r)}D_{B}^{s}\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s+2r-4)\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s+2r-8)I_{\infty ,2r}^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D719},s),\end{eqnarray}$$

where

(6.6) $$\begin{eqnarray}I_{\infty ,2r}^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D719},s)=\int _{U_{0}(\mathbf{R})\backslash \!\operatorname{GSp}_{6}(\mathbf{R})}\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(g)|\unicode[STIX]{x1D708}(g)|^{s}\unicode[STIX]{x1D6F7}_{\infty ,2r}(f_{{\mathcal{O}}}g)\,dg.\end{eqnarray}$$

We prove the following theorem.

Theorem 6.3. Suppose $\unicode[STIX]{x1D719}$ is as in Definition 1.1, and that $\unicode[STIX]{x1D719}$ corresponds to the level-one Siegel modular form  $f$ . Then $I_{\infty ,2r}^{\ast }(\unicode[STIX]{x1D719},s)$ equals

$$\begin{eqnarray}a_{f}(T)\unicode[STIX]{x1D6E4}_{\mathbf{C}}(s+r-4)\unicode[STIX]{x1D6E4}_{\mathbf{C}}(s+r-3)\unicode[STIX]{x1D6E4}_{\mathbf{C}}(s+r-2)\unicode[STIX]{x1D6E4}_{\mathbf{C}}(s+3r-5)\end{eqnarray}$$

up to a nonzero constant.

Our first task is to define the Schwartz function $\unicode[STIX]{x1D6F7}_{\infty ,2r}$ on $W(\mathbf{R})$ . To do this, we digress slightly and discuss the Hermitian symmetric space ${\mathcal{H}}$ associated to $\text{G}(\mathbf{R})$ .

6.2.1 The Hermitian symmetric space for $\text{G}(\mathbf{R})$

We assume $B_{\mathbf{R}}=B\otimes \mathbf{R}$ is the Hamilton quaternions. Set $B_{\mathbf{C}}=B_{\mathbf{R}}\otimes _{\mathbf{R}}\mathbf{C}$ , with the involution on $B_{\mathbf{C}}$ the one from $B_{\mathbf{R}}$ , extended $\mathbf{C}$ -linearly to $B_{\mathbf{C}}$ . Associated to $B_{\mathbf{C}}$ , we have the space $H_{3}(B_{\mathbf{C}})$ of $3\times 3$ Hermitian matrices over $B_{\mathbf{C}}$ . Elements of $H_{3}(B_{\mathbf{C}})$ are formal expressions $Z=X+iY$ , with $X,Y$ in $H_{3}(B_{\mathbf{R}})$ . Note that the $i$ in this expression is not the $i$ in $B_{\mathbf{R}}$ ; this $i$ is in the center of $B_{\mathbf{C}}$ . The Hermitian symmetric space ${\mathcal{H}}$ is the set of $Z=X+iY$ in $H_{3}(B_{\mathbf{C}})$ with $Y$ positive definite.

Denote by $\text{G}^{+}(\mathbf{R})$ the subgroup of $\text{G}(\mathbf{R})$ that is the connected component of the identity. We explain the action of $\text{G}^{+}(\mathbf{R})$ on ${\mathcal{H}}$ . First, note that we have an inclusion $\text{G}(\mathbf{R})\rightarrow \text{G}(\mathbf{C})$ , and $\text{G}(\mathbf{C})$ acts on $W(\mathbf{C})=W(\mathbf{R})\otimes _{\mathbf{R}}\mathbf{C}$ . Set $e=(1,0,0,0)$ , a rank-one element of  $W$ . Identify $Z\in {\mathcal{H}}$ with the rank-one element

$$\begin{eqnarray}r(Z)=en(-Z)=(1,-Z,Z^{\sharp },-N(Z)).\end{eqnarray}$$

Consider the projective space $\mathbf{P}(W(\mathbf{C})^{rk=1})$ of rank-one $\mathbf{C}$ -lines in $W(\mathbf{C})$ . Then $\text{G}(\mathbf{R})$ acts on this space on the left via $g\cdot v=vg^{-1}$ . We will see momentarily that $\text{G}^{+}(\mathbf{R})$ preserves the image of ${\mathcal{H}}$ inside $\mathbf{P}(W(\mathbf{C})^{rk=1})$ , and acts on it transitively. In fact, the factor of automorphy

$$\begin{eqnarray}j_{\text{G}(\mathbf{R})}(g,Z):\text{G}^{+}(\mathbf{R})\times {\mathcal{H}}\rightarrow \mathbf{C}^{\times }\end{eqnarray}$$

and the action of $\text{G}^{+}(\mathbf{R})$ on ${\mathcal{H}}$ are simultaneously defined by the equality

$$\begin{eqnarray}r(Z)g^{-1}=j_{\text{G}(\mathbf{R})}(g,Z)r(gZ).\end{eqnarray}$$

We summarize what we need in a proposition. Recall the rank-one element $f=(0,0,0,1)$ .

Proposition 6.4. Suppose $Z\in {\mathcal{H}}$ , and $g\in \text{G}^{+}(\mathbf{R})$ . Then the complex number $j_{\text{G}(\mathbf{R})}(g,Z)=\langle r(Z)g^{-1},f\rangle$ is nonzero, and thus

$$\begin{eqnarray}r(Z)g^{-1}=j_{\text{G}(\mathbf{R})}(g,Z)r(gZ)\end{eqnarray}$$

for some $gZ$ in $H_{3}(B_{\mathbf{C}})$ . The element $gZ$ is in ${\mathcal{H}}$ , and thus this equality defines an action of $\text{G}^{+}(\mathbf{R})$ on ${\mathcal{H}}$ . Under the natural embedding ${\mathcal{H}}_{3}\rightarrow {\mathcal{H}}$ , the two actions of $\operatorname{GSp}_{6}^{+}(\mathbf{R})$ are the same. Furthermore, $j_{\operatorname{GSp}_{6}}(g,Z)=j_{\text{G}(\mathbf{R})}(g,Z)$ when $g\in \operatorname{GSp}_{6}^{+}(\mathbf{R})$ and $Z\in {\mathcal{H}}_{3}$ .

Proof. We sketch the proof. First, a minimal parabolic subgroup $P_{0}$ of $\text{G}$ is discussed in § 7.1. One can check that the subgroup $P_{0}^{+}(\mathbf{R})$ of $\text{G}^{+}(\mathbf{R})$ has $j_{\text{G}(\mathbf{R})}(P_{0}^{+}(\mathbf{R}),{\mathcal{H}})\neq 0$ and acts transitively on ${\mathcal{H}}$ . Now suppose $g\in \text{G}^{+}(\mathbf{R})$ , $Z\in {\mathcal{H}}$ , and $p\in P_{0}^{+}(\mathbf{R})$ satisfies $p\cdot i=Z$ . Then

$$\begin{eqnarray}\langle r(Z)g^{-1},f\rangle =j_{\text{G}(\mathbf{R})}(p,i)^{-1}\langle r(i)p^{-1}g^{-1},f\rangle =\unicode[STIX]{x1D708}(gp)^{-1}j_{\text{G}(\mathbf{R})}(p,i)^{-1}\langle r(i),fgp\rangle .\end{eqnarray}$$

This last quantity is then nonzero by Lemma 6.5 below. Hence $r(Z)g^{-1}=j_{\text{G}(\mathbf{R})}(g,Z)r(gZ)$ for some $gZ$ in $H_{3}(B_{\mathbf{C}})$ . Next, one verifies the equality $N(\operatorname{Im}(W))=(1/8i)\langle \overline{r(W)},r(W)\rangle$ for any $W\in H_{3}(B_{\mathbf{C}})$ . One deduces from this equality that $N(\operatorname{Im}(gZ))>0$ . Then that $gZ\in {\mathcal{H}}$ follows from a continuity argument.

To check that the actions and factors of automorphy agree for $g\in \operatorname{GSp}_{6}^{+}(\mathbf{R})$ and $Z\in {\mathcal{H}}_{3}$ , it suffices to check this on generators $n(X),J_{6}$ and $M_{P,6}^{+}(\mathbf{R})$ , which may be done easily.◻

Before defining $\unicode[STIX]{x1D6F7}_{\infty ,2r}$ , we need an additional lemma. Recall the element $J$ of $\text{G}(\mathbf{Q})$ that sends $(a,b,c,d)\mapsto (-d,c,-b,a)$ .

Lemma 6.5. If $v=(a,b,c,d)$ , define $\Vert v\Vert ^{2}=\langle v,vJ\rangle$ . Then $\Vert v\Vert ^{2}=a^{2}+\operatorname{tr}(b,b)+\operatorname{tr}(c,c)+d^{2}$ , and hence if $v\in W(\mathbf{R})$ , $\Vert v\Vert ^{2}\geqslant 0$ and is only $0$ when $v=0$ . If furthermore $v$ is of rank one, then $|\langle r(i),v\rangle |^{2}=\Vert v\Vert ^{2}$ .

Proof. The formula for $\Vert v\Vert ^{2}$ is trivial. Suppose $v$ is rank one. We have $\langle r(i),v\rangle =(d-\operatorname{tr}(b))+i(\operatorname{tr}(c)-a)$ . Hence

$$\begin{eqnarray}\displaystyle |\langle r(i),v\rangle |^{2} & = & \displaystyle (d-\operatorname{tr}(b))^{2}+(a-\operatorname{tr}(c))^{2}\nonumber\\ \displaystyle & = & \displaystyle d^{2}-2\,d\operatorname{tr}(b)+\operatorname{tr}(b)^{2}+a^{2}-2a\operatorname{tr}(c)+\operatorname{tr}(c)^{2}\nonumber\\ \displaystyle & = & \displaystyle a^{2}+(\operatorname{tr}(b)^{2}-2\operatorname{tr}(b^{\sharp }))+(\operatorname{tr}(c)^{2}-2\operatorname{tr}(c^{\sharp }))+d^{2},\nonumber\end{eqnarray}$$

where in the last equality, since $v$ is rank one, $db=c^{\sharp }$ and $ac=b^{\sharp }$ . For general $W$ one has the equality $\operatorname{tr}(W)^{2}-2\operatorname{tr}(W^{\sharp })=\operatorname{tr}(W,W)$ . The lemma follows.◻

6.2.2 Archimedean calculation

Define

(6.7) $$\begin{eqnarray}\unicode[STIX]{x1D6F7}_{\infty ,2r}(v)=e^{-\Vert v\Vert ^{2}}\langle r(i),v\rangle ^{2r}.\end{eqnarray}$$

This is obviously a Schwartz function on $W(\mathbf{R})$ . Note that we have $\langle r(i),vgk\rangle =j(k,i)\langle r(i),vg\rangle$ for $k$ in $K_{\infty ,\operatorname{Sp}_{6}}$ . Our assumption on $\unicode[STIX]{x1D719}$ gives

$$\begin{eqnarray}\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(g)=a_{f}(T)\unicode[STIX]{x1D708}(g)^{-r}j(g,i)^{-2r}e^{2\unicode[STIX]{x1D70B}i\operatorname{tr}(Tg\cdot i)}\end{eqnarray}$$

if $g\in \operatorname{GSp}_{6}^{+}(\mathbf{R})$ , and $\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(g)=0$ if $\unicode[STIX]{x1D708}(g)<0$ . (The modular form associated to $\unicode[STIX]{x1D719}$ has negative definite Fourier coefficients when restricted to the lower half-space, ${\mathcal{H}}_{3}^{-}$ .) Furthermore, for $g\in \operatorname{GSp}_{6}^{+}(\mathbf{R})$ , one has the identity

(6.8) $$\begin{eqnarray}\unicode[STIX]{x1D708}(g)^{-1}j(g,i)^{-1}\langle r(i),f_{{\mathcal{O}}}g\rangle =\operatorname{tr}(T(g\cdot i)).\end{eqnarray}$$

Note that the left-hand side of (6.8), as a function of  $g$ , is right $Z(\mathbf{R})K_{\infty ,\operatorname{Sp}_{6}}$ -invariant. Hence to prove the equality, it suffices to take $g$ in the Siegel parabolic of $\operatorname{GSp}_{6}$ , and then (6.8) follows from (6.2).

Now, putting (6.7) into (6.6) and integrating over the center $Z(\mathbf{R})$ , we obtain

$$\begin{eqnarray}\displaystyle I_{\infty ,2r}^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D719},s) & = & \displaystyle \unicode[STIX]{x1D6E4}(s+r)\int _{U^{0}(\mathbf{R})Z(\mathbf{R})\backslash \!\operatorname{GSp}_{6}(\mathbf{R})}\frac{|\unicode[STIX]{x1D708}(g)|^{s}\langle r(i),f_{{\mathcal{O}}}g\rangle ^{2r}\unicode[STIX]{x1D719}_{\unicode[STIX]{x1D712}}(g)}{|\langle r(i),f_{{\mathcal{O}}}g\rangle |^{2s+2r}}\,dg\nonumber\\ \displaystyle & = & \displaystyle \unicode[STIX]{x1D6E4}(s+r)\int _{U^{0}(\mathbf{R})Z(\mathbf{R})\backslash \!\operatorname{GSp}_{6}^{+}(\mathbf{R})}a(T)\frac{|\unicode[STIX]{x1D708}(g)|^{s+r}\operatorname{tr}(TZ)^{2r}e^{2\unicode[STIX]{x1D70B}i\operatorname{tr}(TZ)}}{|\langle r(i),f_{{\mathcal{O}}}g\rangle |^{2s+2r}}\,dg,\nonumber\end{eqnarray}$$

where $Z=g\cdot i=X+iY$ . One has

$$\begin{eqnarray}\frac{|\unicode[STIX]{x1D708}(g)|^{s+r}}{|\langle r(i),f_{{\mathcal{O}}}g\rangle |^{2s+2r}}=\frac{|\unicode[STIX]{x1D708}(g)^{-1}j(g,i)^{-2}|^{s+r}}{|\operatorname{tr}(TZ)|^{2s+2r}}=\frac{N(Y)^{s+r}}{|\operatorname{tr}(TZ)|^{2s+2r}}.\end{eqnarray}$$

Hence, $I_{\infty ,2r}^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D719},s)=a(T)\unicode[STIX]{x1D6E4}(s+r)N(T)^{-(s+r)}J(s)$ , with

(6.9) $$\begin{eqnarray}\displaystyle J(s) & = & \displaystyle \int _{U^{0}(\mathbf{R})Z(\mathbf{R})\backslash \!\operatorname{GSp}_{6}^{+}(\mathbf{R})}\frac{N(TY)^{s+r}\operatorname{tr}(TZ)^{2r}e^{2\unicode[STIX]{x1D70B}i\operatorname{tr}(TZ)}}{|\operatorname{tr}(TZ)|^{2s+2r}}\,dg\nonumber\\ \displaystyle & = & \displaystyle \int _{U_{1}^{0}(\mathbf{R})Z(\mathbf{R})\backslash \!\operatorname{GSp}_{6}^{+}(\mathbf{R})}\frac{N(Y)^{s+r}\operatorname{tr}(Z)^{2r}e^{2\unicode[STIX]{x1D70B}i\operatorname{tr}(Z)}}{|\operatorname{tr}(Z)|^{2s+2r}}\,dg,\end{eqnarray}$$

where $U_{1}^{0}=\{(\!\begin{smallmatrix}1 & X\\ & 1\end{smallmatrix}\!):\operatorname{tr}(X)=0\}$ . The change of variables implicit in equality (6.9) is $g=m_{T}^{-1}g^{\prime }$ , where $m_{T}=(\!\begin{smallmatrix}T^{1/2} & \\ & T^{-1/2}\end{smallmatrix}\!)$ and $T^{1/2}$ a positive-definite symmetric square root of  $T$ .

We now explain how to compute the integral $J(s)$ . As the global integral representation is an analogue of Garrett’s triple product [Gar87], we follow the archimedean calculation of [Gar87]. We compute up to nonzero constants. First, we have that

$$\begin{eqnarray}J(s)=\int _{Z(\mathbf{R})\backslash M_{P,6}^{+}(\mathbf{R})}N(Y)^{s+r-2}e^{-2\unicode[STIX]{x1D70B}\operatorname{tr}(Y)}\int _{\mathbf{R}}\frac{(x+i\operatorname{tr}(Y))^{2r}}{|x+i\operatorname{tr}(Y)|^{2s+2r}}e^{2\unicode[STIX]{x1D70B}ix}\,dx\,d^{\ast }Y.\end{eqnarray}$$

Here $d^{\ast }Y$ is the $M_{P,6}^{+}(\mathbf{R})$ invariant measure on $Y$ , and we have used that $\unicode[STIX]{x1D6FF}_{P}(g)=\det (Y)^{2}$ for $g\in M_{P,6}^{+}(\mathbf{R})$ . Making a variable change, the inner integral becomes

$$\begin{eqnarray}\operatorname{tr}(Y)^{1-2s}\int _{\mathbf{R}}\frac{(x+i)^{2r}}{|x+i|^{2s+2r}}e^{2\unicode[STIX]{x1D70B}i\operatorname{tr}(Y)x}\,dx,\end{eqnarray}$$

using that $\operatorname{tr}(Y)$ is positive. We now introduce a $\unicode[STIX]{x1D6E4}$ -integral, and change the order of integration to obtain

$$\begin{eqnarray}\displaystyle (2\unicode[STIX]{x1D70B})^{1-2s}\unicode[STIX]{x1D6E4}(2s-1)J(s) & = & \displaystyle \int _{\mathbf{R}}\frac{(x+i)^{2r}}{|x+i|^{2s+2r}}\int _{Y}N(Y)^{s+r-2}e^{2\unicode[STIX]{x1D70B}i(x+i)\operatorname{tr}(Y)}\nonumber\\ \displaystyle & & \displaystyle \times \,\biggl(\int _{t\geqslant 0}e^{-2\unicode[STIX]{x1D70B}t\operatorname{tr}(Y)}t^{2s-1}\,\frac{dt}{t}\biggr)\,d^{\ast }Y\,dx.\nonumber\end{eqnarray}$$

Define a logarithm on $\mathbf{C}$ by removing the nonpositive reals, and use this logarithm to define $z^{s}$ for such a  $z$ . Then switching the order of integration we obtain

$$\begin{eqnarray}\displaystyle (2\unicode[STIX]{x1D70B})^{1-2s}\unicode[STIX]{x1D6E4}(2s-1)J(s) & = & \displaystyle \int _{\mathbf{R}}\int _{t\geqslant 0}\frac{(x+i)^{2r}}{|x+i|^{2s+2r}}(1+t-ix)^{-3(s+r-2)}t^{2s-1}\nonumber\\ \displaystyle & & \displaystyle \times \,\biggl(\int _{Y}N(Y^{\prime })^{s+r-2}e^{-2\unicode[STIX]{x1D70B}\operatorname{tr}(Y^{\prime })}\,d^{\ast }(Y^{\prime })\biggr)\,\frac{dt}{t}\,dx,\nonumber\end{eqnarray}$$

where $Y^{\prime }=(1+t-ix)Y$ . The integral over $Y$ is the so-called Siegel integral, and it is a nonzero constant times

$$\begin{eqnarray}2^{-(3s+3r-6)}\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s+2r-4)\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s+2r-5)\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s+2r-6).\end{eqnarray}$$

See, for instance, [FK94, Theorem VII.1.1] for this fact.

We are thus left to evaluate the integral

$$\begin{eqnarray}K(s,r)=\int _{\mathbf{R}}\int _{t\geqslant 0}\frac{(x+i)^{2r}}{|x+i|^{2s+2r}}(1+t-ix)^{-3(s+r-2)}t^{2s-1}\frac{dt}{t}\,dx,\end{eqnarray}$$

which may be done exactly as in [Gar87, Proof of Proposition 5.1]. Up to a nonzero constant, one obtains

$$\begin{eqnarray}K(s,r)=2^{-(3s+3r-6)}\frac{\unicode[STIX]{x1D6E4}(s+3r-5)\unicode[STIX]{x1D6E4}(2s-1)}{\unicode[STIX]{x1D6E4}(s+r)\unicode[STIX]{x1D6E4}(2s+2r-5)}.\end{eqnarray}$$

Thus, again up to nonzero constants,

$$\begin{eqnarray}J(s)=(2\unicode[STIX]{x1D70B})^{2s-1}4^{-(3s+3r-6)}\frac{\unicode[STIX]{x1D6E4}(s+3r-5)}{\unicode[STIX]{x1D6E4}(s+r)\unicode[STIX]{x1D6E4}(2s+2r-5)}\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s+2r-4)\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s+2r-5)\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s+2r-6).\end{eqnarray}$$

Via the duplication formula $\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s+2r-5)\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s+2r-4)=\unicode[STIX]{x1D6E4}_{\mathbf{C}}(2s+2r-5)$ , $J(s)$ simplifies to

$$\begin{eqnarray}4^{-(3s+3r-6)}\frac{\unicode[STIX]{x1D6E4}(s+3r-5)}{\unicode[STIX]{x1D6E4}(s+r)}\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s+2r-6),\end{eqnarray}$$

again up to nonzero constants. Hence $I_{\infty ,2r}^{\unicode[STIX]{x1D6F7}}(\unicode[STIX]{x1D719},s)=a(T)D_{B}^{-s}4^{-(2s+2r-6)}\unicode[STIX]{x1D6E4}(s+3r-5)\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s+2r-6)$ , since $D_{B}=4N(T)$ by Corollary 3.4. Thus, up to a nonzero constant,

$$\begin{eqnarray}I_{\infty ,2r}^{\ast }(\unicode[STIX]{x1D719},s)=a(T)\unicode[STIX]{x1D6E4}_{\boldsymbol{ C}}(s+r-4)\unicode[STIX]{x1D6E4}_{\mathbf{C}}(s+r-3)\unicode[STIX]{x1D6E4}_{\mathbf{C}}(s+r-2)\unicode[STIX]{x1D6E4}_{\mathbf{C}}(s+3r-5),\end{eqnarray}$$

completing the proof of Theorem 6.3.

7 The group $\text{G}(\mathbf{A})$ and its Siegel Eisenstein series

The purpose of this section is to prove Theorem 5.2, where $\unicode[STIX]{x1D6F7}_{2r,\infty }$ is defined in (6.7). We will prove this theorem by first proving it for $E_{0}^{\ast }(g,s)$ , the Eisenstein series that is spherical at all places. Then we will define essentially a Maass–Shimura differential operator ${\mathcal{D}}$ , and show that ${\mathcal{D}}E_{2r}^{\ast }(g,s)=E_{2r+2}^{\ast }(g,s)$ . With this identity, Theorem 5.2 for $r\geqslant 0$ implies the same statement for $r+1$ . Here, ${\mathcal{D}}$ is an element of the universal enveloping algebra of the Lie algebra of $\text{G}(\mathbf{R})$ . To analyze $E_{0}^{\ast }(g,s)$ , we apply Langlands’ theorem of the constant term and his functional equation [Lan76]. In order to do the requisite intertwining operator calculations needed to apply these theorems of Langlands, we need some basic results about $\text{G}(\mathbf{A})$ , such as a description of the minimal parabolic subgroup and Iwasawa decomposition. We begin with a summary of these results on $\text{G}(\mathbf{A})$ .

7.1 Minimal parabolic and Iwasawa decomposition

Suppose $F$ is a field of characteristic zero, $B$ is a quaternion algebra over $F$ , $J=H_{3}(B)$ , and $\text{G}$ is the reductive $F$ -group that preserves the symplectic and quartic form on $W=F\oplus J\oplus J\oplus F$ , up to similitude. We describe a parabolic subgroup $P_{0}$ of $\text{G}$ , which is minimal when $B$ is a division algebra, and the corresponding Iwasawa decomposition.

For an element $x$ of $W$ , denote by $a(x),b(x),c(x),d(x)$ the elements of $F,J,J,F$ respectively so that $x=(a(x),b(x),c(x),d(x)).$ If $b$ is in $H_{3}(B)\subseteq M_{3}(B)$ , we denote $b_{ij}$ the element of $B$ in the $i,j$ position of $b$ . We denote by $e_{ij}$ the element of $M_{3}(B)$ that is $1$ at the $i,j$ position and zero elsewhere.

Denote by $S_{4}$ the subspace of $W$ consisting of $(a,b,c,d)$ with $a=0$ , $b_{ij}=0$ unless $i,j=1$ , $c_{1k}=c_{j1}=0$ for all $j,k$ . Then $S_{4}$ is an eight-dimensional isotropic subspace of  $W$ . Furthermore, every element $s\in S_{4}$ has rank at most two. Inside of $S_{4}$ we consider the two-dimensional subspace $S_{2}$ spanned by $(0,0,e_{33},0)$ and $(0,0,0,1)$ . Then every element of $S_{2}$ has rank at most one. Finally, inside of $S_{2}$ we consider the line $S_{1}=F(0,0,0,1)$ . (The reasoning for the numbering will become apparent shortly.) Set ${\mathcal{F}}_{0}$ to be the flag $S_{4}\supseteq S_{2}\supseteq S_{1}$ , and define $P_{0}$ to be the subgroup of $\text{G}$ stabilizing ${\mathcal{F}}_{0}$ . Now, define $S_{3}$ to be the seven-dimensional space in $S_{4}$ consisting of the $x=(0,b,c,d)$ with $b=0$ , i.e., $S_{3}=S_{4}\cap (0,0,\ast ,\ast )$ . It can be shown that since $P_{0}$ stabilizes $(0,0,0,\ast )$ , it automatically stabilizes $(0,0,\ast ,\ast )$ , and thus $P_{0}$ stabilizes  $S_{3}$ .

Proposition 7.1. The subgroup $P_{0}$ is a parabolic subgroup of $\text{G}$ . Denote by $T$ the diagonal maximal torus in $\operatorname{GSp}_{6}$ , considered inside of $\text{G}$ . If $B$ is not split over $F$ , then $T$ is a maximal split torus of $\text{G}$ , $P_{0}$ is a minimal parabolic subgroup, and $\text{G}$ has rational root type  $C_{3}$ .

Now suppose $F=\mathbf{Q}$ , and $B_{\mathbf{R}}$ is a division algebra. We describe the associated Iwasawa decomposition of $\text{G}(\mathbf{A})$ . Define $K_{p}$ to be the subgroup of $\text{G}(\mathbf{Q}_{p})$ that acts invertibly on the lattice $W(\mathbf{Z}_{p})$ . Define $K_{\infty }$ to be the subgroup of $\text{G}^{1,+}(\mathbf{R})$ that commutes with  $J$ . Here $\text{G}^{1}$ denotes the elements of $\text{G}$ with similitude equal to 1. Because $(v,w):=\langle v,wJ\rangle$ is an inner product on $W(\mathbf{R})$ , $K_{\infty }$ is compact.

Proposition 7.2. The subgroup $K_{\infty }$ is the stabilizer of $i$ for the $\text{G}^{1,+}(\mathbf{R})$ action on the Hermitian symmetric space ${\mathcal{H}}$ . For all places $v$ of $\mathbf{Q}$ , one has the decomposition $\text{G}(\mathbf{Q}_{v})=P_{0}(\mathbf{Q}_{v})K_{v}$ .

We omit the proofs of Propositions 7.1 and 7.2, as they appear to be well known.

7.2 The spherical Eisenstein series

We are now ready to analyze the spherical normalized Eisenstein series. Recall we denote by $P$ the parabolic subgroup of $\text{G}$ that is the stabilizer of the line spanned by $f=(0,0,0,1)$ . If $p\in P$ , then $p$ acts on this line as multiplication by some $d$ in $\operatorname{GL}_{1}$ , and acts on the one-dimensional space $W/(f)^{\bot }$ as multiplication by some $a$ in $\operatorname{GL}_{1}$ . Define a character $\unicode[STIX]{x1D712}_{s}:P(\mathbf{A})\rightarrow \mathbf{C}^{\times }$ as $\unicode[STIX]{x1D712}_{s}(p)=|a/d|^{s}$ . For every place $v$ of $\mathbf{Q}$ , set $f_{v}(g,s)$ to be the unique element in $\text{Ind}_{P(\mathbf{Q}_{v})}^{\text{G}(\mathbf{Q}_{v})}(\unicode[STIX]{x1D712}_{s})$ that is $1$ on $K_{v}$ . The unnormalized spherical Eisenstein series is

$$\begin{eqnarray}E_{0}(g,s)=\mathop{\sum }_{\unicode[STIX]{x1D6FE}\in P(\mathbf{Q})\backslash \text{G}(\mathbf{Q})}f(\unicode[STIX]{x1D6FE}g,s),\end{eqnarray}$$

where $f(g,s)=\prod _{\text{all}~v}f_{v}(g_{v},s)$ and the sum converges for $\text{Re}(s)\gg 0$ . The normalized Eisenstein series $E_{0}^{\ast }(g,s)$ is

(7.1) $$\begin{eqnarray}\displaystyle E_{0}^{\ast }(g,s) & = & \displaystyle D_{B}^{s}\biggl(\mathop{\prod }_{v<\infty ~\text{split}}\unicode[STIX]{x1D701}_{v}(2s)\unicode[STIX]{x1D701}_{v}(2s-2)\unicode[STIX]{x1D701}_{v}(2s-4)\biggr)\biggl(\mathop{\prod }_{v<\infty ~\text{ramified}}\unicode[STIX]{x1D701}_{v}(2s)\unicode[STIX]{x1D701}_{v}(2s-4)\biggr)\end{eqnarray}$$

(7.2) $$\begin{eqnarray}\displaystyle & & \displaystyle \times \,\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s)\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s-4)\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s-8)E_{0}(g,s).\end{eqnarray}$$

Here $D_{B}=\prod _{p<\infty ~\text{ramified}}p$ is the product of the finite primes $p$ for which $B\otimes \mathbf{Q}_{p}$ is not split. By the theory of Langlands, $E_{0}^{\ast }$ has meromorphic continuation in $s$ and is a function of moderate growth where it is finite. The purpose of this section is to describe the proof of the following theorem.

Theorem 7.3. The Eisenstein series $E_{0}^{\ast }(g,s)$ has at most finitely many poles, all contained in the set of integer and half-integer points in the interval $[0,5]$ , and satisfies the functional equation $E_{0}^{\ast }(g,s)=E_{0}^{\ast }(g,5-s)$ .

The proof will be an application of Langlands’ theory of the constant term, and the Langlands’ functional equation. We begin with the computation of intertwining operators on $\operatorname{GL}_{2}(B)$ .

For a finite prime $p$ , set $K_{p}^{B}\subseteq \operatorname{GL}_{2}(B_{p})$ to be the subgroup acting invertibly on the lattice $B_{0}(\mathbf{Z}_{p})\oplus B_{0}(\mathbf{Z}_{p})$ . Define $K_{\infty }^{B}$ to be the subgroup of $\operatorname{GL}_{2}(B_{\infty })$ consisting of those $m\in \operatorname{GL}_{2}(B_{\infty })$ with $mm^{\ast }=1$ . Consider the function $f_{v}^{B}(g,s_{1},s_{2}):\operatorname{GL}_{2}(B_{v})\rightarrow \mathbf{C}$ defined by

$$\begin{eqnarray}f_{v}^{B}((\!\begin{smallmatrix}b_{1} & \ast \\ & b_{2}\end{smallmatrix}\!),s_{1},s_{2})=|n(b_{1})|^{s_{1}+1}|n(b_{2})|^{s_{2}-1}\end{eqnarray}$$

and $f_{v}^{B}(K_{v},s_{1},s_{2})=1$ .

Definition 7.4. If $v<\infty$ , and $B_{v}$ is split, define $\unicode[STIX]{x1D701}_{B_{v}}(s)=\unicode[STIX]{x1D701}_{v}(s)\unicode[STIX]{x1D701}_{v}(s-1)$ . If $v<\infty$ and $B_{v}$ is ramified, define $\unicode[STIX]{x1D701}_{B_{v}}(s)=\unicode[STIX]{x1D701}_{v}(s)$ . If $B_{\infty }$ is ramified (as we assume throughout this section), define $\unicode[STIX]{x1D701}_{B_{\infty }}(s)=D_{B}^{s/2}\unicode[STIX]{x1D6E4}_{\mathbf{C}}(s)$ .

Lemma 7.5. For $x$ in $B$ , define $\overline{n}(x)\in \operatorname{GL}_{2}(B)$ to be the matrix $(\!\begin{smallmatrix}1 & \\ x & 1\end{smallmatrix}\!)$ . Suppose the measure on $B(\mathbf{A})$ is normalized so that $B(\mathbf{Q})\backslash B(\mathbf{A})$ has measure one, and $B_{0}(\mathbf{Z}_{p})$ has measure one for all finite primes, both split and ramified. Then, if $\text{Re}(s_{1}-s_{2})\gg 0$ ,

$$\begin{eqnarray}\int _{B_{v}}f_{v}^{B}(\overline{n}(x),s_{1},s_{2})\,dx=\frac{\unicode[STIX]{x1D701}_{B_{v}}(s_{1}-s_{2})}{\unicode[STIX]{x1D701}_{B_{v}}(s_{1}-s_{2}+2)}.\end{eqnarray}$$

Proof. Write $e,f$ for the basis of $B^{2}$ . For a finite prime $v$ , define $\unicode[STIX]{x1D6F7}_{v}$ on $B_{v}^{2}$ to be the characteristic function of $B_{0}(\mathbf{Z}_{v})^{2}$ . Define $\unicode[STIX]{x1D6F7}_{\infty }(b_{1}e+b_{2}f)=e^{-2\unicode[STIX]{x1D70B}(n(b_{1})+n(b_{2}))}$ . Note that, for all $v$ , $\unicode[STIX]{x1D6F7}_{v}(b_{1}e+b_{2}f)=\unicode[STIX]{x1D6F7}_{v}(b_{1}e)\unicode[STIX]{x1D6F7}_{v}(b_{2}f)$ , and that $\unicode[STIX]{x1D6F7}_{v}(wK_{v})=\unicode[STIX]{x1D6F7}_{v}(w)$ for all $w$ in $B_{v}^{2}$ . Set $s=s_{1}-s_{2}$ . Consider

$$\begin{eqnarray}f^{\unicode[STIX]{x1D6F7}}(g,s_{1},s_{2})=|\text{det}(gg^{\ast })|^{s_{1}+1}\int _{B_{v}^{\times }}\unicode[STIX]{x1D6F7}_{v}(yfg)|n(y)|^{s+2}\,dy.\end{eqnarray}$$

Then, the transformation properties of $f^{\unicode[STIX]{x1D6F7}}(g,s_{1},s_{2})$ implies the identity

$$\begin{eqnarray}f^{\unicode[STIX]{x1D6F7}}(g,s_{1},s_{2})=f^{\unicode[STIX]{x1D6F7}}(1,s_{1},s_{2})f_{v}^{B}(g,s_{1},s_{2}).\end{eqnarray}$$

We compute

$$\begin{eqnarray}\displaystyle \int _{B_{v}}f^{\unicode[STIX]{x1D6F7}}(\overline{n}(x),s_{1},s_{2})\,dx & = & \displaystyle \int _{B_{v}^{\times }}\int _{B_{v}}\unicode[STIX]{x1D6F7}(y(xe+f))|n(y)|^{s+2}\,dx\,dy\nonumber\\ \displaystyle & = & \displaystyle \int _{B_{v}^{\times }}|n(y)|^{s+2}\unicode[STIX]{x1D6F7}_{v}(yf)\int _{B_{v}}\unicode[STIX]{x1D6F7}_{v}(yxe)\,dx\,dy\nonumber\\ \displaystyle & = & \displaystyle \biggl(\int _{B_{v}^{\times }}|n(y)|^{s}\unicode[STIX]{x1D6F7}_{v}(yf)\,dy\biggr)\biggl(\int _{B_{v}}\unicode[STIX]{x1D6F7}_{v}(xe)\,dx\biggr).\nonumber\end{eqnarray}$$

Hence

$$\begin{eqnarray}\int _{B_{v}}f_{v}^{B}(\overline{n}(x),s_{1},s_{2})\,dx=\frac{(\int _{B_{v}^{\times }}|n(y)|^{s}\unicode[STIX]{x1D6F7}_{v}(yf)\,dy)(\int _{B_{v}}\unicode[STIX]{x1D6F7}_{v}(xe)\,dx)}{(\int _{B_{v}^{\times }}|n(y)|^{s+2}\unicode[STIX]{x1D6F7}_{v}(yf)\,dy)}.\end{eqnarray}$$

When $v<\infty$ , the first integral in the numerator of this fraction gives $\unicode[STIX]{x1D701}_{B_{v}(s)}$ and the second integral gives 1. When $v=\infty$ , the first integral in the numerator gives $\unicode[STIX]{x1D6E4}_{\mathbf{C}}(s)$ , up to a nonzero constant. Via Lemma 7.6, the second integral gives $D_{B}^{-1}$ . The lemma follows.◻

Since $B_{\mathbf{R}}$ is the Hamilton quaternions, we have $B_{\mathbf{R}}=\mathbf{R}\oplus \mathbf{R}i\oplus \mathbf{R}j\oplus \mathbf{R}k$ with $i^{2}=j^{2}=k^{2}=-1$ , and $ij=k$ .

Lemma 7.6. Suppose $\unicode[STIX]{x1D707}_{\text{std}}$ is the measure on $B_{\mathbf{R}}$ that gives the box spanned by $1,i,j,k$ measure one, and $\unicode[STIX]{x1D707}_{\text{norm}}$ is the measure determined by the statement of Lemma 7.5. Then $\unicode[STIX]{x1D707}_{\text{norm}}=4D_{B}^{-1}\unicode[STIX]{x1D707}_{\text{std}}$ .

Proof. The normalized measure is characterized by $\unicode[STIX]{x1D707}_{\text{norm}}(B_{0}\backslash B_{\mathbf{R}})=1$ . Suppose $B_{0}$ corresponds to the half-integral symmetric matrix  $T$ . Then the covolume of $B_{0}$ in $B_{\mathbf{R}}$ in $\unicode[STIX]{x1D707}_{\text{std}}$ is $\det (T)$ . Hence $\unicode[STIX]{x1D707}_{\text{norm}}=\det (T)^{-1}\unicode[STIX]{x1D707}_{\text{std}}$ . Since $D_{B}=4\det (T)$ by Corollary 3.4, we are done.◻

Denote by $M_{0}\subseteq P_{0}$ the centralizer of the maximal split torus $T$ defined in Proposition 7.1. We now describe some characters on $M_{0}(\mathbf{A})$ . First, the weight spaces for the action of $T$ on $W$ are exactly the coordinate spaces, i.e., the $a(v),b_{ij}(v),c_{ij}(v),d(v)$ , with $i\leqslant j$ . Since $T$ acts on these spaces with distinct weights, $M_{0}$ is exactly the subgroup of $\text{G}$ preserving the coordinate spaces.

Consistent with the notation in (2.2), we write $a_{i}(b(v))=b_{i+1,i-1}(v)$ and $c_{i}(b(v))=b_{i,i}(v)$ , with the indices in the set $\{1,2,3\}$ and taken modulo 3. If $m\in M_{0}$ , one can verify easily that $m$ preserves the norm on the coordinate spaces $a_{i}(b(v))$ , up to scaling. For $m\in M_{0}$ , define $m_{i}$ to be the endomorphism of $B=a_{i}(\{b(v):v\in W\})$ so that $a_{i}(b(vm))=\unicode[STIX]{x1D708}(m)(a_{i}(b(v)))m_{i}$ for all $v$ in  $W$ . (In fact, one can show that $\unicode[STIX]{x1D708}$ and the $m_{i}$ determine  $m$ .) Since $m_{i}$ preserves the norm on $B$ up to scaling, define $|m_{i}|$ by the equality $|n(bm_{i})|=|m_{i}|\,|n(b)|$ , $b\in B$ . Similarly, define $x_{i}(m)$ by the equality $c_{i}(b(vm))=\unicode[STIX]{x1D708}(m)x_{i}(m)c_{i}(b(v))$ . One immediately sees $|m_{1}|=|x_{2}x_{3}|$ , $|m_{2}|=|x_{1}x_{3}|$ , $|m_{3}|=|x_{1}x_{2}|$ .

Recall the elements $e=(1,0,0,0)$ and $f=(0,0,0,1)$ of $W$ . Since $M_{0}$ preserves the quartic form $Q$ up to similitude squared, $fm=(\unicode[STIX]{x1D708}(m)x_{1}x_{2}x_{3})^{-1}f$ . Since $M_{0}$ preserves the symplectic form up to similitude, $em=\unicode[STIX]{x1D708}(m)^{2}x_{1}x_{2}x_{3}e$ . Thus $\unicode[STIX]{x1D712}_{s}(m)=|\unicode[STIX]{x1D708}(m)|^{3s}|m_{1}|^{s}|m_{2}|^{s}|m_{3}|^{s}$ . One has the formula $\unicode[STIX]{x1D6FF}_{P_{0}}(m)=|\unicode[STIX]{x1D708}(m)|^{15}|m_{1}|^{9}|m_{2}|^{5}|m_{3}|$ . Set

$$\begin{eqnarray}\unicode[STIX]{x1D706}_{s}(m)=|\unicode[STIX]{x1D708}(m)|^{3s-15/2}|m_{1}|^{s-9/2}|m_{2}|^{s-5/2}|m_{3}|^{s-1/2},\end{eqnarray}$$

so that $\unicode[STIX]{x1D712}_{s}(m)=\unicode[STIX]{x1D706}_{s}(m)\unicode[STIX]{x1D6FF}_{P_{0}}^{1/2}(m)$ .

To understand the analytic properties of $E_{0}^{\ast }(g,s)$ , it suffices to restrict to $\text{G}^{1}$ , so from now on we drop the similitudes $\unicode[STIX]{x1D708}$ . From Proposition 7.1, $\text{G}$ is of type $C_{3}$ ; let us write $\pm u_{i}\pm u_{j},$ $i\neq j$ , $\pm 2u_{i}$ for its root system determined by the unipotent radical of  $P_{0}$ . Before proceeding to the intertwining operator calculations, we state one more preliminary fact. Consider the space of characters $\text{Char}(M_{0}(\mathbf{A}))^{|\cdot |}$ of $M_{0}(\mathbf{A})$ that factor through the maps $|m_{i}|$ ; i.e., this is the space of characters consisting exactly of the maps

$$\begin{eqnarray}m\mapsto |m_{1}|^{s_{1}}|m_{2}|^{s_{2}}|m_{3}|^{s_{3}}\end{eqnarray}$$

for $s_{1},s_{2},s_{3}\in \mathbf{C}$ . We write $s_{1}u_{1}+s_{2}u_{2}+s_{3}u_{3}$ for this character. Denote by $\text{W}_{\text{Sp}_{6}}=(\pm 1)^{3}\rtimes S_{3}$ the Weyl group of $\operatorname{Sp}_{6}$ , embedded in  $\text{G}$ . We identify $\text{W}_{\text{Sp}_{6}}$ with the subset of $\operatorname{Sp}_{6}$ that preserves the set $\{\pm e_{1},\pm e_{2},\pm e_{3},\pm f_{1},\pm f_{2},\pm f_{3}\}$ . Using the action of $\text{W}_{\text{Sp}_{6}}$ on $\bigwedge ^{3}(W_{6})$ , one checks easily that $\text{W}_{\text{Sp}_{6}}$ preserves $\text{Char}(M_{0}(\mathbf{A}))^{|\cdot |}$ and acts on it in the usual way; e.g., the transposition (12) in $S_{3}$ takes $s_{1}u_{1}+s_{2}u_{2}+s_{3}u_{3}$ to $s_{2}u_{1}+s_{1}u_{2}+s_{3}u_{3}$ and the inversion $(-1,1,1)$ in $(\pm 1)^{3}$ takes this character to $-s_{1}u_{1}+s_{2}u_{2}+s_{3}u_{3}$ .

Restricting characters in $\text{Char}(M_{0}(\mathbf{A}))^{|\cdot |}$ to $T(\mathbf{A})$ and dividing by two gives an identification

$$\begin{eqnarray}\text{Char}(M_{0}(\mathbf{A}))^{|\cdot |}\rightarrow X_{un}^{\ast }(T(\mathbf{A}))\simeq X^{\ast }(T)\otimes _{\boldsymbol{ Z}}\mathbf{C},\end{eqnarray}$$

where $X_{un}^{\ast }(T(\mathbf{A}))$ denotes the characters on $T(\mathbf{A})$ that factor through $|\cdot |$ . Thus, if $\unicode[STIX]{x1D6FC}^{\vee }$ is a coroot for $\operatorname{Sp}_{6}$ , and $\unicode[STIX]{x1D707}$ is a character of $M_{0}(\mathbf{A})$ in $\text{Char}(M_{0}(\mathbf{A}))^{|\cdot |}$ , we may write $\langle \unicode[STIX]{x1D6FC}^{\vee },\unicode[STIX]{x1D707}\rangle$ . For example, for the character $\unicode[STIX]{x1D707}=s_{1}u_{1}+s_{2}u_{2}+s_{3}u_{3}$ , and $\unicode[STIX]{x1D6FC}^{\vee }=(u_{1}+u_{2})^{\vee }$ , $\langle \unicode[STIX]{x1D6FC}^{\vee },\unicode[STIX]{x1D707}\rangle =s_{1}+s_{2}$ .

Now we describe the constant term and functional equation computations. For the functional equation, set $w_{8}\in \text{W}_{\text{Sp}_{6}}$ to be the element represented by

so that $w_{8}(e_{i})=f_{4-i}$ and $w_{8}(f_{i})=-e_{4-i}$ . Then for the spherical section $f(g,s)$ in $\text{Ind}_{P_{0}(\mathbf{A})}^{\text{G}(\mathbf{A})}(\unicode[STIX]{x1D712}_{s})$ , define

$$\begin{eqnarray}M(f)(g,s)=\int _{U_{P}(\mathbf{A})\cong H_{3}(B)(\mathbf{A})}f(w_{8}^{-1}n(X)g,s)\,dX\end{eqnarray}$$

for $\text{Re}(s)$ sufficiently large. The measure is normalized so that every closed connected subgroup $N$ of $U_{0}$ has $N(\mathbf{Q})\backslash N(\mathbf{A})$ given measure one. Then $M(f)$ is a spherical section in $\text{Ind}_{P_{0}(\mathbf{A})}^{\text{G}(\mathbf{A})}(\unicode[STIX]{x1D712}_{5-s})$ . Indeed, $w_{8}$ is the Weyl group element taking $u_{1}\mapsto -u_{3}$ , $u_{2}\mapsto -u_{2}$ , $u_{3}\mapsto -u_{1}$ , and thus that $M(f)$ is in $\text{Ind}_{P_{0}(\mathbf{A})}^{\text{G}(\mathbf{A})}(\unicode[STIX]{x1D712}_{5-s})$ follows immediately from the computation of $\unicode[STIX]{x1D6FF}_{P_{0}}$ above. That $M(f)$ is spherical of course follows from the Iwasawa decomposition. Hence $M(f_{s})$ is equal to $c(s)f(g,5-s)$ for some function $c(s)$ . Langlands’ functional equation of the Eisenstein series is $E_{0}^{\ast }(g,f_{s})=E_{0}^{\ast }(g,M(f_{s}))$ . We will determine $c(s)$ shortly, and then use this to check the functional equation stated in Theorem 7.3.

Now for the constant term of the Eisenstein series. We must compute

$$\begin{eqnarray}E_{0}^{\ast ,P_{0}}(g,s):=\int _{U_{0}(\mathbf{Q})\backslash U_{0}(\mathbf{A})}E_{0}^{\ast }(ug,s)\,du.\end{eqnarray}$$

Langlands’ theory of the constant term says that the Eisenstein series has analytic continuation to the same region as that of the constant term. For a character $\unicode[STIX]{x1D712}$ in $\text{Char}(M_{0}(\mathbf{A}))^{|\cdot |}$ , denote by $\unicode[STIX]{x1D6F7}_{\unicode[STIX]{x1D712}}^{\text{G}}$ the unique spherical element of $\text{Ind}_{P_{0}(\mathbf{A})}^{\text{G}(\mathbf{A})}(\unicode[STIX]{x1D6FF}_{P_{0}}^{1/2}\unicode[STIX]{x1D712})$ . For $w\in \text{W}_{\text{Sp}_{6}}$ , set

$$\begin{eqnarray}A(w,\unicode[STIX]{x1D712})(g):=\int _{N_{w}(\mathbf{A})}\unicode[STIX]{x1D6F7}_{\unicode[STIX]{x1D712}}^{\text{G}}(w^{-1}ng)\,dn\end{eqnarray}$$

when it is absolutely convergent. Here $N_{w}$ is the group $\prod _{\unicode[STIX]{x1D6FC}>0,w^{-1}(\unicode[STIX]{x1D6FC})<0}U_{\unicode[STIX]{x1D6FC}}$ , and $U_{\unicode[STIX]{x1D6FC}}$ is the root group in $\text{G}$ corresponding to the root $\unicode[STIX]{x1D6FC}$ . Then $A(w,\unicode[STIX]{x1D712})(g)$ is in $\text{Ind}_{P_{0}(\mathbf{A})}^{\text{G}(\mathbf{A})}(\unicode[STIX]{x1D6FF}_{P_{0}}^{1/2}w(\unicode[STIX]{x1D712}))$ , and is equal to $c_{w}(\unicode[STIX]{x1D712})\unicode[STIX]{x1D6F7}_{w(\unicode[STIX]{x1D712})}^{\text{G}}$ for a certain factorizable function $c_{w}=\prod _{v}c_{w}^{v}$ on $\text{Char}(M_{0}(\mathbf{A}))^{|\cdot |}$ . By the usual argument [GPSR87, (5.6)], one has

$$\begin{eqnarray}E_{0}^{P_{0}}(g,s)=\mathop{\sum }_{w\in \unicode[STIX]{x1D6FA}}c_{w}(\unicode[STIX]{x1D706}_{s})\unicode[STIX]{x1D6F7}_{w(\unicode[STIX]{x1D706}_{s})}^{\text{G}}(g),\end{eqnarray}$$

where $\unicode[STIX]{x1D6FA}$ is a set of representatives in $\text{W}_{\text{Sp}_{6}}$ for the double coset $P_{0}(\mathbf{Q})\backslash \text{G}(\mathbf{Q})/P(\mathbf{Q})$ .

This double coset has eight elements. The eight Weyl group elements $w_{i}\in \unicode[STIX]{x1D6FA}$ can be obtained from [GPSR87, Lemma 5.1]. They are indexed by the sets $S_{i}$ of negative roots $\unicode[STIX]{x1D6FD}$ for which $w_{i}(\unicode[STIX]{x1D6FD})$ is positive. The sets are:

1. (i) $S_{1}=\varnothing$ ; $c_{1}^{v}(s)=1$ ;

2. (ii) $S_{2}=\{-2u_{3}\}$ ; $c_{2}^{v}(s)=\unicode[STIX]{x1D701}_{v}(2s-1)/\unicode[STIX]{x1D701}_{v}(2s)$ ;

3. (iii) $S_{3}=S_{2}\cup \{-(u_{2}+u_{3})\}$ ; $c_{3}^{v}(s)=c_{2}^{v}(s)(\unicode[STIX]{x1D701}_{B_{v}}(2s-3)/\unicode[STIX]{x1D701}_{B_{v}}(2s-1))$ ;

4. (iv) $S_{4}=S_{3}\cup \{-2u_{2}\}$ ; $c_{4}^{v}(s)=c_{3}^{v}(s)(\unicode[STIX]{x1D701}_{v}(2s-5)/\unicode[STIX]{x1D701}_{v}(2s-4))$ ;

5. (v) $S_{5}=S_{3}\cup \{-(u_{1}+u_{3})\}$ ; $c_{5}^{v}(s)=c_{3}^{v}(s)(\unicode[STIX]{x1D701}_{B_{v}}(2s-5)/\unicode[STIX]{x1D701}_{B_{v}}(2s-3))$ ;

6. (vi) $S_{6}=S_{5}\cup \{-2u_{2}\}$ ; $c_{6}^{v}(s)=c_{5}^{v}(s)(\unicode[STIX]{x1D701}_{v}(2s-5)/\unicode[STIX]{x1D701}_{v}(2s-4))$ ;

7. (vii) $S_{7}=S_{6}\cup \{-(u_{1}+u_{2})\}$ ; $c_{7}^{v}(s)=c_{6}^{v}(s)(\unicode[STIX]{x1D701}_{B_{v}}(2s-7)/\unicode[STIX]{x1D701}_{B_{v}}(2s-5))$ ;

8. (viii) $S_{8}=S_{7}\cup \{-2u_{1}\}=\overline{U_{P}}$ ; $c_{v}(s)=c_{8}^{v}(s)=c_{7}^{v}(s)(\unicode[STIX]{x1D701}_{v}(2s-9)/\unicode[STIX]{x1D701}_{v}(2s-8))$ .

We have also listed the functions $c_{i}^{v}(s):=c_{w_{i}}^{v}(\unicode[STIX]{x1D706}_{s})$ . Here, for $v=\infty$ , $\unicode[STIX]{x1D701}_{v}(s)$ means $\unicode[STIX]{x1D6E4}_{\mathbf{R}}(s)$ . The computation of the functions $c_{i}(s)$ proceeds by the usual Gindikin–Karpelevich argument of factorization of intertwining operators and reduction to rational rank one. Recall that if the length of $w\in \text{W}_{\text{Sp}_{6}}$ is $r$ , and $w=s_{1}s_{2}\cdots s_{r}$ is an expression of $w$ as a product of $r$ reflections, $s_{i}$ the reflection in the simple root $\unicode[STIX]{x1D6FC}_{i}$ , then

(7.3) $$\begin{eqnarray}c(w,\unicode[STIX]{x1D712})=\mathop{\prod }_{k=1}^{r}\biggl(\int _{U_{\unicode[STIX]{x1D6FC}_{k}}(\mathbf{A})}\unicode[STIX]{x1D6F7}_{s_{k+1}\cdots s_{r}(\unicode[STIX]{x1D712})}^{\text{G}}(s_{k}^{-1}n)\,dn\biggr).\end{eqnarray}$$

If $\unicode[STIX]{x1D6FC}$ is a short simple root, then the root groups $U_{\unicode[STIX]{x1D6FC}}$ and $U_{-\unicode[STIX]{x1D6FC}}$ sit inside a $\operatorname{GL}_{2}(B)/\unicode[STIX]{x1D707}_{2}$ in $\text{G}$ , that acts on $W$ via the maps $m(t)$ of § 2. Now, one checks that if $K_{v}^{B}$ is the compact subgroup of $\operatorname{GL}_{2}(B_{v})$ defined above Lemma 7.5, then $K_{v}^{B}/\unicode[STIX]{x1D707}_{2}$ sits inside $K_{v}$ , for both of these short simple roots, and for all places $v$ of $\mathbf{Q}$ . It thus follows from Lemma 7.5 that if $\unicode[STIX]{x1D6FC}_{k}$ is a short simple root,

$$\begin{eqnarray}\int _{U_{\unicode[STIX]{x1D6FC}_{k}}(\mathbf{Q}_{v})}\unicode[STIX]{x1D6F7}_{s_{k+1}\cdots s_{r}(\unicode[STIX]{x1D706}_{s})}^{\text{G}}(s_{k}^{-1}n)\,dn=\frac{\unicode[STIX]{x1D701}_{B_{v}}(\langle \unicode[STIX]{x1D6FC}_{k}^{\vee },s_{k+1}\cdots s_{r}(\unicode[STIX]{x1D706}_{s})\rangle )}{\unicode[STIX]{x1D701}_{B_{v}}(\langle \unicode[STIX]{x1D6FC}_{k}^{\vee },s_{k+1}\cdots s_{r}(\unicode[STIX]{x1D712})\rangle +2)}=\frac{\unicode[STIX]{x1D701}_{B_{v}}(\langle \unicode[STIX]{x1D6FD}_{k}^{\vee },\unicode[STIX]{x1D706}_{s}\rangle )}{\unicode[STIX]{x1D701}_{B_{v}}(\langle \unicode[STIX]{x1D6FD}_{k}^{\vee },\unicode[STIX]{x1D706}_{s}\rangle +2)}\end{eqnarray}$$

where $\unicode[STIX]{x1D6FD}_{k}=s_{r}\cdots s_{k+1}(\unicode[STIX]{x1D6FC}_{k})$ .

If $\unicode[STIX]{x1D6FC}_{k}$ is long root, the analogous formula is

$$\begin{eqnarray}\int _{U_{\unicode[STIX]{x1D6FC}_{k}}(\mathbf{Q}_{v})}\unicode[STIX]{x1D6F7}_{s_{k+1}\cdots s_{r}(\unicode[STIX]{x1D706}_{s})}^{\text{G}}(s_{k}^{-1}n)\,dn=\frac{\unicode[STIX]{x1D701}_{v}(2\langle \unicode[STIX]{x1D6FD}_{k}^{\vee },\unicode[STIX]{x1D706}_{s}\rangle )}{\unicode[STIX]{x1D701}_{v}(2\langle \unicode[STIX]{x1D6FD}_{k}^{\vee },\unicode[STIX]{x1D706}_{s}\rangle +1)}.\end{eqnarray}$$

This formula is even easier to establish, because the long root subgroups sit inside $\operatorname{Sp}_{6}$ , so the computation may done in an $\operatorname{SL}_{2}$ sitting inside  $\operatorname{Sp}_{6}$ .

For $w=s_{1}\cdots s_{r}$ , the roots $-\unicode[STIX]{x1D6FD}_{k}$ are exactly the negative roots $\unicode[STIX]{x1D6FD}$ for which $w(\unicode[STIX]{x1D6FD})>0$ . The computation of the $c_{i}^{v}(s)$ above now follows. Putting in the normalization from (7.1), one now checks without much effort that the constant term of the normalized Eisenstein series $E_{0}^{\ast }(g,s)$ has at worst finitely many poles, and they all occur at the integral and half-integral points in the interval $[0,5]$ . This completes the proof of the statement about the poles of $E_{0}^{\ast }(g,s)$ .

Now for the functional equation. Recall $M(f_{s})=c(s)f_{5-s}$ . Then $c(s)=c_{8}(s)$ is determined above. For $v<\infty$ split, one finds

$$\begin{eqnarray}c_{v}(s)=\frac{\unicode[STIX]{x1D701}_{v}(2s-5)\unicode[STIX]{x1D701}_{v}(2s-7)\unicode[STIX]{x1D701}_{v}(2s-9)}{\unicode[STIX]{x1D701}_{v}(2s)\unicode[STIX]{x1D701}_{v}(2s-2)\unicode[STIX]{x1D701}_{v}(2s-4)}.\end{eqnarray}$$

For $v<\infty$ ramified, one gets

$$\begin{eqnarray}c_{v}(s)=\frac{\unicode[STIX]{x1D701}_{v}(2s-5)\unicode[STIX]{x1D701}_{v}(2s-7)\unicode[STIX]{x1D701}_{v}(2s-9)}{\unicode[STIX]{x1D701}_{v}(2s)\unicode[STIX]{x1D701}_{v}(2s-4)\unicode[STIX]{x1D701}_{v}(2s-8)}.\end{eqnarray}$$

Using the duplication formula $\unicode[STIX]{x1D6E4}_{\mathbf{R}}(z)\unicode[STIX]{x1D6E4}_{\mathbf{R}}(z+1)=\unicode[STIX]{x1D6E4}_{\mathbf{C}}(z)$ , one obtains

$$\begin{eqnarray}c_{\infty }(s)=D_{B}^{-3}\frac{\unicode[STIX]{x1D6E4}_{\mathbf{C}}(2s-5)\unicode[STIX]{x1D6E4}_{\mathbf{C}}(2s-7)\unicode[STIX]{x1D6E4}_{\mathbf{C}}(2s-9)}{\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s)^{2}\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s-4)^{2}\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s-8)^{2}}.\end{eqnarray}$$

Now, we have

$$\begin{eqnarray}E_{0}^{\ast }(g,s)=D_{B}^{s}\unicode[STIX]{x1D6EC}_{\mathbf{Q}}(2s)\unicode[STIX]{x1D6EC}_{\mathbf{Q}}(2s-2)\unicode[STIX]{x1D6EC}_{\mathbf{Q}}(2s-4)\biggl(\mathop{\prod }_{v<\infty ~\text{ramified}}\frac{1}{\unicode[STIX]{x1D701}_{v}(2s-2)}\biggr)\frac{\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s-8)}{\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s-2)}E_{0}(g,s)\end{eqnarray}$$

where $\unicode[STIX]{x1D6EC}_{\mathbf{Q}}(s)=\prod _{v}\unicode[STIX]{x1D701}_{v}(s)$ denotes the completed Riemann zeta function. Thus, since $\unicode[STIX]{x1D6EC}_{\mathbf{Q}}(s)=\unicode[STIX]{x1D6EC}_{\mathbf{Q}}(1-s)$ ,

$$\begin{eqnarray}\displaystyle E_{0}^{\ast }(g,5-s) & = & \displaystyle D_{B}^{5-s}\unicode[STIX]{x1D6EC}_{\mathbf{Q}}(2s-9)\unicode[STIX]{x1D6EC}_{\mathbf{Q}}(2s-7)\unicode[STIX]{x1D6EC}_{\mathbf{Q}}(2s-5)\nonumber\\ \displaystyle & & \displaystyle \times \,\biggl(\mathop{\prod }_{v<\infty ~\text{ramified}}\frac{1}{\unicode[STIX]{x1D701}_{v}(8-2s)}\biggr)\frac{\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2-2s)}{\unicode[STIX]{x1D6E4}_{\mathbf{R}}(8-2s)}E_{0}(g,5-s).\nonumber\end{eqnarray}$$

The intertwining operator gives

$$\begin{eqnarray}\displaystyle E_{0}^{\ast }(g,s) & = & \displaystyle D_{B}^{s-3}\unicode[STIX]{x1D6EC}_{\mathbf{Q}}(2s-9)\unicode[STIX]{x1D6EC}_{\mathbf{Q}}(2s-7)\unicode[STIX]{x1D6EC}_{\mathbf{Q}}(2s-5)\nonumber\\ \displaystyle & & \displaystyle \times \,\biggl(\mathop{\prod }_{v<\infty ~\text{ramified}}\frac{1}{\unicode[STIX]{x1D701}_{v}(2s-8)}\biggr)\frac{\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s-6)}{\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s)}E_{0}(g,5-s).\nonumber\end{eqnarray}$$

Thus

$$\begin{eqnarray}\frac{E_{0}^{\ast }(g,5-s)}{E_{0}^{\ast }(g,s)}=D_{B}^{8-2s}\biggl(\mathop{\prod }_{v<\infty ~\text{ram}}\frac{\unicode[STIX]{x1D701}_{v}(2s-8)}{\unicode[STIX]{x1D701}_{v}(8-2s)}\biggr)\frac{\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2-2s)\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s)}{\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s-6)\unicode[STIX]{x1D6E4}_{\mathbf{R}}(8-2s)}.\end{eqnarray}$$

But $\unicode[STIX]{x1D701}_{v}(s)/\unicode[STIX]{x1D701}_{v}(-s)=-p^{s}$ , and $\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2z)\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2-2z)=1/(\sin (\unicode[STIX]{x1D70B}z))$ . Thus the above fraction is

$$\begin{eqnarray}D_{B}^{8-2s}\biggl(\mathop{\prod }_{v<\infty ~\text{ram}}-p^{2s-8}\biggr)\frac{\sin (\unicode[STIX]{x1D70B}(s-3))}{\sin (\unicode[STIX]{x1D70B}s)}=1\end{eqnarray}$$

since $B$ is ramified at an even number of places. This completes the proof of Theorem 7.3.

7.3 Differential operators

In this section we analyze the normalized Eisenstein series $E_{2r}^{\ast }(g,s)$ that is used in the Rankin–Selberg integral for a Siegel modular form on $\operatorname{GSp}_{6}$ of even weight $2r$ . In all that follows, $r\geqslant 0$  is an integer. This Eisenstein series is defined as follows. At finite places $p<\infty$ , define $f_{p}^{\ast }(g,s)$  the normalized section as above, so that $f_{p}^{\ast }(K_{p},s)=\unicode[STIX]{x1D701}_{p}(2s)\unicode[STIX]{x1D701}_{p}(2s-4)$ if $p$ is ramified for $B$ , and $f_{p}^{\ast }(K_{p},s)=\unicode[STIX]{x1D701}_{p}(2s)\unicode[STIX]{x1D701}_{p}(2s-2)\unicode[STIX]{x1D701}_{p}(2s-4)$ if $p$ is split for  $B$ . At infinity, the normalized section is defined as follows. Define $f_{\infty ,2r}$ to be the unique element of $\text{Ind}_{P(\mathbf{R})}^{\text{G}(\mathbf{R})}(\unicode[STIX]{x1D712}_{s})$ that satisfies $f_{\infty ,2r}(k,s)=j(k,i)^{2r}$ for all $k\in K_{\infty }$ . Then the normalized section

$$\begin{eqnarray}f_{\infty ,2r}^{\ast }(g,s)=\unicode[STIX]{x1D6E4}_{\boldsymbol{ R}}(2s+2r)\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s+2r-4)\unicode[STIX]{x1D6E4}_{\mathbf{R}}(2s+2r-8)f_{\infty ,2r}(g,s).\end{eqnarray}$$

Finally,

$$\begin{eqnarray}E_{2r}^{\ast }(g,s)=D_{B}^{s}\mathop{\sum }_{\unicode[STIX]{x1D6FE}\in P(\mathbf{Q})\backslash G(\mathbf{Q})}f_{2r}^{\ast }(\unicode[STIX]{x1D6FE}g,s),\end{eqnarray}$$

where $f_{2r}^{\ast }(g,s)=f_{\infty ,2r}^{\ast }(g,s)\prod _{v<\infty }f_{v}^{\ast }(g,s)$ . Thus when $r=0$ , we recover the spherical normalized Eisenstein series.

We will use (essentially) the Maass–Shimura differential operators [Maa71, § 19], [Shi81] to obtain the desired analytic properties of $E_{2r}^{\ast }(g,s)$ from $E_{0}^{\ast }(g,s)$ , as stated in the following theorem.

Theorem 7.7. There is an element ${\mathcal{D}}$ of ${\mathcal{U}}(\mathfrak{g})\otimes \mathbf{C}$ , the complexified universal enveloping algebra of the Lie algebra of $\text{G}(\mathbf{R})$ , such that ${\mathcal{D}}E_{2r}^{\ast }(g,s)=E_{2r+2}^{\ast }(g,s)$ . Consequently, $E_{2r}^{\ast }(g,s)=E_{2r}^{\ast }(g,5-s)$ and $E_{2r}^{\ast }(g,s)$ has at worst prescribed poles at the integral and half-integral points in the interval $[0,5]$ .

The proof will work as follows. First, recall that $j(g,i)=\langle r(i)g^{-1},f\rangle$ , so that $j(tg,i)=t^{-1}j(g,i)$ if $t\in \operatorname{GL}_{1}$ denotes, by slight abuse of notation, the element of $\text{G}$ that acts as multiplication by the scalar $t$ on  $W$ . Define $J(g,i)=\unicode[STIX]{x1D708}(g)j(g,i)=\langle r(i),fg\rangle$ , and for $v\in W(\mathbf{R})$ , recall that $\unicode[STIX]{x1D6F7}_{\infty ,2r}(v)=e^{-\Vert v\Vert ^{2}}\langle r(i),v\rangle ^{2r}$ . Then

$$\begin{eqnarray}\unicode[STIX]{x1D6F7}_{\infty ,2r}(fg)=e^{-|J(g,i)|^{2}}J(g,i)^{2r}.\end{eqnarray}$$

One has

(7.4) $$\begin{eqnarray}f_{\infty ,2r}^{\ast }(g,s)=\unicode[STIX]{x1D70B}^{-(3s+3r-6)}\unicode[STIX]{x1D6E4}(s+r-2)\unicode[STIX]{x1D6E4}(s+r-4)|\unicode[STIX]{x1D708}(g)|^{s}\int _{\operatorname{GL}_{1}(\mathbf{R})}\unicode[STIX]{x1D6F7}_{2r}(tfg)|t|^{2s}\,dt.\end{eqnarray}$$

Indeed, the function of $g$ on the right-hand side of (7.4) is in the correct induction space, has the correct transformation with respect to $K_{\infty }$ , and at $g=1$ is $\unicode[STIX]{x1D70B}^{-(3s+3r-6)}\unicode[STIX]{x1D6E4}(s+r)\unicode[STIX]{x1D6E4}(s+r-2)\unicode[STIX]{x1D6E4}(s+r-4)$ since the inner integral at $g=1$ is $\unicode[STIX]{x1D6E4}(s+r)$ . Hence this $f_{\infty ,2r}^{\ast }$ agrees with normalized section used in the body of the paper. In different notation,

$$\begin{eqnarray}f_{\infty ,2r}^{\ast }(g,s)=\unicode[STIX]{x1D70B}^{-(3s+3r-6)}\unicode[STIX]{x1D6E4}(s+r)\unicode[STIX]{x1D6E4}(s+r-2)\unicode[STIX]{x1D6E4}(s+r-4)|\unicode[STIX]{x1D708}(g)|^{s}\frac{J(g,i)^{2r}}{|J(g,i)|^{2s+2r}}.\end{eqnarray}$$

To compute the action of ${\mathcal{D}}$ on $f_{\infty ,2r}^{\ast }(g,s)$ and thus $E_{2r}^{\ast }(g,s)$ , we compute it on the function of  $g$   $g\mapsto \unicode[STIX]{x1D6F7}_{\infty ,2r}(fg)$ . We will find

(7.5) $$\begin{eqnarray}{\mathcal{D}}\unicode[STIX]{x1D6F7}_{\infty ,2r}(fg)=\unicode[STIX]{x1D70B}^{-3}e^{-|J(g,i)|^{2}}\{|J(g,i)|^{4}-9|J(g,i)|^{2}+15\}J(g,i)^{2r+2}.\end{eqnarray}$$

Now since

$$\begin{eqnarray}\int _{\operatorname{GL}_{1}(\mathbf{R})}e^{-t^{2}|J(g,i)|^{2}}|J(g,i)|^{2v}J(g,i)^{2r+2}|t|^{2s+2r+2+2v}\,dt=\unicode[STIX]{x1D6E4}(s+r+1+v)\frac{J(g,i)^{2r+2}}{|J(g,i)|^{2s+2r+2}},\end{eqnarray}$$

(7.5) implies

$$\begin{eqnarray}\displaystyle & & \displaystyle \int _{\operatorname{GL}_{1}(\mathbf{R})}{\mathcal{D}}\unicode[STIX]{x1D6F7}_{\infty ,2r}(tg)|t|^{2s}\,dt\nonumber\\ \displaystyle & & \displaystyle \quad =\unicode[STIX]{x1D70B}^{-3}\{\unicode[STIX]{x1D6E4}(s+r+3)-9\unicode[STIX]{x1D6E4}(s+r+2)+15\unicode[STIX]{x1D6E4}(s+r+1)\}\frac{J(g,i)^{2r+2}}{|J(g,i)|^{2s+2r+2}}\nonumber\\ \displaystyle & & \displaystyle \quad =\unicode[STIX]{x1D70B}^{-3}\{(s+r+2)(s+r+1)-9(s+r+1)+15\}\unicode[STIX]{x1D6E4}(s+r+1)\frac{J(g,i)^{2r+2}}{|J(g,i)|^{2s+2r+2}}\nonumber\\ \displaystyle & & \displaystyle \quad =\unicode[STIX]{x1D70B}^{-3}(s+r-2)(s+r-4)\unicode[STIX]{x1D6E4}(s+r+1)\frac{J(g,i)^{2r+2}}{|J(g,i)|^{2s+2r+2}}.\nonumber\end{eqnarray}$$

Thus Theorem 7.7 follows as soon as we establish (7.5).

Let us now define ${\mathcal{D}}$ and prove the identity (7.5). Set $h=n(i)\overline{n}(i/2)\in \text{G}(\mathbf{C})$ . Then $r(i)h=e$ and $r(-i)h=-8if$ . Denote by $\overline{\mathfrak{u}_{P}}(\mathbf{R})$ the Lie algebra of the unipotent subgroup of $\text{G}(\mathbf{R})$ consisting of the $\overline{n}(X)$ , for $X\in J_{\mathbf{R}}$ , with $J=H_{3}(B)$ . We will define ${\mathcal{D}}_{0}$ in ${\mathcal{U}}(\overline{\mathfrak{u}_{P}}(\mathbf{R}))\simeq \operatorname{Sym}(J_{\mathbf{R}})$ , the universal enveloping algebra of $\overline{\mathfrak{u}_{P}}(\mathbf{R})$ , and then ${\mathcal{D}}$ will be given by $(2\unicode[STIX]{x1D70B}i)^{-3}h{\mathcal{D}}_{0}h^{-1}$ . Here we identify $J$ with $\overline{\mathfrak{u}_{P}}$ via the logarithm $\overline{L}$ of the map $\overline{n}$ . One has

$$\begin{eqnarray}(a,b,c,d)\overline{L}(X)=(\operatorname{tr}(b,X),c\times X,dX,0).\end{eqnarray}$$

To define ${\mathcal{D}}_{0}$ , we first rewrite the formula for the norm of an element of $J$ in coordinates. For $a\in B_{\mathbf{R}}$ , denote by $a(1),a(i),a(j),a(k)$ the real numbers defined by the equality $a=a(1)+a(i)i+a(j)j+a(k)k$ . If

$$\begin{eqnarray}x=\left(\begin{array}{@{}ccc@{}}c_{1} & a_{3} & a_{2}^{\ast }\\ a_{3}^{\ast } & c_{2} & a_{1}\\ a_{2} & a_{1}^{\ast } & c_{3}\end{array}\right),\end{eqnarray}$$

then

(7.6) $$\begin{eqnarray}\displaystyle N(x) & = & \displaystyle c_{1}c_{2}c_{3}-\biggl(\mathop{\sum }_{i=1,2,3}\mathop{\sum }_{u\in \{1,i,j,k\}}c_{i}a_{i}(u)^{2}\biggr)+2a_{1}(1)a_{2}(1)a_{3}(1)\nonumber\\ \displaystyle & & \displaystyle -\,2\biggl(\mathop{\sum }_{i=1,2,3}\mathop{\sum }_{u\in \{i,j,k\}}a_{i}(1)a_{i+1}(u)a_{i+2}(u)\biggr)-2\det (a_{j}(u)).\end{eqnarray}$$

Here

$$\begin{eqnarray}(a_{j}(u))=\left(\begin{array}{@{}ccc@{}}a_{1}(i) & a_{2}(i) & a_{3}(i)\\ a_{1}(j) & a_{2}(j) & a_{3}(j)\\ a_{1}(k) & a_{2}(k) & a_{3}(k)\end{array}\right).\end{eqnarray}$$

View the coordinates $c_{i},a_{j}(u)$ on the right-hand side of (7.6) as elements of $J^{\vee }$ , the dual of $J$ , so that the expression becomes an element of $\operatorname{Sym}^{3}(J^{\vee })$ . The trace form $\operatorname{tr}(\;,\;)$ gives an identification $J\simeq J^{\vee }$ , and thus the norm gives us an element ${\mathcal{D}}_{0}$ of $\operatorname{Sym}^{3}(J)\subseteq {\mathcal{U}}(\overline{\mathfrak{u}_{P}})\subseteq {\mathcal{U}}(\mathfrak{g})$ . This is the differential operator ${\mathcal{D}}_{0}$ defined above. More explicitly, recall that $e_{ij}$ denotes the element of $M_{3}(B)$ with a $1$ at position $i,j$ and zeroes elsewhere. Define $v_{1}(u)=ue_{23}+u^{\ast }e_{32}$ , and similarly for $v_{2}(u)$ , $v_{3}(u)$ . Then under the identification of $J^{\vee }$ with $J$ , $c_{i}$ becomes $e_{ii}$ and $a_{i}(u)$ becomes $v_{i}(u)/2$ . Hence

(7.7) $$\begin{eqnarray}\displaystyle {\mathcal{D}}_{0} & = & \displaystyle e_{11}e_{22}e_{33}-\frac{1}{4}\biggl(\mathop{\sum }_{1\leqslant i\leqslant 3,u\in \{1,i,j,k\}}e_{ii}v_{i}(u)^{2}\biggr)+\frac{1}{4}v_{1}(1)v_{2}(1)v_{3}(1)\nonumber\\ \displaystyle & & \displaystyle -\,\frac{1}{4}\biggl(\mathop{\sum }_{1\leqslant i\leqslant 3,u\in \{i,j,k\}}v_{i}(1)v_{i+1}(u)v_{i+2}(u)\biggr)-\frac{1}{4}\det (v_{j}(u)).\end{eqnarray}$$

Now ${\mathcal{D}}$ is a product of terms of the form $D(x):=h\overline{L}(-x)h^{-1}$ , for $x\in J$ . Let us compute the action of such a term on $\langle r(i),fg\rangle$ and $\langle r(-i),fg\rangle$ .

Lemma 7.8. $D(x)\langle r(i),fg\rangle =0$ . $D(x)\langle r(-i),fg\rangle =-8i\langle (0,0,x,0),fgh\rangle$ .

Proof. We have

$$\begin{eqnarray}\langle r(i),fghe^{t\overline{L}(-x)}h^{-1}\rangle =\langle ee^{t\overline{L}(x)},fgh\rangle =\langle e,fgh\rangle\end{eqnarray}$$

is independent of $t$ , proving the first claim. Similarly,

$$\begin{eqnarray}\langle r(-i),fghe^{t\overline{L}(-x)}h^{-1}\rangle =-8i\langle fe^{t\overline{L}(x)},fgh\rangle =-8i\langle f,fgh\rangle -8it\langle (0,0,x,0),fgh\rangle +O(t^{2})\end{eqnarray}$$

proving the second statement. ◻

We now compute $D(z)D(y)D(x)e^{-|J(g,i)|^{2}}$ . First, since $|J(g,i)|^{2}=\langle r(i),fg\rangle \langle r(-i),fg\rangle$ ,

$$\begin{eqnarray}D(x)e^{-|J(g,i)|^{2}}=e^{-|J(g,i)|^{2}}J(g,i)(8i\langle (0,0,x,0),fgh\rangle )=e^{-|J(g,i)|^{2}}\langle (0,0,x,0),v\rangle\end{eqnarray}$$

where $v=8iJ(g,i)fgh$ . Now, we have $D(y)\langle (0,0,x,0),fgh\rangle =\langle (0,x\times y,0,0),fgh\rangle$ . Continuing, we thus get

$$\begin{eqnarray}D(y)D(x)e^{-|J(g,i)|^{2}}=e^{-|J(g,i)|^{2}}\{\langle (0,0,y,0),v\rangle \langle (0,0,x,0),v\rangle +\langle (0,x\times y,0,0),v\rangle \}.\end{eqnarray}$$

Finally,

$$\begin{eqnarray}D(z)D(y)D(x)e^{-|J(g,i)|^{2}}=e^{-|J(g,i)|^{2}}(D_{1}(x,y,z)(v)+D_{2}(x,y,z)(v)+D_{3}(x,y,z)(v)),\end{eqnarray}$$

where, for $x,y,z\in J$ and $w\in W$ ,

$$\begin{eqnarray}\displaystyle D_{1}(x,y,z)(w) & = & \displaystyle \langle (0,0,x,0),w\rangle \langle (0,0,y,0),w\rangle \langle (0,0,z,0),w\rangle ,\nonumber\\ \displaystyle D_{2}(x,y,z)(w) & = & \displaystyle \langle (0,0,x,0),w\rangle \langle (0,y\times z,0,0),w\rangle +\langle (0,0,y,0),w\rangle \langle (0,z\times x,0,0),w\rangle \nonumber\\ \displaystyle & & \displaystyle +\,\langle (0,0,z,0),w\rangle \langle (0,x\times y,0,0),w\rangle ,\nonumber\\ \displaystyle D_{3}(x,y,z)(w) & = & \displaystyle \langle (\operatorname{tr}(z,x\times y),0,0,0),w\rangle .\nonumber\end{eqnarray}$$

Proposition 7.9. Write ${\mathcal{D}}_{0}=\sum \unicode[STIX]{x1D6FC}\,xyz$ , $\unicode[STIX]{x1D6FC}\in \mathbf{R}$ , $x,y,z\in J$ , as abstract expression for the right-hand side of (7.7). Suppose $w=(a,b,c,d)$ . Then

$$\begin{eqnarray}\sum \unicode[STIX]{x1D6FC}D_{1}(x,y,z)(w)=\det (b);\quad \sum \unicode[STIX]{x1D6FC}D_{2}(x,y,z)(w)=-3\operatorname{tr}(b,c);\quad \sum \unicode[STIX]{x1D6FC}D_{3}(x,y,z)(w)=15d.\end{eqnarray}$$

Proof. This is a straightforward but tedious calculation, so we omit it. ◻

Recall that $v=8iJ(g,i)fgh=:(a,b,c,d)$ . We obtain

$$\begin{eqnarray}(8i)^{-1}h(-{\mathcal{D}}_{0})h^{-1}e^{-|J(g,i)|^{2}}=(8i)^{-1}e^{-|J(g,i)|^{2}}(\det (b)-3\operatorname{tr}(b,c)+15d).\end{eqnarray}$$

But now,

$$\begin{eqnarray}a=\langle -f,v\rangle =J(g,i)\langle r(-i),fg\rangle =|J(g,i)|^{2}\end{eqnarray}$$

and

$$\begin{eqnarray}d=\langle e,v\rangle =8iJ(g,i)\langle r(i),fg\rangle =8iJ(g,i)^{2}.\end{eqnarray}$$

Since $v=(a,b,c,d)$ is rank one, $\det (b)=a^{2}d$ and $\operatorname{tr}(b,c)=3ad$ . Thus

$$\begin{eqnarray}\det (b)-3\operatorname{tr}(b,c)+15d=8i\{|J(g,i)|^{4}J(g,i)^{2}-9|J(g,i)|^{2}J(g,i)^{2}+15J(g,i)^{2}\}.\end{eqnarray}$$

Hence

$$\begin{eqnarray}(2\unicode[STIX]{x1D70B}i)^{-3}h{\mathcal{D}}_{0}h^{-1}e^{-|J(g,i)|^{2}}=\unicode[STIX]{x1D70B}^{-3}\{|J(g,i)|^{4}-9|J(g,i)|^{2}+15\}J(g,i)^{2}e^{-|J(g,i)|^{2}}.\end{eqnarray}$$

Since all the differential operators act trivially on the $J(g,i)$ , we get

$$\begin{eqnarray}{\mathcal{D}}(J(g,i)^{2r}e^{-|J(g,i)|^{2}})=\unicode[STIX]{x1D70B}^{-3}\{|J(g,i)|^{4}-9|J(g,i)|^{2}+15\}J(g,i)^{2r+2}e^{-|J(g,i)|^{2}},\end{eqnarray}$$

proving (7.5), and thus Theorem 7.7.